How I Use Pkgsrc

Some previous posts for reference:

So I’ve previously talked about why I think pkgsrc is cool as well as how to use pkgsrc with Clang instead of GCC. Now for this post, I’m going to go over how I am actually using pkgsrc. This may have been a little bit backwards, but it’s just the way I ended up writing stuff.

I had a few specific goals in mind that were ultimately solved by pkgsrc:

I was honestly only looking for a way to build software from source and have the code mirrored on my local network. I was partially doing this manually myself until I found pkgsrc. Then pkgsrc (and I suppose BSD in general) convinced me that having a separate installation location for system packages versus user packages would be nice. And on top of that, installing stuff into my local user’s home directory would prevent me from needing root privileges to install stuff (at the expense of only installing the software for my one user).

First of all, I’m using pkgsrc on Debian. Yes, I can install Debian packages on my local network (I run my own mirror of that too). I think there’s a way to install Debian packages from source, but I’ve never done it. Also Debian packages install stuff into locations that kinda mix base system stuff and user stuff which makes it annoying to know what I can safely remove without breaking my system. So I decided to try pkgsrc on top of Debian to install packages separate from the base system, and it works very nicely.

I read through both the pkgsrc guide and this pkgsrc mini handbook blogpost to figure out what commands and options I needed. I re-ran the bootstrap process several times and finally landed on what I wanted as my setup.

There’s two main sections to this: how I bootstrap pkgsrc and how I use pkgsrc.

First we need to go over how to setup and bootstrap pkgsrc in the first place.

Feel free to skip this section and move on to Actual bootstrapping.

I previously wrote about using pkgsrc with Clang, and in my experiments with doing so, I may have gone a little overboard on the bootstrapping side of this. I wanted to bootstrap pkgsrc with Clang and then use Clang built with pkgsrc as the default compiler. I even went a step further and wanted to be able to do this even if I only have GCC available as the system compiler.

The details on doing this are probably not relevant to most people, so just as a brief summary, I have several scripts that go through this process:

This is totally crazy, and I actually don’t recommend it myself.

So ignoring the above, lets assume I have Clang installed somewhere (it could be from my separate pkgsrc bootstrap or just installed as a normal Debian package).

First we need to get the pkgsrc code. Again, one my goals to be able

to use this on my local network, so I am actually using my own Git

mirror of pkgsrc (syncing from their Github mirror). So we

first need to clone the repo. Then I personally prefer to work on a

quarterly release branch rather than trunk.

git clone https://github.com/NetBSD/pkgsrc.git

cd pkgsrc

git checkout pkgsrc-2022Q1

Remember where you cloned the code because we’ll need to reference it

later in our mk.conf.

The standard bootstrap process as described in the docs doesn’t do everything I would like. So let’s look at some of the options we can change.

cd bootstrap

./bootstrap --help

Before bootstrapping, as mentioned in my Clang post, you may need to help pkgsrc know where your compiler is.

export CLANGBASE=/usr/lib/llvm-11

export PKGSRC_COMPILER=clang

export CC=$CLANGBASE/bin/clang

export CXX=$CLANGBASE/bin/clang++

Then I define some other variables and do some initial setup:

export PREFIXDIR="${HOME}/pkg"

export MAKE_JOBS=4

export WORKDIR="${PREFIXDIR}/usr/work"

export DISTDIR="${PREFIXDIR}/usr/distfiles"

mkdir -p $DISTDIR

Then this is the actual bootstrap command that I use:

./bootstrap \

--unprivileged \

--compiler clang \

--prefix $PREFIXDIR \

--make-jobs $MAKE_JOBS \

--workdir $WORKDIR \

--mk-fragment "${d}/mk.conf"

Let’s go through these options a bit just for completeness. I want to

install pkgsrc for my local user only with --unprivileged. I want

the bootstrap process to use Clang as the compiler with --compiler clang. I want to install everything into my home directory under

~/pkg with the --prefix flag. I can make better use of my

processor with --make-jobs 4 (feel free to adjust or use something

like $(nproc)). I would like the temporary work dir to be

centralized outside of the pkgsrc directory, so I set --workdir ~/pkg/usr/work. Otherwise you may end up with a bunch of work

directories inside your pkgsrc git mirror directory. Finally I have

--mk-fragment set which will initialize my mk.conf with some

values.

Before I explain more about my mk.conf, there is one last thing my

bootstrap script does. I need to create a symlink to the pkgsrc Git

mirror. This isn’t strictly necessary, but it helped keep my mk.conf

file uniform even if I cloned the pkgsrc code into a different

directory.

My script assumes you’re in the pkgsrc boostrap dir, but otherwise the

PKGSRC_DIR is simply the pkgsrc root directory.

PKGSRC_DIR=$(dirname $(pwd))

PKGSRC_LINK="${PREFIXDIR}/usr/pkgsrc"

rm -f $PKGSRC_LINK

ln -s "${PKGSRC_DIR}" $PKGSRC_LINK

mk.confNow that pkgsrc is bootstrapped, we need to configure it.

In the above bootstrap command, I had a mk.conf fragment

specified. After bootstrapping, the default pkgsrc mk.conf file is

placed in ~/pkg/etc/mk.conf and then this fragment file that I

specified is appended to the end of it.

Here is my mk.conf fragment:

# Separate work and distfiles outside of pkgsrc

WRKOBJDIR= ${LOCALBASE}/usr/work

DISTDIR= ${LOCALBASE}/usr/distfiles

# Set the source dir for pkgsrc itself

PKGSRCDIR= ${LOCALBASE}/usr/pkgsrc

# Use the processor more

MAKE_JOBS= 4

# Use my distfile mirror by default

MASTER_SITE_OVERRIDE= https://mirrors.wisellama.rocks/netbsd/pkgsrc/distfiles/

# Keep partial downloads

PKG_RESUME_TRANSFERS= YES

# Extend acceptable licenses

ACCEPTABLE_LICENSES+= gnu-agpl-v3

ACCEPTABLE_LICENSES+= unrar-license

# Set Python version

PYTHON_VERSION_DEFAULT= 310

# Package-specific options

PKG_OPTIONS.cmus= pulseaudio

PKG_OPTIONS.strawberry= pulseaudio

PKG_OPTIONS.vlc= -lirc

PKG_OPTIONS.retroarch= opengl pulseaudio

PKG_OPTIONS.dolphin-emu= llvm pulseaudio

You’ll notice that the WRKOBJDIR and the DISTDIR line up with the

values we set up during bootstrapping. This way my actual pkgsrc

installation uses them just like the bootstrap script did. Similarly

the PKGSCRDIR is set to the symlink for the pkgsrc Git clone, and

the MAKE_JOBS I have set to 4 again.

Then we start getting into some extra config options.

I wanted my system to be able to download distfiles using my own

mirror. To do that, I specified a MASTER_SITE_OVERRIDE pointing to

my distfile mirror. Without this, pkgsrc will simply download the

distfiles from the project website (e.g. Github releases, sourceforge,

etc.).

The PKG_RESUME_TRANSFERS option has actually screwed me up a few

times, so I would only recommend setting it to YES when using a bad

internet connection.

The ACCEPTABLE_LICENSES stuff is used to allow me to install some

software that isn’t in the default allowed license list. This variable

is probably intended to be used by companies to be able to limit their

licenses for legal reasons.

PYTHON_VERSION_DEFAULT is kinda self-explanatory. You can have

multiple versions of Python installed, so pkgsrc needs to know which

one to use by default when other packages say they depend on Python,

particularly for the build process.

Finally, the PKG_OPTIONS.* settings are specific setting overrides

for individual packages. For example, I’m telling cmus to build with

pulseaudio support.

So, pkgsrc is bootstrapped and the mk.conf is configured. We

can actually use the pkgsrc installation.

I have my .zshrc file configured to add pkgsrc stuff to my PATH

and MANPATH. Alternatively you could always run the binaries

directly like ~/pkg/bin/bmake, but I like the convenience of just

running bmake.

# Add pkgsrc

export PATH="${HOME}/pkg/bin:${HOME}/pkg/sbin:${HOME}/pkg/gnu/bin:${PATH}"

export MANPATH="${HOME}/pkg/man:/usr/share/man:${MANPATH}"

export XDG_DATA_DIRS="${HOME}/pkg/share":"${XDG_DATA_DIRS}"

Finally we can use pkgsrc to install something. Let’s install pfetch

as a simple example. Go back into you pkgsrc Git clone directory and

then run the following:

cd sysutils/pfetch

bmake install

Then on zsh, I have to run rehash to get it to recognize that a

new binary has been added to my path. Then I can run pfetch and see

my system overview.

_____

/ __ \ os Debian GNU/Linux 11 (bullseye)

| / | host 20NL0007US ThinkPad X395

| \___- kernel 5.10.0-13-amd64

-_ uptime 8d 20h 57m

--_ pkgs 4514

memory 3826M / 5915M

Now that we’ve confirmed a simple package is working, we can move on to installing some more complicated things.

One of the main reasons I love pkgsrc is that it allows me to build Firefox and Thunderbird from source very easily. So let’s try installing Firefox (I’m going to do the ESR version, but the normal version is there too).

cd www/firefox-esr

bmake install

Now…Firefox is a beast of a project. To build it from source, pkgsrc

needs to build an install a lot of other projects. When using Clang,

I’ve run into several packages that require a little extra coercion to

build correctly, so you may run into errors. When that happens, pkgsrc

should tell you which package it was trying to build. I then go to

that package’s directory, fix whatever was wrong, finish installing it

with bmake, and then move back to building Firefox again.

Notably for Firefox, I think audio/pulseaudio requires GCC to

build. Also lang/rust requires libgcc. I have some more details

over on my pkgsrc with Clang article

I’ve also run into a problem with Thunderbird where I have to actually change my default Python install back to 3.9 or 3.8 to be able to build it. Then I can switch back.

Building Firefox can take several hours, potentially even all

night. But once it completes, you can launch firefox (in this case

firefox91) and it will launch…and look like crap.

Some programs like Firefox depend on fonts being installed. Even though I’m on Debian with fonts already installed somewhere, pkgsrc likes to act as its own separate system and won’t find my current system fonts. This is both good and bad. Ultimately, I viewed it as a cool good thing that pkgsrc was so separate from the base system that it didn’t even find my system fonts.

Anyway, this was simple to fix after some web searches. I had to install a font and tell Firefox how to use it. Let’s install Noto fonts as an example:

cd fonts/noto-ttf

bmake install

Even then, I couldn’t get some programs to recognize my fonts unless I added a symlink to my home directory.

ln -s ~/pkg/share/fonts/X11 ~/fonts

And still, Firefox had some issues depending on how I launched it. I

finally found a setting in Firefox’s about:config that let me use

these fonts correctly, although I suppose this technically degrades

the security of Firefox slightly:

security.sandbox.content.level = 2

Then I finally had a working version of Firefox that I had built from source with pkgsrc. I don’t think I can properly describe just how happy this made me.

pkg_alternatives)Just like how Debian has an alternatives system, so does pkgsrc. This is a simple script that lets you specify which version of a package should be used as the default in case you have multiple versions installed. First we need to install it because it is itself a package:

cd pkgtools/pkg_alternatives

bmake install

Now we can set the default version for something like Python for example:

pkg_alternatives manual python310

Note that this only works for packages that specifically have an alternative listed. You can see those options with

pkg_alternatives list

pkg_rolling-replace)So you’ve been using pkgsrc for a while and maybe a new quarterly

release has come out. How do you upgrade all of your pkgsrc packages?

Well, one of the pkgtools is called pkg_rolling-replace. This

package will determine which packages are out-of-date, build a

dependency tree to know which packages to upgrade first, and then go

through each package running bmake replace for them.

Remember way back in our mk.conf where we defined the PKGSRCDIR

variable? This is where it’s finally used. pkg_rolling-replace will

check the pkgsrc code to determine if your installed packages are out

of date. So first of all, make sure you update the pkgsrc code

repo. This could just be doing a git pull for trunk, but for a

quarterly release you need to switch to the new branch.

cd pkgsrc

git fetch -p

git checkout pkgsrc-2022Q2

Then pkg_rolling-replace needs to be installed:

cd pkgtools/pkg_rolling-replace

bmake install

I tend to run pkg_rolling-replace with some extra flags. I forget

where I found these as recommended, but they work well for me.

pkg_rolling-replace -rusv

Similar to the warning I gave for Firefox, sometimes packages will

have errors while upgrading too. When that happens, I go into that

package directory, fix whatever was wrong, manually run bmake replace, and then continue running the pkg_rolling-replace replace.

Very rarely, I may need to completely uninstall and reinstall a

package to upgrade it. To remove a package, I used pkg_delete <package_name>. Then I can run bmake install to install it again.

Some package tools are installed by default including the pkg_delete

I mentioned above to uninstall packages. There’s also pkg_info which

I very useful. It’s very similar to apt show where it will give you

the details of an installed package.

And that’s it. That’s how I’m using pkgsrc. I bootstrap it crazily to make sure it was built with Clang. I set it up to use its own version of Clang as the default. I tend to use the quarterly release branches and update each quarter.

I’ve been trying to run as much software as I can using pkgsrc. Some stuff like my desktop environment doesn’t make sense because pkgsrc really only as that stuff set up to support NetBSD. However, everything else that’s like userland-level software I’ve tried to build and install with pkgsrc.

I’m writing this blog post in emacs running in tmux and zsh, all built with pkgsrc. I’m testing it with hugo and firefox, built with pkgsrc. I’m listening to music in cmus and checking my email with thunderbird, built with pkgsrc. You get the idea. I’m using pkgsrc for everything that I can, a bit inspired by this blog post from Rubenerd.

Pkgsrc scratched an itch I’ve had for a while. I’ve always been a free and open source software advocate (since I was like 12), but pkgsrc seemed like a culmination of that philosophy into a fantastic project. I remember an old friend teasing me about open source software, asking if I’ve actually inspected the source code and built it myself. And now with pkgsrc, I can finally answer yes, I have.