Imported source, pkgsrc on Arm woes, and make includes

2020-08-01

I think we'll start this post discussing some of the differences and misconceptions of same between BSD make and GNU make. In the case of BSD make we're primarily referring to NetBSD make, which is a greatly improved version of the historical Unix make utility, and has also been imported into FreeBSD. Among the BSD camp, GNU make is seen as a second citizen. Their version of make is leveraged for building the entire system, without using autotools except in the case of imported external source which has been autotooled by it's authors. It is also the utility that wraps all of the functionality of FreeBSD ports and NetBSD pkgsrc - downloading, patching, compiling and packaging third party software. The complete infrastructure is quite impressive.

Much of this functionality lies in leveraging included Makefiles, much in the way that C headers can be included to pull in function definitions, and also similar to the concept of shared libraries. These included .mk files, usually located in /usr/share/mk on a BSD system, allow for extremely concise Makefiles throughout the source tree to just specify a few variables and then pull in all of the targets from an inclued file. The most prominent are probably bsd.prog.mk and bsd.lib.mk, which tell make how to build C programs and C libraries respectively.

It is a common misconception that this functionality only exists in BSD make. GNU make does have this capability as well and I leverage it quite heavily in HitchHiker. Admittedly, it is more developed in BSD make, but the limitations of GNU make in comparison are fairly trivial and easy to work around. One of the sillier things is the default search path that GNU make uses the look for included Makefiles. By default this path starts in the current directory and proceeds through /usr/local/include, /usr/include and finally /usr/gnu/include. Leveraging this would sprinkle .mk files through what are by convention the location of C header files, or alternatively using /usr/gnu/include. I have yet to see a system that has a /usr/gnu directory (maybe it was planned for Hurd?) and I'm not keen on it. As of the last commit, HitchHiker now patches make to no longer search /usr/gnu/include and to now search instead /usr/local/include/mk and /usr/include/mk, the latter being our new default directory. This allows us to include a .mk file by name, without specifying a full or relative path, and at the same time keeps all of our .mk files separated from the system C headers.

So what exactly does BSD make have over GNU make?

• Slightly more powerful conditionals. In addition to ifdef, ifndef, ifeq and ifneq BSD make also has a simple if, and allows chaining conditionals with the && and || operators much like shell syntax. By comparison GNU make must nest conditionals for similar functionality.
• Included Makefiles can be relative to the install or source prefix. This is quite powerful for building large projects in custom site locations.
• Possibly others I have not discovered yet, however in practice GNU make can do -everything- that BSD make does if you take the time to learn how to use it.

As part of creating a more integrated build framework, my most recent commit contains a basic analog for bsd.prog.mk, hhl.cprog.mk (cprog due to my not assuming that all programs are written in C!) which now allows extremely concise Makefiles for source code that has been imported into the base system. In the case of rpcgen the following Makefile is all that is needed:

# Makefile - hhl - /usr/src/world/rpcgen
# Copyright 2020 Nathan Fisher <nfisher.sr@gmail.com>
#
progname = rpcgen
cflags += -g
include hhl.cprog.mk

As can be seen, there are really only three lines. Lines 1-3 are comments, line 4 specifies the program name, line 5 adds -g to the compiler flags, and line 6 includes hhl.cprog.mk. If you look inside the rpcgen subdirectory of the build tree you will see the following structure:

.
  ├── Makefile
  ├── man
  │   └── rpcgen.1
  ├── README
  └── src
    ├── config.h
    ├── proto.h
    ├── rpc_clntout.c
    ├──***snip

Our included Makefile will generate object files for any .c files in the src/ subdirectory, link those objects into an executable and install man/rpcgen.1 in ${install_prefix}/share/man/man1. It has logic to differentiate man1 from man8 and also installs html and text docs in the doc/ subdirectory. If we needed to specify additional cppflags, cflags, ldflags or libs to link with we would just append them to those vars (by convention I use lower case so as to not interfere with any CFLAGS etc. specified on the command line or in an external build system). If we want to create links to our executable they just need to be specified in the binlinks variable. Defining the variable finish and writing a finish target allows us to do anything else additional that might need done, like installing configuration files or runtime data. In short, while being a fairly tidy 102 lines (some of which is whitespace and comments) hhl.cprog.mk successfully works as an analog for bsd.prog.mk.

The short list of programs leveraging this framework now include the ee text editor, rpcgen, mksh and lksh shells and the pax archive/copy/link utility. At some later date the source tree will be reorganized to move these directories into bin or usr.bin much like is done in the BSD source trees, as well as importing other sources directly into the tree and "HitchHikerizing" their build systems.

The title promised some pkgsrc updates relating to arm challenges. In a nutshell pkgsrc seems to be broken in several places for arm on Linux. Perhaps most annoyingly is the buildlink system, which is supposed to make it simpler for packages to find the libs that they are to link against by moving everything into work/.buildlink/lib (relative to the subdirectory that bmake was run from). So far I've discovered that libuuid, libexpat, libstdc++, libacl and libattr libtool archives are skipped every time on arm for reasons that I have not yet discovered. I can work around it by manually linking the .la files in from /usr/lib, but this requires an annoying level of manual intervention. Nevertheless I have managed to build the full LXDE, XFCE and Mate desktops, several shells, emacs, nano and pico editors, Inkscape and am on to building Gimp presently on my RPI. On x86_64 I have all of that plus a fairly complete server stack, Abiword, Gnumeric and the Firefox browser built.

In short, when q3 releases (and it really is coming soon) there will be a fairly comprehensive selection of software available on both architectures via pkgsrc. However, in the long run we're going to revisit the idea of creating our own ports tree which leverages GNU make and the infrastructure of .mk files that now reside in /usr/include/mk, adding funcionality for creating binary package tarballs and downloading and installing them. This will most likely involve a simple homebrewed package format that takes some inspiration from BSD tarballs, Slackware tarballs and Arch Linux tarballs. A sneak peak in the current build tree at how we register the installation of the base system will give you an idea. We're going to keep things purposely simple and avoid writing a full fledged package manager. More details to come.

Tags for this post: