Re: What is BSD make?

[Aleksej Saushev <asau%inbox.ru@localhost>]

Peter Bex <Peter.Bex%xs4all.nl@localhost> writes:

> On Sat, Oct 11, 2008 at 03:29:02PM +0300, Aleksey Cheusov wrote:
>> After years of development using GNU make under Linux I switched my
>> home workstation to NetBSD. Here I've found a number of interesting
>> things. One of them is NetBSD make.
>> 
>> 3) In my view bsd.*.mk scripts is a main thing about BSD make. It is
>>    really amazing thing and may be much more important than make itself.
>>    I believe that MK scripts with _some extensions and additions_
>>    may be excellent replacement for GNU make + automake + libtool.
>> 
>>    I really love declarative nature in writing Makefiles, I like this
>>    approach and think that things like automake/libtool must die.
>
> I agree 100%. The Makefiles are short, elegant and the system itself
> does not require so much overhead as most of the GNU tools do. In fact,
> it's already installed on all my systems :)
>
>>    The question is (rhetorical again) why BSD makes are used mainly to
>>    build respective *BSD systems and not used by software developers
>>    outside *BSD. That is, why BSD make and especially MK scripts are
>>    promoted so badly :-(
>
> One thing that might be lacking is good documentation; I've found almost
> no documentation about the .mk system.  I wrote down a few notes and
> put those on the wiki some time ago: http://wiki.netbsd.se/BSD_Make
> and http://wiki.netbsd.se/Basic_Unix_programming#Using_BSD_Make
> but those are just some quick incoherent notes; a poor substitute for
> real documentation.
>
> I think the biggest problem with BSD make currently is that it's very
> unportable. There are dozens of variants out there, all with slight
> incompatibilities. If you distribute a Real Program that uses BSD Make
> with the .mk files as its build system, it will only build on the BSD
> that you wrote it on. Since each of the individual BSDs are by
> themselves relatively small players on the Unix market, this is
> unrealistic for a large program that should run on many Unix systems.
> A coordinated effort to make The One BSD Make to Rule Them All could
> solve this problem, but I'm not sure if that's doable since the
> rulesets are (AFAIK) pretty system-specific.
>
> For these reasons, I only use BSD Make for small personal projects that
> are not released. I also use LaTeX-mk (in pkgsrc as print/latex-mk) for
> my LaTeX projects, which adds LaTeX-specific rules to BSD Make (but
> also includes a gmake ruleset). I'd love to be able to use BSD Make for
> other projects as well.

There was conversation between us on #netbsd @OPN.
I think that a summary or even  of it might be useful.

I use BSD make for work, and I'd like to share some experience.

At work I use FreeBSD with bmake, which comes with pkgsrc.
Since I'm accustomed to bmake so much (and I find it superior),
I frequently link it to ~/bin/make, thus "make" on my FreeBSD
system calls NetBSD make, not the native one.
I also worked in cooperation with linux-oriented developers,
who wrote their own (GNU) Makefiles, and I saw how they work.
Thus I had chances to compare different approaches.

At first, I should point to mistake I meet, that GNU make doesn't
provide default way to build and link C programs like BSD make
does with its <sys.mk>.

GNU make does provide default suffix rules for ".c.o" and such,
they are just built in, hardcoded, and you can't go and simply
change them by means of editing text file. You can easily prove
it, just write "hello world" program, call it "hw.c", and type
"gmake hw".

Since these rules do exist, you _can_ write GNU makefiles in a
declarative (mostly) way. And linux-oriented developers do it,
their usual makefiles look like:

OBJS= this.o that.o main.o
all:
        cc -o prog ${OBJS}
clean:
        rm -f prog ${OBJS} *~

Compare that to usual BSD makefile from /usr/src/usr.bin, and
you'll see only minor differences: OBJS vs. SRCS, explicit "all"
and "clean" targets vs. CLEANFILES (if any). All differences
come mainly from another source.

Now let's turn to BSD make. What makes it different?
I see following major differences:
1. sys.mk and ability to change or extend pre-defined rules globally.
2. Existence of standard path where to search files with make rules
with good and (almost) comprehensible set of predefined files
(all <bsd.*.mk> files).
3. Global configuration file (NetBSD mk.conf, FreeBSD make.conf),
which is configurable itself (MAKECONF in NetBSD, MAKE_CONF or
like in FreeBSD).
4. Makefile.inc
5. .depend
6. .OBJDIR
7. ".for", "!=" and various variable modifiers.


I'd like to present more elaborate comments on listed above.

1. sys.mk

It is very nice to have one place, where you can change
usual way of running things. It isn't known well enough,
that you can override /usr/share/mk path with your own.
Search "MAKESYSPATH" in manual page for details.
It would be nice to see this explained in "ENVIRONMENT"
section. At least it would be useful to note, that it is
"used to search files to .include, /usr/share/mk by default".
In addition to that "FILES" section should be expanded to note,
that sys.mk placement may be overridden by environment variable.

2. <bsd.*.mk> scripts.

This is the most frequently used and powerful part of BSD makes,
and this makes BSD make superior to GNU make, even given restriction,
that you can't easily build library and program in the same directory.
If the distinction is dissolved in case of simple programs, when it
comes to libraries, it is the simplest way to build and install them.
When I defined LIB variable and wrote ".include <bsd.lib.mk>",
my linux colleagues had to write explicit rules to run "ar",
"ranlib" and other tools.

On the other hand, there's much inconsistency and incompatibility
between different BSDs and even within particular system: NOMAN vs.
NO_MAN, NOPROFILE vs MKPROFILE, NOSHARED vs. MKPIC.  Most of these
variables are not documented, though they are definitly useful,
sometimes you simply don't have a way to override particular
behaviour inside rather complex chain.  FreeBSD frequently checks
that variable is just defined, NetBSD frequently checks its value
to be "[Yy][Ee][Ss]" or "[Nn][Oo]". Even clarifying and documenting
this part is quite a large task.

3. mk.conf/make.conf

It would be very nice, if the existence of MAKECONF was documented
in mk.conf(5), just put a simple note in "FILES", that it
"can be overriden by setting MAKECONF variable", reading bsd.README
is a bit larger undertake, it shouldn't be needed, if you're building
separate customized distribution without changing source tree.

4. Makefile.inc

I don't understand, why the path is "${.CURDIR}/../Makefile.inc"
(see bsd.init.mk).  I'd definitly like to see the path relative
to ${.PARSEDIR}, it simplifies many things.

5. .depend

This would be nice to reuse, though it isn't obvious to me,
which way. I have a feeling, that this can cover some things,
which autoconf tries to do. Unfortunatly, there're no publicized
examples of how to do it.

6. .OBJDIR

This is the most sophisticated part. Noone knows how it works exactly,
as some recent posts suggest, not many developers know of its existence
either, some don't even understand the purpose of it.

In short, there is a concept of object directory, the directory,
where all actual work is done. If you don't make additional steps,
object directory is the same as current directory, but this can
be changed: you may choose to put all object and target files
into "obj" subdirectory or some another directory. Now you know,
that this makes you keep your source tree clean. You can even
run separate builds from the same tree and your builds won't
stomp on each other. That is what is called "separate build
directory" in GNU world.

Contrary to GNU world, which only starts to discover it,
BSD world has it already, but it is still not well-known.
I see three reasons for that: a) it is almost undocumented;
b) it requires special handling; c) the handling is rather
sophisticated.

To address those problems, I see it necessary to clarify,
probably in LARGE FONT, that BEFORE make starts the work,
it CHANGES working directory to OBJECT one, which DIFFERS from
SOURCE directory in general case. Thus, in general all target
files are placed into another directory, which is defined
by some rules (currently not described well), and which can be
defined using ".OBJDIR: /path/to/there" directive.
That object directory is the same as source one, is a mere
coincidence, and properly written makefile should allow
changing it.

Hence, it is developer's task to either use predefined rules,
which handle .OBJDIR correctly, or refer properly to source
files, which are "${.CURDIR}/source.c" when written explicitly,
because "this.o" refers to OBJDIR, not to CURDIR, where you are,
when you type "make" in your shell.

This issue is rather hard to explain, at least I met some
resistance, when I explained, why those makefiles were wrong.

Object directory also complicates linking.
You cant simply write:

LDADD+= ../../lib/this/this.a
DPADD+= ../../lib/this/this.a

You have to:

# find object directory first:
LIBTHISOBJ!= cd ${.CURDIR}/../../lib/this && ${PRINTOBJDIR}
# link to library
LDADD+= ${LIBTHISOBJ}/libthis.a
DPADD+= ${LIBTHISOBJ}/libthis.a
# build library, if it doesn't exist:
${LIBTHISOBJ}/this.a:
        cd ${.CURDIR}/../../lib/this && ${MAKE} libthis.a
# NB! Flags should be passed along, but it isn't easy.

Now we meet technical problems:
1. This is too much to type, and even Makefile.inc,
which can provide all necessary information, is to be written
manually. It is obvious, that it is much repetitive work,
which could be done in bsd.lib.mk
2. You have to track dependencies manually, unless you're using
pkgsrc and building package.
3. Recursive makes.

I don't have ready answers to any of those yet.

There should be some way to use .PARSEDIR variable to address
dependency tracking, but this is non-portable: FreeBSD make
doesn't have it, and this causes my head ache.

There should be some way to avoid repetitive typing of LDADD/DPADD
stuff, but I haven't tried to write any code yet. It is rather hard,
and I run into incompatibilities: FreeBSD make lacks many variable
modifiers present in NetBSD make.

I have absolutely no idea how to handle recursive makes right now.

To sweeten this a bit I should add that my linux colleagues didn't
even try to address OBJDIR.

GNU world seems to lack good understanding of it.
What they provide is separate build directory within autoconf
framework: you create directory yourself, chdir there, run
configure script, which generates Makefiles in your build
directory.

I see following drawbacks of this approach:
a) autoconf is required to work this way;
b) even if you grok "a", your current directory is build directory,
not the source directory, where you work;
c) you can't use different objdir selection scheme for different
parts of your tree (I like to have "obj" subdirectory in my working
directory, while I don't care of the rest tree).

BSD OBJDIR approach is superior.


7. ".for", "!=" and various variable modifiers.

There's not much to comment here.

There is some controversy about "!=" and other run-shell-command
facilities, some developers consider it harmful. On the other hand
they provide autoconf-like (restricted) functionality, when staying
inside usual framework (make).


I hope that these comments are useful to improve our own make.

As for "one make to rule them all", I have only one thing to say.
I don't see any evidence that FreeBSD people use or try to use
their make anywhere outside FreeBSD itself and ports collection.
FreeBSD make lacks many convenient things, which are present in
NetBSD make. Given my experience with FreeBSD, I don't think that
FreeBSD is going to adopt NetBSD make or import any changes from
it. I don't even think they're going to improve it at all. I may
be wrong here, and I'd like to be wrong, but that's my opinion.

NetBSD make is definitly superior. It is more reasonable to fix
FreeBSD-specifics in NetBSD scripts and use it on FreeBSD, than
to use FreeBSD make. The hardest part here is convincing your
colleagues.


-- 
HE CE3OH...