The problem is NOT with not polling the pipe or the sleep(1).
The committed patch differs from the original I tested (and
which worked great) in a crucial detail:
- if (count < 0 && jobTokensRunning != 0) {
+ if (count < 0) {
This change alone on top of stock bmake causes very serious stalls.
In fact, with this change what is normally a 50 second run for me is
still stuck after 540 seconds.
Take a look at stock bmake (no poll changes and consider what happens
in Job_TokenWithdraw if jobTokensRunning is 0 and the read returns -1:
count = read(tokenWaitJob.inPipe, &tok, 1);
if (count == 0) /* NOT TAKEN */
if (count < 0 && jobTokensRunning != 0) { /* NOT TAKEN */
if (count == 1 && tok != '+') { /* NOT TAKEN */
if (count == 1 && jobTokensRunning == 0) /* NOT TAKEN */
/* This one *IS* executed */
jobTokensRunning++;
return TRUE;
If you take the stock bmake and make the aforementioned change,
the following happens instead;
if (count < 0) { /* TAKEN */
[..]
wantToken = 1;
return FALSE;
}
The function returns FALSE instead of TRUE and does not bump the
jobTokensRunning counter either and this is where bmake experiences
super slow times.
That said, here is a short time table for with sample times on FreeBSD:
P(OLLTOKEN) - whether the token pipe is being polled
C(HECKRUNNING) - whether the jobTokensRunning check is used
R(EADSLEEP) - whether sleep(1) is used after failed read
So in particular 1 1 0 is the stock bmake (as seen after the revert).
make -s -j 128 buildkernel
P C R
0 0 0: did not finish in 120 seconds, aborted by hand
0 0 1: did not finish in 120 seconds, aborted by hand
0 1 0: 3285.16s user 720.16s system 8499% cpu 47.124 total
0 1 1: 3193.32s user 648.69s system 7881% cpu 48.746 total
1 0 0: did not finish in 120 seconds, aborted by hand
1 0 1: did not finish in 120 seconds, aborted by hand
1 1 0: 3377.30s user 726.73s system 8096% cpu 50.690 total
1 1 1: 3198.32s user 490.50s system 7584% cpu 48.634 total
I can post a longer benchmark with *significantly* higher wins
later (don't have the numbers handy for the original run
unfortunately).
The original issue is bmake forking itself *a lot* (hundreds of
processes) all polling *one* pipe. They would first contend on
the pipe mutex to see if there is anything to read and since
for the majority this is not the case, they would go to sleep.
Then any write wakes them *all* up and they beat each other again.
Since only few can read anything in the first place, the majority
goes back to sleep. And so on.
This is a catastrophic slowdown.
The kernel has no choice but to wake everyone waiting on this pipe up
since it does not know what they are going to do (e.g. what if they
don't read anything anyway).
This behaviour can be made significantly less slow in the kernel, but
I don't think this is worth working on. No matter what, this is still
going to be significanlyt slower than not polling the pipe with such
high concurrency and not forking that much to begin with.
Just not polling is a workaround which is good enough for the time
being imo. Most of the issue can be avoided with a switch to kqueue.
NetBSD also has the problem, but it may be hidden by other issues.