On June 12, I sent a call for donations to the netbsd-advocacy mailing
list.  This was widely publicized, and our community of donors
responded extremely generously.  Over the next few months, we received
almost $30,000 of donations, including a number of donations of
several thousand dollars each.

As a project, we believe in transparency.  When you donate to The
NetBSD Foundation, we want to be sure you know where your money is
going.  It's not sitting in some kind of "slush fund" and it's not
hidden under anyone's mattress.

We spend the money we receive to facilitate NetBSD development and
distribution.  We want you to know how those dollars are spent.

Shortly after we posted the call for donations, we posted a list of
configurations of the systems we would buy with the money raised in
the fundraising drive.  Because we raised more money than we expected,
we bought more machines, and slightly adjusted the configurations.

WHAT DID WE BUY?

We bought three new systems for the autobuild cluster: dual dual-core
Opterons (four processors total) with 4GB of memory per system, and
dual 74GB disks on each system.

The systems we originally specified had single-core processors.
During the fundraising drive, dual-socket, dual-core Opteron systems
became widely available.  Therefore, we purchased only three physical
boxes -- but we purchased a total of twelve CPUs.  This purchase
quadruples the number of CPUs in our build cluster.  The new CPUs are
also slightly faster per-core than the original CPUs.  This is an
enormous increase in resources for our autobuild system.

We bought two new systems to replace the current AnonCVS server.
These systems have dual single-core Opteron processors (so two
processors total), 8GB of memory per system, and four 36GB disks on
each system.

We also bought a managed network switch, an APC remote-controlled
power distribution unit, and a console server, which is an Acme Micro
FA1426 1/2U server with two usbgear.com USBG-4X232 multi-port USB
serial cables.

HOW MUCH DID WE SPEND?

All told, a little less than $20,000.  We have retained the rest of
your donations to cover future hardware needs, as well as Foundation
overhead.

We could have bought more autobuild hosts, but thought it more
responsible to retain some money in case of urgent future needs --
which always seem to occur, and which developers have often had to
cover from their own pockets in the past.

WHAT TOOK SO LONG?

While preparing to install the new systems, we ran into a power and
cooling issue at our main colocation facility.  This caused a long and
unpleasant delay which was very frustrating for all involved.

We ultimately concluded that we should move the autobuild cluster to
a new facility.  With the generous assistance of Professor Angelos
Keromytis, Dr. John Ioannidis, and Professor Steve Bellovin of
Columbia University's Department of Computer Science, we were able to
bring the new build systems online at Columbia in mid-September.

This required duplicating a significant amount of infrastructure from
our existing colocation facility: remote power switching, secure
console access, and a private network.  Getting all the pieces in
place and configured took well over a month even once the space at
Columbia was secured.

Finally, we ran into a networking problem.  Our build cluster uses
multiple network segments.  With half the cluster in California and
half in New York -- but only until we could relocate the original two
cluster nodes -- the simplest solution seemed to be to bridge the
private network segments across an IPsec tunnel.  Unfortunately, this
caused a performance issue due to a previously undiscovered livelock
condition when bridging from a busy Ethernet to an etherip tunnel, a
problem which was made worse by the overhead of IPsec processing.

We solved the network performance issue by reconfiguring the network
at each end of the New York to California tunnel to avoid bridging
across the tunnel.  Eventually we will move the original two build
cluster nodes to Columbia and eliminate the tunnel entirely.

SO WHERE ARE WE NOW?

The autobuild cluster now builds every major active branch of NetBSD,
for every supported architecture, approximately every 30 hours.  This
means that no matter which release you're tracking -- or even if
you track NetBSD-current -- there will be fresh binaries for you
almost every day.  We anticipate speeding this up somewhat through
enhancements to the autobuild framework itself and hope to achieve our
goal of building every branch for every architecture within 24 hours.

The new, faster continuous-build system has already helped us find
and fix a number of bugs that caused build problems, thus sparing
our users the frustration of downloading the entire system only to
discover that it would not build.

By the end of the year, we will move the two original build cluster
nodes to Columbia.  This will make space, power, and cooling avaiable
at our original facility so that we can deploy the two new AnonCVS
servers there.  We know that the current AnonCVS service could be
a lot better, and we ask for your patience -- we promise, help is
coming.  We apologize for the delays caused by our need to shuffle
equipment between locations, but we know that the result will be
better and more reliable service for everyone and we believe it's
worth the temporary inconvenience.

WHO TO THANK?

First of all, everyone who responded so generously to our call for
donations.  You did the real work -- we're just reaping the benefits
of your decision to give.

Christos Zoulas and Lex Wennmacher coordinated the receipt of a huge
number of donations with little prior warning.  They accomodated many
donors who needed to make special arrangements to transmit funds.
The sheer quantity of donations pushed us over a state reporting
threshold, forcing Christos to fill out a number of obscure and
complicated tax forms.  This is vital work for the future of the
project and the contributions of the people who do it are often
overlooked.

James Chacon and the rest of the Release Engineering team did major
rework on the build system to accomodate the new machines and put up
with many midnight reboots during the shaking-out process.

Eric Haszlakiewicz sat patiently in an overheated colo facility for
almost two days while we rewired, moved hardware, and tried to resolve
the power and cooling issues that ultimately forced us to move the
build cluster to New York.

Jan Schaumann, Brian Moriarty, Michael Forde, Oliver Gould, and Hubert
Feyrer at the Stevens Institute of Technology provided temporary space
for configuration and testing of the new machines, and assembled the
rather mechanically perplexing rack rail kits.

Professor Angelos Keromytis, Dr. John Ioannidis, and Professor Steve
Bellovin at Columbia provided space and network connectivity for the
build cluster.  Dr. Ioannidis personally spent several hours helping
rack and wire the new machines.  Angelos Stavrou, also of the Columbia
CS Department, helped arrange access to the systems while we debugged
the network stack issue mentioned above.  We are tremendously grateful
to Columbia, its Computer Science Department, and these researchers
for the help they have provided, and continue to provide, to NetBSD.

Finally, the NetBSD admin team gave generously of its time, spending
tens and possibly hundreds of hours -- so much time, it's hard to
keep track -- ordering the new systems, reconfiguring our existing
hardware and software systems to accomodate them, and extending our
administrative "footprint" across the continent to New York to allow
lights-out management of all of our systems, regardless of location.

And once more, all of you, the NetBSD community, should give
yourselves a hand: your donations made this possible.  We're doing our
best to make you glad that you gave, and hope that you will give again
in the future.  Your contributions make a better, more robust NetBSD a
reality for everyone.

DID WE THANK YOU IN PERSON?

Our policy is to give each donor a receipt for their donation and
optionally acknowledge the donation on our website:
 
    http://www.netbsd.org/donations

If you have not yet received a receipt for your donation (possibly
because you donated via check and we could not determine your e-mail
address), please send mail to finance-exec@netbsd.org with your name,
the donation amount, and date so that we can send you a receipt.