On 8/30/06, Charles M. Hannum <mycroft@mit.edu> wrote:
> The NetBSD Project has stagnated to the point of irrelevance.  It has
> gotten to the point that being associated with the project is often
> more of a liability than an asset.  I will attempt to explain how this
> happened, what the current state of affairs is, and what needs to be
> done to attempt to fix the situation.
>
> As one of the 4 originators of NetBSD, I am in a fairly unique position.
> I am the only one who has continuously participated and/or watched the
> project over its entire history.  Many changes have taken place, and at
> the same time many things have remained the same -- including some of
> our early mistakes.
>
> I'd like to say that I'm some great visionary, who foresaw the whole OSS
> market, but the fact is that's BS.  When we started the project, Linux
> and 386BSD were both little hobbyist systems, both pretty buggy, and
> both lacking a lot of important hardware support.  Mostly we were
> scratching an itch: there was no complete package of 386BSD plus the
> necessary patches to make it run on more systems and fix bugs, and there
> was no sign that Bill Jolitz was going to resurface and do anything.
>
> Much of the project structure evolved because of problems we had early
> on.  Probably our best choice was to start using central version control
> right off; this has enabled a very wide view of the code history and
> (eventually) made remote collaboration with a large number of developers
> much easier.  Some other things we fudged; e.g. Chris got tired of being
> the point man for everything, and was trying to graduate college, so we
> created an internal "cabal" for managing the project, which became known
> as the "core group".  Although the web was very new, we set up a web
> site fairly early, to disseminate information about the project and our
> releases.
>
> Much of this early structure (CVS, web site, cabal, etc.) was copied
> verbatim by other open source (this term not being in wide use yet)
> projects -- even the form of the project name and the term "core".  This
> later became a kind of standard template for starting up an open source
> project.
>
> Unfortunately, we made some mistakes here.  As we've seen over the
> years, one of the great successes of Linux was that it had a strong
> leader, who set goals and directions, and was able to get people to do
> what he wanted -- or find someone else to do it.  This latter part is
> also a key element; there was no sense that anyone else "owned" a piece
> of Linux (although de facto "ownership" has happened in some parts); if
> you didn't produce, Linus would use someone else's code.  If you wanted
> people to use your stuff, you had to keep moving.
>
> NetBSD did not have this.  Partly due to lack of people, and partly due
> to a more corporate mentality, projects were often "locked".  One person
> would say they were working on a project, and everyone else would be
> told to refer to them.  Often these projects stagnated, or never
> progressed at all.  If they did, the motivators were often very slow.
> As a result, many important projects have moved at a glacial pace, or
> never materialized at all.
>
> I'm sorry to say that I helped create this problem, and that most of the
> projects which modeled themselves after NetBSD (probably due to its high
> popularity in 1993 and 1994) have suffered similar problems.  FreeBSD
> and XFree86, for example, have both forked successor projects (Dragonfly
> and X.org) for very similar reasons.
>
> Unfortunately, these problems still exist in the NetBSD project today,
> and nothing is being done to fix them.
>
> --
>
> I won't attempt to pin blame on any specific people for this, except to
> say that some of it is definitely my fault.  It's only in retrospect
> that I see so clearly the need for a very strong leader.  Had I pursued
> it 10 years ago, things might be very different.  Such is life.  But
> let's talk about the situation today.
>
> Today, the project is run by a different cabal.  This is the result of a
> coup that took place in 2000-2001, in which The NetBSD Foundation was
> taken over by a fraudulent change of the board of directors.  (Note:
> It's probably too late for me to pursue any legal remedy for this,
> unfortunately.)  Although "The NetBSD Project" and "The NetBSD
> Foundation" were intended from the start to be separate entities -- the
> latter supplying support infrastructure for the former -- this
> distinction has been actively blurred since, so that the current "board"
> of TNF has rather tight control over many aspects of TNP.
>
> Were TNF comprised of a good set of leaders, this situation might be
> somewhat acceptable -- though certainly not ideal.  The problem is,
> there are really no leaders at this point.  "Goals" for releases are not
> based on customer feedback or looking forward to future needs, but
> solely on the basis of what looks like it's bubbled up enough that it
> might be possible to finish in time.  There is no high-level direction;
> if you ask "what about the problems with threads" or "will there be a
> flash-friendly file system", the best you'll get is "we'd love to have
> both" -- but no work is done to recruit people to code these things, or
> encourage existing developers to work on them.
>
> This vacuum has contributed materially to the project's current
> stagnation.  Indeed, NetBSD is very far behind on a plethora of very
> important projects.  Threading doesn't really work across multiple CPUs
> -- and is even somewhat buggy on one CPU.  There is no good flash file
> system.  There is no file system journaling (except for LFS, which is
> still somewhat experimental).  Although there's been some recent work on
> suspend support, it's still mostly broken.  Power management is very
> primitive.  Etc.  Even new hardware support is generally not being
> originated in NetBSD any more; it's being developed by FreeBSD and
> OpenBSD, and being picked up later.  (I think the only recent exception
> to this of any significance is Bluetooth support.)
>
> For these reasons and others, the project has fallen almost to the point
> of irrelevance.  (Some people will probably argue that it's beyond that
> point, but I'm trying to be generous.)  This is unfortunate, especially
> since NetBSD usage -- especially in the embedded space -- was growing at
> a good rate in 2000 and 2001, prior to the aforementioned coup.
>
> --
>
> At this point most readers are probably wondering whether I'm just
> writing a eulogy for the NetBSD project.  In some ways, I am -- it's
> clear that the project, as it currently exists, has no future.  It will
> continue to fall further behind, and to become even less relevant.  This
> is a sad conclusion to a project that had such bright prospects when it
> started.
>
> I admit that I may be wrong about this, but I assume that most people
> who have contributed to NetBSD, and/or continue to do so, do not desire
> to see the project wallow away like this.  So I will outline what I
> think is the only way out:
>
> 1) There must be a strong leadership, and it is not the current one.
>    The leadership must honestly want NetBSD to be a premier, world class
>    system with leading edge features.  The leadership must set
>    aggressive goals, and actively recruit people to make them happen.
>
> 2) There must be no more "locking" of projects.  Just because one person
>    is supposedly working on a problem, that doesn't mean you shouldn't.
>    If there ideas are dumb, or even just suboptimal, do it better!  If
>    there is no progress, hop on it.  Don't wait for someone else.
>
> 3) The project must become an *actual* meritocracy, not what I call a
>    "volumetocracy".  Right now, the people who exert the most influence
>    are often the people who produce the least useful product.  Indeed,
>    they are often people who produce little more than fluff (e.g.
>    changing line-ending whitespace!), and often break things.
>
> 4) Speaking of which, there must be negative feedback to discourage
>    people from breaking stuff.  This has been a continual problem with
>    certain "developers" for more than a decade.
>
> 5) There are a number of aspects of the NetBSD architecture that are
>    flat out broken, and need serious rehabilitation.  Again, the
>    leadership needs to recruit people to do these things.  Some of them
>    include:
>
>    * serious problems with the threading architecture (including the
>      user-kernel interface), as mentioned earlier;
>    * terrible support for kernel modules;
>    * the horrible mess that is 32/64-bit compatibility, resulting in
>      32-bit apps often not working right on 64-bit kernels; and
>    * unbounded maintenance work due to inappropriate and rampant use of
>      "quirk" tables and chip-specific tables; e.g. in SCSI, ATAPI, IDE,
>      ACPI and SpeedStep support.  (I actually did much of this work for
>      SCSI, but am not currently able to commit it.)
>
> 6) The existing NetBSD Foundation must be disbanded, and replaced with
>    an organization that fulfills its original purpose: to merely handle
>    administrative issues, and not to manage day-to-day affairs.  The
>    extra committees, which mostly do nothing, must be disbanded -- they
>    serve only to obfuscate things.  Everything else must revert to the
>    historically separate entity, the NetBSD Project, to be managed based
>    on technical merits.  There must be no perceived glamour in
>    participating in the Foundation; it must be composed of people doing
>    it because they are dedicated and want to help the project.
>
>    (I will note here that this is not due to bitterness over the coup.
>    Keeping the NetBSD Project as an unincorporated association actually
>    helps protect it.)
>
> 7) The "core" group must be replaced with people who are actually
>    competent and dedicated enough to review proposals, accept feedback,
>    and make good decisions.  More to the point, though, the "core" group
>    must only act when *needed* -- most technical decisions should be
>    left to the community to hash out; it must not preempt the community
>    from developing better solutions.  (This is how the "core" group
>    worked during most of the project's growth period.)
>
> 8) There must be a set of commit standards -- e.g. about when it is or
>    is not acceptable to commit changes that do not change functionality;
>    when multiple changed must be batched in one commit; etc.  Right now
>    it is difficult to sort the wheat from the chaff.  In addition, there
>    must be standards of review.
>
> I must repeat a point I've made earlier.  The current "management" of
> the project is not going to either fix the project's problems, or lead
> the project to solutions.  They are going to maintain the status quo,
> and nothing else.  If the project is to rise from its charred stump,
> this "management" must be disbanded and replaced wholesale.  Anything
> less is a non-solution.
>
> --
>
> To some of you, I would like to apologize.  There *are* NetBSD
> developers doing good work even now.  I'd like to particularly recognize
> and thank those working on kernel locking and UVM problems; wireless
> support (though I'm not sure what happened to my extensive set of rtw
> bug fixes); Bluetooth; G5; and improved ARM support.  This is all good
> stuff.  In the bigger picture, though, the project needs to do a lot
> more.
>
> --
> - Charles Hannum - past founder, developer, president and director of
>   The NetBSD Project and The NetBSD Foundation; sole proprietor of The
>   NetBSD Mission; proprietor of The NetBSD CD Project.
>
> [I'm CCing this to FreeBSD and OpenBSD lists in order to share it with
> the wider *BSD community, not to start a flame war.  I hope that people
> reading it have the tact to be respectful of their peers, and consider
> how some of these issues may apply to them as well.]
>

Are you sure this isn't some sort of clandestine call for more
development?  Or are you, possibly, in one of those introspective
come-downs from a hard night of consumption?

I've only been using netbsd since the times of Itojun and lukem, but
I've really liked it the entire time.  Am I missing something?  The
key features you mention as missing are, really, only working well
with commercial products/support.  Linux only got functional threads
with help from Red Hat, Intel, and IBM.  (Solaris has always had good
threads, but solaris has always had everything. ;)  Is a really
powerful threading model even that important for embedded platforms
that you say were becoming -the- niche for netbsd?  I'd like to see
improvements there too, but I don't even know if anyone is focusing on
it.

Linux filesystems also aren't that great, unless you start to look for
commercial ones- XFS and Veritas come to mind.  LFS has recently
become very stable, so that's a good thing.

I agree with your proposed features wishlist (even if I don't agree
with the delivery), but mine looks different.  I'd like to see more
work in the installer for unattended stuff, and more work on network
appliance-type things like load balancing.  Who cares if netbsd is
getting drivers from other projects?  I want application-layer
filtering easily accessible and nics that failover without the kludgey
setup of solaris IPMP or linux bonding.  Also, all of elad's stuff has
been right in-line what where I'd like to see things go.  veriexec is
useful for many things, even in commercial space.

As far as the core team/management goes, I have no idea.  I've donated
to the project, so I like to think that they're doing something.  The
only complaint I have is that an answer to using CDDL would be nice.

Maybe you could publish an insider's view of the politics behind
netbsd for the past fifteen years.  I'd definitely love to read it.
;)

Sorry to ramble like this, but I just don't see negative comparisons
to linux as being very fair.  It's just so different.  Do you know how
much money is tied up in linux right now?  it's disgusting.  Do you
realize what a mess the entire distro quagmire continues to be?  There
are a million problems for every project, but your proposed solutions
seem a little misdirected.  Start another coup if you really want to.
Get the old guys back!  I think it sounds interesting.

_Matt