Re: (Re-)Structuring NetBSD user documentation

[David Holland <dholland-docs%netbsd.org@localhost>]

On Fri, Sep 21, 2012 at 12:16:06AM +0200, Julian Djamil Fagir wrote:
 > Imho, currently NetBSD's documentation has, despite being great and better
 > than anybody else's, three major problems:
 > [snip]

Rather than tackle your points one by one, I'm going to try to write a
single response, to avoid getting the outline lost in a pile of
details.

It is true that there are too many places documentation appears. In an
ideal world there should be (the online equivalent of) one large
bookshelf that has everything in it. It is also true that there's a
problem with tools and formats, and that the problem with tools and
formats is a direct cause of the documentation being scattered.

However, there's more to it than just that. One issue is that the
bookshelf needs to ship with the system and be readable from the
console: there are still plenty of cases where machines are behind
restrictive firewalls or the network isn't working and you're trying
to fix it. Or something broke your X server, you're limited to the
console until you sort it out, and you don't have a spare machine
handy. Those are often the times you need the docs most, so while they
should certainly be posted on the web site, having them *only* on the
web site isn't good enough.

Also, it's not the case that we can assume people already know Unix.
There are a lot of Linux users; most of them know little about Linux,
let alone about traditional Unix. And even things that are genuinely
standard, like basic vi usage, still need to be documented... let
alone things like PAM where even with full documentation it's very
easy for experts to blunder.

I've pointed out before in other contexts that there are multiple
types of documentation; most importantly, there are user's guides and
tutorials, which cover the important aspects of what they document and
are structured to provide an organized and coherent introduction to
the material when read through; and then there are reference manuals,
which provide complete descriptions of all aspects of what they
document, are usually heavily cross-referenced, are meant to be
searched, and make no attempt to avoid or explain forward references.
(Other types include FAQs and HOWTO guides, which are user-centric and
goal-centric rather than subject-centric.)

We need both a user's guide and a complete reference manual; they
should be separate, and even though they may overlap by topic, because
of their difference in purpose they should not in general share text.

It is also a bad idea to try to automatically assemble large documents
from collections of small ones. Texinfo works that way and it almost
guarantees you badly organized documentation. Trying to use a pile of
wiki articles as a single large document will almost certainly produce
the same results.

Once upon a time, BSD Unix came with a documentation shelf. We still
have bits of that. However, it's bitrotted: the problem is not that
what's in /usr/share/doc is obsolete (although much of it should at
this point just be flushed) but that we no longer have the ability to
build and ship the shelf: as you noted the available docs are too
scattered. There isn't *a* shelf any more, there's just a lot of
stuff, and because there's a lot of stuff in a lot of places, it's no
longer possible to exercise any kind of effective editorial oversight.
Another thing we've lost is that there used to be an organizational
scheme for the shelf, but it's no longer adequate; the world's gotten
vastly messier and more complicated and the documentation structure
needs to be strengthened accordingly.

As I understand things, the Guide was intentionally created separate
from the old stuff in an attempt to move away from a legacy structure
that was no longer working very well. It is possible that all we need
to do is flog the tools until we can install the Guide into
/usr/share/doc, merge a few things into the Guide, and call it done.
However, I suspect that if we really try to wedge everything into the
Guide its internal structure (which is already a bit problematic) will
come unstuck. So I think what we need to do is to come up with a new
structure, taking the 4.4-era one into account. And also taking
account of things like the distinction between user's guides and
reference manuals.

Then we really need to collect all the documentation into one place
and, well, edit it, in the editorial sense, so it fits the structure
we've chosen. This also entails figuring out how to publish it both in
/usr/share/doc, on the project web site, and possibly in other places,
all using the one master copy.

Tools are a secondary problem in this: it does not help that we have
multiple toolsets and that many of them suck in assorted ways, but I
don't think it's a critical part of the problem. The critical problem
is organizational. In particular, I've had all the tools installed for
years (docbook, jade, etc.) and I've never understood why people think
installing them and updating them occasionally is such a big problem.

However, because we really ought to be shipping *a* shelf, and we want
it to come with the base system, we need to come to some realistic
conclusions about what tools we're willing to have in base and how
that relates to the source formats we want to maintain documentation
in.

I don't think anyone particularly likes docbook, for example, so I
don't think there's any great need to argue against importing the
entire docbook toolchain into base, where it wouldn't be particularly
welcome anyway. AFAICT the serious existing source format candidates
are markdown and roff -mdoc. There doesn't seem to be any great reason
to choose any other wiki markup language over markdown; AFAICT they're
all fundamentally similar.

In theory we could just adopt these for everything; however, there are
three major problems:

   (1) it isn't clear that everything we currently need to publish as
       HTML can be represented adequately in markdown;

   (2) last I remember mdoc -> HTML was still moderately problematic,
       and also we may not want to maintain web pages in roff;

   (3) AFAICT mdoc is not suitable for miscellaneous documents that
       aren't man pages. (If this is not true, please explain how to
       convert src/share/doc/smm/04.quotas/quotas.ms.)

These points can probably be addressed by defining a variant (or
extension) of mdoc that is suitable for miscellaneous documents; it
might also be helpful in the long run to cook up an alternate (read:
less ugly) input language for mdoc and/or this alternate. However,
there are other possible approaches, and this mail is already too
long...

-- 
David A. Holland
dholland%netbsd.org@localhost