Re: The lamentation of proplib(3)

[David Holland <dholland-tech%netbsd.org@localhost>]

On Wed, Jan 29, 2014 at 03:05:41AM +0000, Mindaugas Rasiukevicius wrote:
 > In this case, the proplib implementation has such major deficiencies that
 > replacing it itself is a virtue.  Fixing is not the case here, as it would
 > basically mean rewriting.

Since this thread has died down again, here are some suggestions on
how we should move forward:

(1) It seems the consensus over the last 5+ years is that the library
we want is a data transfer library, not a data storage library. If you
want the latter, we have libdb and sqlite. The difference here is
whether the data you're handling lives primarily in the library or if
you normally maintain it in your own representation and stuff it into
the library only for transfer. (And pull it out immediately after
receive.)

If anyone disagrees violently with this conclusion it's time to speak
up.

(2) We have not yet converged on a data model. The sane choices are,
pretty much, (a) flat key-value store, (b) recursive tree of key-value
stores, (c) relational tables, (d) property graph model, or *maybe*
(e) rdf data model.

The sense I have from the various discussions is that (a) isn't enough
for at least some of the applications we have. My guess at this point
is that we want (b). The chief advantage of the more elaborate data
models is that they allow sane encoding of more complicated data; also
they have, or can have, simpler transfer encodings because they
*aren't* recursive.

I don't think we have any immediate use for graph data, but I think we
might in the future; e.g. save files for games, and also the device
attachment tree in the kernel keeps threatening to turn into a graph.

Similarly I don't see any immediate use for relational data, but we
might in the future; in particular if we improve the nsswitch
interface we might want to use a relational data transfer library to
ship config data around. Inasmuch as Unix config data is tabular, and
it isn't always key/value, a relational library would fit this case
better.

However, I think what we should do for the moment is go with (b), and
if we want a relational or graph library later, we can write another
library with a similar interface -- unlike the current proplib the new
library is supposed to be small and simple, and I don't see that
having as many as three of them with different purposes would
constitute a problem. And if we want we can always build them so they
share enough of the transfer format that they can read each other's
simple forms.

(3) It is clear that we need schema definitions and schema
enforcement. I think this should be handled as follows:
(let "libfoo" be the name of the replacement library)
   (a) the machine readable form of a libfoo schema should itself be a
       libfoo blob;
   (b) there should be a separate libfooschema library (or maybe
       libfoocheck, validate, whatever) that checks a libfoo blob
       against a libfoo schema blob;
   (c) there should be a human-readable DDL and code that reads it
       into a libfoo schema blob (maybe in libfooschema, maybe in a
       third library);
   (d) there should be a program that takes a libfoo schema (probably
       in the DDL form) and generates code that packs and unpacks
       libfoo blobs of this schema into real data structures;
   (e) there should maybe also be a program that does this and
       generates code that reads and writes the transfer format
       directly without materializing a libfoo blob, but we don't need
       this up front;

All of this is reasonably straightforward, but it can't be written
until we settle the other open issues.

(4) We need to converge on an API. I think what we should do is:
everyone who has ideas on the subject design one, or collaborate with
someone else who's designing one. Keep talking, and everybody steal
everybody else's best ideas. :-) That way we can probably converge on
a small number of alternatives.

I committed a copy of my 2008 jetlib strawman proposal to othersrc,
but it needs some attention before it can be a serious candatate. The
FreeBSD libnv is another candidate.

(5) We need to converge on a transfer format. Nobody likes XML, so we
should pick something else as the native form. I think we should
seriously consider JSON and one of the saner RDF formats for text, and
XDR and one or more ASN.1 protocols for binary, as well as perhaps
others -- there is also a simple binary encoding in the jetlib code
that we could adopt but it has nothing much to offer. And there are
probably others. Nothing says we have to allow the full generality of
whatever we choose, just as proplib doesn't allow general XML.

Note that the new library has to be able to read and write compatible
proplib XML bundles regardless, as standard compat obligations require
that.

It would be nice to be able to load transfer formats as plugins,
including in the form of the library that gets compiled into the
kernel. The jetlib code is structured so that this would be possible
(but it doesn't directly support it) -- there's also an issue of
being able to know which format you're about to receive.

(6) We also need to converge on an implementation behind the API, but
I don't think there's much to discuss about that. There shouldn't be
much "there" there -- one of the problems with proplib is that there's
a lot of internal glop that doesn't do anything useful, but I don't
think we're at risk of making that mistake over again.

-- 
David A. Holland
dholland%netbsd.org@localhost