Re: proplist [was: Importing xmlgrep into base]

[Nhat Minh Le <nhat.minh.le%huoc.org@localhost>]

At Tue, 13 Apr 2010 23:40:12 -0400,
Matthew Mondor wrote:
> 
> On Tue, 13 Apr 2010 18:49:28 -0400
> "Martin S. Weber" <Ephaeton%gmx.net@localhost> wrote:
> 
> > Which really makes one wonder why proplib uses a half-baked 
> > XML format anyways. Pity s-exp weren't used back then :)
> 
> I prefer S-EXP to XML myself (or even JSON if for such simple data),
> but unfortuately there are two problems: we don't have a lisp in base
> (although of course an S-EXP importer/exporter is extremely easy to
> write, if we only want to use it for data representations), and
> unfortunately XML is what people now expect for interchange and
> transformations, considering its popularity (other than people scared
> away by other formats they're not used to).

Actually, the xmltools are built around libregxml, which is made of
regxml(3) and regxml_xml(3) at the moment, and the actual XML
parsing/printing code is isolated in regxml_xml(3). It would be easy
to add any format which shares an isomorphism with XML. I'm no fan of
XML myself, but as you say, it's the format that's popular these
days.

Now there's also the fact that, as awkward as the XML standard is[1],
it's still standard, and translating things like processing
instructions or even attributes to/from s-expressions would require
more than the commonly accepted subset of "s-exp" as a data
representation language. We might represent attributes as keyword
arguments (i.e. `:name value'), and PIs in, say, square brackets
(i.e. `[target "content"]'). But it'd just be an arbitrary choice; we
could as well write PIs as `#(target "content")'.

> As for the ifconfig examples it's already possible to extract useful
> information using older text tools such as grep and awk, but I
> understand the utility of an official output format for use with
> scripts (i.e. adding or deleting a column would cause issues, while
> specialized tools can query specific fields), especially if those
> formats were standard (or popular and expected by some existing
> analysis/statistics tools).
> 
> 
> I participated to a thread on the XML propery lists months ago and
> suggested ideas like supporting an optional on-disk binary format
> designed for efficient queries, along with an importing/exporting tool,
> which optionally could support various formats added as
> "modules" (i.e. .so dlopen-able).  And the possibility of having a
> vipw-like tool which would lock, export to the user's favorite format,
> launch EDITOR and re-import to the binary (or official text format if
> no DB available).

I find one of the best features of text formats is that you can
reasonably version them. But if you're passing through an
import/export filter everytime, what you write might not be what will
be exported for you to edit next time. So in the end, that still means
you have to keep your own copy of the text files somewhere and just
import that into the binary format whenever you make a change. So I'm
not sure a vipw-like tool would be that useful.

> Unfortunately I couldn't write or propose code so far, being busy
> with life necessities, and there seemed to be no follow-up to the
> thread.  I however still encourage others to think about this idea:
> 
> This would allow property lists to potentially even serve the role of
> other existing formats where there's a need to export to a binary DB
> format for performance, as well as allow users to use their favorite
> analysis/editing/configuration tools.  For instance, knowing I have a
> CL implementation on my system, I'd write and propose an s-exp
> import/export tool and could use those, allowing me to use a format
> which is both readable by my code and eyes.
> 
> Another user might prefer a fully XML compliant format for use with
> xpath/xslt, and be able to use that, too.  And none of this would
> matter, because in-base code would work with the official APIs and
> format...  But of course this would need some work to implement (and
> it has to first be considered worthwhile by more than a single person).

Actually, I wonder about the usefulness of this multi-format approach
as far as scripts are concerned, because having multiple export
formats also means everybody will be using whatever format he prefers,
so that we might see complementary tools that work on one format but
not another, and combining them would take several layers to convert
back and forth.

Wouldn't it make more sense that base utilities always output and read
only the official format, and provide conversion tools as simple
separate text-to-text userland tools and libraries? Of course, it
induces an overhead for scripts that don't use the official format,
but still, that overhead would occur if we were to combine
heterogenous scripts as described above, beside having the user do it
instead of the program, which may have better performances, but is not
practical, IMHO.

> In the last few years I've seen proplist being proposed as a solution
> to more and more problems.  If it's to become very ubiquitus, such
> tools are probably good to have.  Especially if it eventually begins to
> be used in roles where a capabilities database would previously have
> been used (which has cap_mkdb).
> 
> Assuming the above-mentioned import/export capabilities were available,
> tools which optionally export a property list (like in the ifconfig
> example) could either accept an export format along with the argument
> to enable proplist output, or extract the user's favorite export format
> from an ENV var...  Or, the user could pipe the output through the
> export tool, which would minimize changes to proplib(3) itself.

As stated above, my vote goes for the "pipe through the export tool"
approach.

> After reading a bit, it seems that OSX has plutil, although it's only
> to convert between the binary/text representations, and I'm unsure the
> binary representation is designed for efficiency (this has to be
> investigated).  It also provides a hierarchical property list
> viewer/editor, so the XML is transparent to users.  It probably would
> not be too difficult to write a similar curses utility.  As for the user
> import/export formats proposed above, they could be handled by plutil
> with additional userland modules as well...
> -- 
> Matt

[1] And xmltools doesn't support the XML standard in full, as stated
  in various places in the manual pages; specifically, xmltools(7) has
  a COMPATIBILITY section dedicated to the subject. Unsupported
  features, though, are mostly irrelevant or easily worked around as
  far as processing of system information, as discussed here, is
  concerned.

--
Nhat Minh