Why not emulate the way info works for the html format?  There is
  a central directory file (called "dir") in every info directory.

Perhaps, but text and pdf should be linked in too.  People want to
read docs, and use the necessary tools (or at least I think they
should), rather than narrowing to format first.  Choosing a format
first is only rational when one knows that this doesn't lose anything.