Re: Package search in modern package managers. NIH approach

[Aleksey Cheusov <cheusov%tut.by@localhost>]

On Fri, Mar 8, 2013 at 9:03 PM, Jeremy C. Reed <reed%reedmedia.net@localhost> 
wrote:
> On Fri, 8 Mar 2013, Aleksey Cheusov wrote:
>
>> I hope you'll find it interesting.
>
> Thanks.  For your information "yum whatprovides" will download and use a
> large filelists_db to find packages containing the file,

Ok. Thanks.

> How do you do your PLIST searches?  Do you convert all to some database?

If case pkg_online there is a special dictd database (dict.mova.org:26280)
that provides PLIST entries.

0 cheusov>time -p nih search -o f::bin/emacs
editors/emacs-snapshot(emacs24) - GNU editing macros (development version)
editors/emacs20           - GNU editing macros (editor)
editors/emacs21           - GNU editing macros (editor)
editors/emacs21-nox11     - GNU editing macros (editor)
editors/emacs22           - GNU editing macros (editor)
editors/emacs22-nox11     - GNU editing macros (editor)
editors/emacs23           - GNU editing macros (editor)
editors/emacs23-nox11     - GNU editing macros (editor)
editors/emacs24           - GNU editing macros (editor)
editors/emacs24-nox11     - GNU editing macros (editor)
wip/emacs-current(emacs)  - GNU editing macros (editor)
wip/emacs-current-nox11(emacs-nox11) - GNU editing macros (editor, no x11)
wip/emacs-rc(emacs)       - GNU editing macros (editor)
real 0.93
user 0.26
sys 0.53
0 cheusov>

This "database" is actually a text file which is able to do a binary search
for "exact" search strategy. Search strategies like "substring" do
linear search which is
also fast enough I think.

0 cheusov>time -p nih search -o f:s:emacs
cad/dinotrace-mode        - Emacs major mode for dinotrace
cad/verilog-mode          - Verilog mode for Emacs
chat/emacs-jabber         - Jabber client for Emacs
chat/irchat-pj            - Emacs lisp interface to Internet Relay Chat
chat/navi2ch              - 2ch BBS browser for Emacs
chat/riece                - IRC client for Emacs
...
real 2.01
user 0.37
sys 0.73
0 cheusov>

"nih search -b" (the default, search in binary repository) cannot provide PLIST
search field because it is absent in pkg_summary.txt downloaded from
ftp.netbsd.org.
If PLIST entries were in pkg_summary, search would be done by awk script without
any special database. It is possible however to use sqlite, bdb, cdb
or whatever you want
as a backend.

"nih search -I" (search in installed packages). Currently the same as -b.
"exact" search strategy can be optimized with a help of "pkg_info -F".

"nih search -p" (search in local pkgsrc tree). The same as -b.
Local pkg_summary plain text file is created incrementally (efficiently)
and then pkg_grep_summary is used for search. No special databases.
On my old Atom it is not very fast (>200Mg pkg_summary)
but PLIST is provided (by default).

The whole idea is to hide implementation details behind stable and simple "API".