[announce] wip/pkg_online - client/server search in pkgsrc packages

[Aleksey Cheusov <cheusov%tut.by@localhost>]

Hi everybody. I'm develping pkg_online - tool for client/server search
in pkgsrc packages. Documentation is not excellent yet, but I hope the
following is clear enough for the beginning.

I've packaged it in wip/pkg_online already.  Just install it and try
client (the instruction is below).  I'm running server at
dictd.xdsl.by:26280 to make testing easier. To run client you don't
need to run your own server, i.e. you should not set PKG_ONLINE_SERVER
and PKG_ONLINE_CLIENT variables. The default values are good.

Server part tool are also packaged. Of course you can play with it
too.

The project now has "need feedback" status ;-)

#################################################################
#################################################################
#################################################################

DESCR:

pkg_online - client/server search in pkgsrc packages.

Every separate field (PKGPATH, PKGNAME, COMMENT etc.) may be queried
separately and multiple fields may be queried too. Lots of search
strategies are available. A set of fields and search strategies may
very depending on your server configuration. For example, you can
search in binary (pkg_summary.gz generated by pkg_info -X) or source
(pkg_src_summary.gz generated by wip/pkg_src_summary) repositories.

Communication protocol is a well known dictionary protocol described
in RFC-2229. Server runs dictd (wip/dict-server)

#################################################################
#################################################################
#################################################################


    Client/Server system for searching in pkgsrc packages.

     Dictionary protocol (RFC-2229) is used for searching.

 There is nothing specific to pkgsrc. It can easily be adopted
              to any other packaging system.

##########

useful URLs:

  http://www.dict.org
  http://www.pkgsrc.org

#################################################################

Requirements:

  Server side (building utilities):

    - pkg_src_summary (for building summary for packages from
      pkgsrc/ tree)

      packaged in pkgsrc/wip/pkg_src_summary

    - pkg_summary4view

      packaged in pkgsrc/wip/pkg_summary4view

    - dictfmt > 1.10.10,

      http://sourceforge.net/projects/dict,
      version packaged in pkgsrc/wip/dict-server is good
        (>= dictd-1.10.10nb2)

    - pipestatus

      http://sourceforge.net/projects/pipestatus

    - dictd dictionary server > 1.10.10,

      http://sourceforge.net/projects/dict

      version packaged in pkgsrc/wip/dict-server is good
        (>= dictd-1.10.10nb2)

  Client side:

    - dictionary protocol client compatible with 'dict >= 1.10.9'

      http://sourceforge.net/projects/dict

      version packaged in pkgsrc/wip/dict-client is good

    - pipestatus

      http://sourceforge.net/projects/pipestatus

    - netcat

      net/netcat pkgsrc package

Installation:

  make all install-dirs install  
  export PATH=$PATH:/usr/local/bin

Sample of client usage:

  Initialization.

    export PKG_ONLINE_SERVER=your.server.org  # default - dictd.xdsl.by
    export PKG_ONLINE_PORT=your.port          # default - 2628
    # By default my server dictd.xdsl.by:26280 is used

    export PKG_ONLINE_NETCAT=/usr/pkg/sbin/nc # default - nc

  Running.

    # ask server for a list of available search strategies
    pkg_online_client -s

    # ask server for a list of fields to search in
    pkg_online_client -f

    # obtain information about packages (PKGNAME here, not PKGPATH!)
    pkg_online_client -i xxkb
    pkg_online_client -i dict-server dict-client
    pkg_online_client -i dictem

    # low-level search tool (full names for strategies and fields!)
    pkg_online_client 'MAINTAINER:prefix:jlam@'
    pkg_online_client 'PKGPATH:suffix:/dictem'
    pkg_online_client 'PKGPATH:substring:wip/p5-'
    pkg_online_client 'COMMENT:re:hash.*table'

    # multiple queries together and short names for strats and fields
    pkg_online_find '::dict'
    pkg_online_find '::dict' 'm:p:cheusov'
    pkg_online_find 'c:w:vpn'
    pkg_online_find 'c:r:dns.*server'
    pkg_online_find 'c:r:dns.*server' 'p:p:net'

    Here field 'c' means COMMENT, 'm' means MAINTAINER
    and 'p' means PKGPATH. Search strategy 'p' means 'prefix',
    '' means '.' (default search strategy, may be set individually per field),
    'r' means 're' and 'w' means 'word'.

    For the list of synonyms look inside pkg_online_client.env.
    Of course it is allowed to combine full names and short synonyms.
    Synonyms are overridable by you, see pkg_online_client.env.

Environment variables:

   PKG_ONLINE_SERVER () - dictionary server host name
   PKG_ONLINE_PORT ()   - dictionary server port
   PKG_ONLINE_CLIENT (pkg_online_client) - override it if you want
      to implement offline search, new search strategies without
      reimplemeneting dictionary server or anything you need.
   PKG_ONLINE_DICT (dict) - path to dictionary protocol client.
   PKG_SUMMARY_TYPE (src) - "src" or "bin" for source or binary summary
      (output of 'wip/pkg_src_summary' or 'pkg_info -X')
   PKG_ONLINE_NETCAT (nc) - path to netcat program

Running the server:
XXX:
   export PKG_SUMMARY_TYPE=src # or bin depending on your summary type
   pkg_summary2all summary.txt
   mv *.index *.dict /path/to/dictd/databases/
   pkg_online_mkconf /path/to/dictd/databases > /path/to/dictd/config.conf
   finish configuring /path/to/dictd/config.conf
   run dictd

Basic concepts:
XXX:
   dictionary protocol server serves a number of
   databases (dictionaries). All these dictionaries are
   specially named, e.g. pkgsrc-src-index-COMMENT,
   pkgsrc-src-index-MAINTAINER and pkgsrc-src-info.
   pkgsrc-src-index-* are for indexing fields
      (field -> pkgname).
   pkgsrc-src-info keeps information about all packages
      (pkgname -> info).
   Dictionary server also provides a number of
   so called search strategies, e.g. exact, prefix, suffix etc.

   Look inside pkg_online_client to understand
   how dictionary protocol/server/client are used.

http://www.dict.org
http://sourceforge.net/projects/dict
RFC-2229

#################################################################

-- 
Best regards, Aleksey Cheusov.