Re: [GsoC] Add pkgsrc support to packagekit

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Wed, Mar 30, 2011 at 5:48 PM, Sylvain Mora
<sylvain.mora%googlemail.com@localhost> wrote:
> Hi,
>
> Thanks you for your response.
>
> Le 29 mars 2011 à 12:12, Julio Merino a écrit :
>>
>> Having an API to interact with pkgsrc and binary packages would be
>> wonderful.  Unfortunately, Python is not part of NetBSD.  If you write
>> the library in Python, it means that we cannot use it in the places
>> where we need it most: in sysinst or in pkg_install so that an
>> out-of-the-box NetBSD installation could easily retrieve/install new
>> packages and keep the system up to date.
>>
>> Having said that, a Python API would work nicely as a prototype.  But
>> you should investigate how this would play with the languages that are
>> part of NetBSD (C, C++, shell and Lua).
>
> The goal is not the create a full API for all the system, but a basic API to 
> help to the creation of the backend. It is a layer which retrieves pkgsrc 
> commands stream.
>  It could be a good idea to create an API for C pkgsrc but I think it is for 
> an other project.

Sorry, but I still don't understand where the need to use Python comes
from.  Is PackageKit written in Python?  If not, how does writing this
intermediate API in Python help the goal of adding pkgsrc support to
PackageKit?

As mentioned, writing this API in C would be very useful, both for the
PackageKit project and for NetBSD in general.  If we had such API,
we'd (I think) very easily integrate pkgsrc support in sysinst, for
example :-)  (In this case, using Lua would make sense though, because
Lua is supported by NetBSD.)

(Given that the major goal of your project is to add PackageKit
support, this API needn't be complete: such API only needs to provide
the set of features required to implement the PackageKit backend.  But
working on an API that can later be easily extended would be a nice
idea.)

>>> Reading the Wiki I think I can make the same work with pkgsrc :
>>> - Write an API for pkgsrc
>>> - Use this API to write a backend
>>>
>>> It may be possible to reuse part of the pkgin API for binary package 
>>> management (or rewrite something equivalent for pkg_add).
>>>
>>> The API will reproduce basics pkgin/pkg_add/pkgsrc commands and add useful 
>>> methods like listing all outdated packages, listing installed packages in 
>>> which package names, versions, etc are separate.
>>
>> In my opinion, the library should seamlessly manage packages from
>> pkgsrc and binary packages (this is quite tricky but certainly
>> doable).  An example: install/update everything from binaries if they
>> are up-to-date, or otherwise fall back to building from pkgsrc.
>
> The wiki says that the user have the choice to fall back, but I'm agree with 
> you. May be it's possible to make the choice with an option :
> - automatically install with pkgsrc if the package is not available as binary
> - ask to the user if he want to install the package with pkgsrc
> - do nothing and say to the user how he can install the package

That sounds good.  A thing to keep in mind: when you say "ask", that
should not be an interactive question.  The user must be able to
pre-configure behavior.  Interactive questions during package
installations are pretty annoying, specially on slow machines!

>>> The harder part will be the capability to build package from source because 
>>> I have to reproduce what did "make" in Python for retrieving dependencies 
>>> etc (The backend allows to make package if it is not available as binary)
>>
>> For this, you can just resort to calling make and parsing the results;
>> this is slow but effective.  Alternatively, if the "summary" database
>> exists, just parse that database to get much faster results; see
>> pkg_summary(5).
>
> Yes, for now I have two choice: parse the "make result" stream, or search by 
> my own way into package files to retrieve deps.

You do not need to scan package files to retrieve dependencies.  Take
a look at pkg_summary(5): this information is already available in a
database that you can query efficiently.  This is most useful when
retrieving dependencies from remote servers.  If the summary is not
available, then yes, you can (and should!) scan package files
directly.

>> Lastly, I'd recommend to extend your proposal a bit more by including
>> a list of deliverables, a preliminary schedule with a set of major
>> milestones, and more examples on how your proposed API would look
>> like.
>
> I expect to make that very clear in my head. Really know what you expect the 
> project.
> But if you really want a schedule, I can make an effort to do this very 
> quickly.

Preparing an schedule will help you plan the project in your head
clearly.  It will also let us see that the project has a rough plan
and a set of milestones.  This is a very important thing to have, as
it gives more value to your project proposal... and a detailed
schedule means that your project has more chances of succeeding in the
allowed time frame.

> For the API I don't know what type of example you want.

I think it'd be nice if you'd list the kind of operations (i.e. the
public interface) you expect this API to support.  For example, start
by detailing what exactly does PackageKit need for its backend, and
then list how these PackageKit operations should translate to pkgsrc
operations.  The more you try to define a specific API, the more you
will realize things that you didn't realize in advance.

>> Hope this helps!
>
> Yes, it helps me. This is the first time I make a project so it's a bit 
> confusing for me.

No problem; that's the whole point of GSoC ;-)

Looking forward to seeing your final proposal.

Cheers,

-- 
Julio Merino / @jmmv