Re: GSoC 2016 project proposal: Split debug symbols for pkgsrc builds

[Matt Sporleder <msporleder%gmail.com@localhost>]

I love this project and think o may have added it. I am, however, a poor choice for primary mentor, which is why I didn't add it to gsoc section. 

If anyone else thinks they can support you I am willing to help with testing and stuff. 

> On Mar 14, 2016, at 3:29 PM, Leonardo Taccari <leot%NetBSD.org@localhost> wrote:
> 
> Hello to the entire tech-pkg@ list,
> I have found an interesting project and if it can be considered a GSoC
> project (actually it isn't) I really would like to work on it as a
> student.
> 
> I am studying for a Master Degree in Computing and Automation
> Engineering at Università Politecnica delle Marche in Ancona, Italy.
> I am also a recent NetBSD developer and have experience in particular
> as a package maintainer. I've also sporadically contributed in htdocs
> and src areas too.
> 
> 
> Rationale
> ---------
> The ability to debug software is important not just when developing
> but also when using it, e.g. post-mortem analysis.
> NetBSD provides MKDEBUG and MKDEBUGLIB variables that can be set in
> mk.conf in order to split debugging symbols for user-land applications
> and libraries. Resulting split debugging symbols are then available
> via the debug.tgz and xdebug.tgz installation sets.
> NetBSD also provides Rump and to some extent also ddb(4),
> ktrace(1) and DTrace that ease analysis, tracing and debugging.
> All these features make NetBSD a great operating system in this regard.
> However, in pkgsrc it is possible only to generate packages with
> debugging symbols by providing proper CFLAGS for debugging and
> setting the INSTALL_UNSTRIPPED flag. This make debugging, especially
> for binary packages users, not very feasible.
> 
> 
> About the project
> -----------------
> A more convenient way - like what RPM and Debian package manager do - is
> to provide a <package>-{debuginfo,dbg} for <package> (where applicable)
> that includes all stripped debugging symbols.
> Thus, the project consists to add a mechanism in the pkgsrc
> infrastructure in order to make the generation of the
> <package>-{debuginfo,dbg} possible and transparent.
> 
> Providing <package>-{debuginfo,dbg} separately is also worth because
> debugging symbols have a cost in term of disk space needs.
> As an example, after quickly analyzing the space for NetBSD/amd64
> -current installation sets it can be observed that {debug,xdebug}.tgz
> sets are (in MB):
> 
> $ ls -sk *debug.tgz | awk '{ s += $1 } END { print (s / 1024) }'
> 492.578
> 
> ...while all the other installation sets (not considering kern-XEN3*.tgz
> kernels and {debug,xdebug}.tgz installation sets) are (in MB):
> 
> $ ls -sk [bcegmt]* x[bcefs]* kern-GENERIC.tgz | awk '{ s += $1 } END { print (s / 1024) }'
> 419.744
> 
> When extracted {debug,xdebug}.tgz need approximately 1.5GB.
> 
> Ideally all the packages that have USE_LANGUAGES "c" and/or "c99" and/or
> "c++" (and maybe also other programming languages) can be compiled with
> debugging symbols and it will be needed to automatically generate the
> PLIST for the <package>-{debuginfo,dbg} given the PLIST of the <package>.
> The <package>-{debuginfo,dbg} will be generated if a mk.conf variable
> is defined (e.g. PKGSRC_MKDEBUG) and also automatically installed. In
> order to handle them the <package>-{debuginfo,dbg} will DEPENDS on
> <package> (of course this can be argued because - strictly speaking -
> the <package>-{debuginfo,dbg} will not depends on any other packages
> but in practice they are useful only if the <package> is installed).
> 
> All the above would - hopefully - not need any change in the
> packages' Makefiles.
> For packages that for some reasons the debugging symbols can not
> be generated it will be needed to add a (per-package) Makefile
> variable in order to avoid the generation and splitting of the
> debug symbols (e.g. the various emulators/suse* packages).
> Also any package that presents some other problems or when generating
> debugging symbols isn't logical can use that variable to inhibit the
> generation of debug symbols.
> 
> 
> Related works
> -------------
> As stated above NetBSD already supports splitting the debugging symbols.
> There are also some existing package management systems like RPM
> or the Debian package manager that support that. Both NetBSD MKDEBUG*,
> RPM and Debian package manager (and maybe also others) can probably
> be taken as a good source for inspiration and design.
> 
> AFAIK, neither FreeBSD ports nor OpenBSD ports support splitting debug
> symbols in separate packages.
> 
> 
> Deliverables
> ------------
> * analyze how splitting debugging symbols works taking NetBSD MKDEBUG*
>   as an example and also considering other package management systems,
>   in particular for the design of the needed infrastructure.
> * design and implement the infrastructure in pkgsrc that handles the
>   compilation, split of the debug symbols and generation of the
>   <package>-{debuginfo,dbg} packages.
> * document the interface in ``The pkgsrc guide'' targeting pkgsrc
>   users, MAINTAINERs and developers. Also provide useful documentation
>   for the on-line pkgsrc documentation via the ``help'' target.
> * run a bulk build with the strip debug functionality turned on for at
>   least a significant subset of packages on NetBSD in order to
>   verify that the implemented infrastructure correctly works.
>   This part will probably reveals problematic packages (e.g. package
>   that ignore CFLAGS). Fixing these packages will also indirectly
>   improve the hygiene of the pkgsrc ecosystem.
> 
> 
> That's just a rough picture of the project and a lot of design choices
> will need further discussions and pointers.
> I consider this potential GSoC project a great opportunity to work
> full-time on NetBSD and pkgsrc and, in particular, to become more
> familiar with internals and various subsystems, not just of pkgsrc.
> Last but not least, it's also a chance to cooperate with the NetBSD
> and pkgsrc community in a not-so-short-term project.
> 
> 
> Please note that currently the discussed project proposal isn't
> considered as a possible GSoC project and neither has a mentor.
> It was although discussed briefly here:
> 
> http://wiki.netbsd.org/projects/project/pkgsrc-debugpkg/
> 
> 
> Please let me know if this can be taken as a possible GSoC project
> proposal (and in that case it will also need a mentor! :)). Please also
> share any further ideas, issues, etc. that I've missed.
> If you know any documentation/examples or any other possible kind of
> resource related to this project I'm very happy to start reading and
> studying it.
> 
> 
> PS: riastradh@ kindly offered as a mentor for this project after quickly
>    discussing that via IRC if another one can co-mentor and has more
>    experience in pkgsrc internals as well.
>    So also a co-mentor highly familiar with pkgsrc internals is
>    needed.
> 
> 
> Thank you very much for the attention!
> Ciao,
> L.