GSoC proposal: NetBSD Kernel Documentation Management Toolchain

[Vladimir Kirillov <proger%wilab.org.ua@localhost>]

Hello, tech-kern@!

I've posted my proposal to google-melange and duplicating here.
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/darkproger/1
http://share.darkproger.net/netbsd/sysmodel/proposal.html

Looking for your inquiries!

NetBSD Kernel Documentation Management Toolchain
================================================
:Author:        Vladimir Kirillov
:Email:         proger%wilab.org.ua@localhost

== Abstract

The purpose of my project is to define a generic approach to simplify
the kernel code management tasks by defining a *unified* component
model of NetBSD kernel programming interfaces using XML representation.

I intend to design it to simplify the code transformation routines
such as generating kernel header files, documentation, configuration
files and code.

The first appliance of the XML specifications which I plan to
implement is the documentation management toolchain.

Kernel documentation is collected in the section 9 of manual pages,
which is maintained by developers manually. The section consists
of:

- intro(9) page which lists available interfaces with brief
  descriptions
- set of manual pages per particular interface/subsystem.
  The manual pages include:
        * function and data structure signatures
        * code references
        * kernel configuration options references
- a Makefile which is used to create links between
  related API functions and subsystem/component manual page

Supporting these features have the following *shortcomings*
(mostly in result of requiring manual intervention):

- code references are not always complete and synchronized immediately
  with manual pages
- function signatures may be outdated
- intro(9)-like pages may be left without attention (pages which mostly
  contain references to other)
- no documentation coverage control

I intend to address these shortcomings in this project by implementing
a tool to manage kernel documentation based on the interface XML specification.

== Goals

- Consolidate all kernel interfaces and data structures
        * use clang to extract type information from code AST
        * use manual pages to extract initial data
          (descriptions, lists of used functions, structures
           and so on)

- Define an XML specification (schema) for kernel API
        * versatile enough to assist code generation
        * providing metadata
                ** e.g. subsystem class

- Use the specification to create a header generation tool (using amd64
  as a reference architecture) and a documentation generation/verification
  tool with the following features:
        * documentation coverage control
        * ToC generator (based on the specification metadata)
        * code reference and code snippets management
        * manpage linking verifier

== Implementation plan

=== Stage 1 (Week 1 -- Week 2)
.Source analysis

        * bring in inquiry(1)
          (a clang AST consumer application which i have been
           working on previously, i plan to adopt in into this
           project, it's purpose it to extract information from
           the AST of the source files and dump it into a simplified
           model of data structures, constants/globals and functions)
                - sync with current clang API
                - implement sqlite database backend

.Deliverables
****
        * inquiry(1) source analyzer (c++)
        * sqlite database with source components (functions/data structure 
entities)
****

=== Stage 2
.Bootstrapping

        * split kernel API into groups, extract descriptions
                - use mdocml libraries
                - initial source: src/share/man/man9
                - manual processing if required

.Deliverables
****
        * annotated sqlite database with source components
        * tool/script to grab data from manuals
****

=== Stage 3
.Generating XML specifications

        * schema (main entity: kernel component/subsystem):
                - meta information
                        ** subsystem class (memory manager, driver, etc)
                        ** description
                - data structures (type information)
                - functions

.Deliverables
****
        * XML specification files
****

=== Stage 4
.Implementing tools

        * implement XML processing tools to generate headers
        * implement a documentation verification tool


Stages 2-4 are to be implemented iteratively during week 3 through 12 (Jun 6 -- 
Aug 14).
Week 13 is to be devoted to final preparations regarding the demonstration kit 
and preparing
the package of deliverables:

- tools, databases and specifications listed above
- updated kernel documentation using these tools
- documentation coverage information for further analysis
- other resources and/or code produced during the project implementation process

.Midterm Goal
****
        * present the database of the source components generated using inquiry 
tool
        * define the XML schema for interface specifications and provide sample
          interface descriptions using the defined specifications
        * present the sample workflow of documentation management routines
****

== About me

I'm an undergraduate student of Kiev Polytechnic Institute
(major: Computer Engineering) with BSD internals experience.

I intend to continue hacking on NetBSD past GSoC in order to
implement my ideas and help the project benefit of them.  I want
my project results to be suitable for inclusion into the NetBSD
tree, it's conclusions and analysis data be an input for further
development decisions.

== Further ideas

The NetBSD kernel is the largest component of the NetBSD operating
system which is not treated like a regular application.  It has
it's own configuration tool set which is designed to make modularization
possible, is documented by developer-provided manual pages and does not
simply yield to generic debugging techniques.

Latest advances in the project show that a lot of kernel code
needs to be reused in the userspace or even within other operating
systems which will eventually require mechanisms to apply code
transformations to fit into other environments except the kernel
itself.

I want to give this approach a chance to be beneficial for the
NetBSD kernel by assisting developers concentrating on the actual
hacking by simplifying and automating routine code management tasks.

http://share.darkproger.net/netbsd/sysmodel/ideas.html
http://share.darkproger.net/netbsd/sysmodel/problems.html