[Fwd: Final writeup for Experiment of early Copy-editing of WG drafts]

[Bill Sommerfeld <sommerfeld%sun.com@localhost>]

-----Forwarded Message-----
From: "Wijnen, Bert (Bert)" <bwijnen%lucent.com@localhost>
To: Jon Bright <jon%siliconcircus.com@localhost>, Bill Sommerfeld <sommerfeld%sun.com@localhost>
Cc: Russ Housley (E-mail) <housley%vigilsec.com@localhost>
Subject: Final writeup for Experiment of early Copy-editing of WG drafts
Date: Fri, 09 Sep 2005 23:01:04 +0200

FYI, here is the final write-up of the experiment

Bert

-----------

Experiment: early copy-editing of WG documents by RFC-Editor

Time:       September 20005 - November 2005

Objectives: Improve document quality early on

  Experiment to perform as much of the editorial work as possible
  early in the process, e.g., before working group last call.

  This note describes a very limited initial experiment that
  should begin to sort out the issues.  We can then decide
  whether further experimentation is warranted.

Expected impact:

  - positive impact on WG Last Call, AD review, IETF Last Call
    and IESG review. This is expected because of clearer/better
    text early on.

  - less copy-editing, so faster process after IESG approval.
    This hopefully reduces the time between IESG approval and RFC
    publication.

  - Reduction of time spend in status AUTH48.
    This is expected because there should be less changes (if any)
    between the approved text and the rfc-to-be-published.

  NOTE: we must realize that this initial and limited experiment
        (with only a small set of testcases) cannot be used to
        draw general conclusions w.r.t. to any possible future
        large scale application of the process followed in this
        experiment. They can just be a small first set of
        datapoints to help us focus on where we can try to
        improve our process.

Approach:

 - co-operation between I-D authors/editors, WG chairs, ADs and
   RFC-Editor in order to make this work. The team of people
   involved in this experiement are all expected to contribute
   to a report to describe how it went and what we measered in
   terms of positive (or negative) results/impacts. The AD and
   WG chairs already work together in the PROTO-TEAM approach,
   and this can continue for this experiement too.

 - Select 3 to 5 documents just before WG Last Call.
   ADs select them with their WG chairs, and channel them through
   Bert Wijnen so as to try and stick to a max of 5 and to
   try and not have them show up at RFC-Editor all at the same
   time.

 - Such documents (for this first experiment) should be in
   (valid) xml source that generated the I-D to be copy-edited.
   This experiment is being limited to xml2rfc files for simplicity.
   Possibly in later experiments we can do other source files
   or maybe nroff files, but not in this initial experiment.

 - Selected document checked into a source control system before
   copy-editing starts (by Bert Wijnen)

 - RFC-Editor to make copy-editing changes to the xml source.

 - RFC-Editor copy-editing is expected to take no longer than
   1 week (This is currently only commited for the 3-5 documents
   to be selected for the test, and any follow up experiments or
   changed process will need to evaluate and decide/negotiate
   for this timing). 

 - RFC-Editor will make changes that are clearly typos, or bad 
   grammar and such.

 - If RFC-Editor may go into a discussion with the author/editor
   to get some clarifications or to discuss suggested text (by
   RFC-Editor). However, this is not intended to be a long
   iterative process. If RFC-Editor and author/editor cannot
   agree quickly, then RFC-Editor just adds the new proposed
   text with tags around it to indicate what they want to change.
   For example it could look like:

      [rfc-editor copy-editor suggests to replace]
         ... old text....
      [rfc-editor copy-editor with]
         ... new text....
      [rfc-editor copy-editor end replacement text]

   The reasoning is that we want this copy-editing to not take
   up more than 1 week elapsed time.

   The WG is then responsible to sort this out by accepting the
   replacement or substituting clear, unambiguous text.

 - When RFC-editor is done, they send the updated .xml source to
   Bert Wijnen. He will check it into the source control system
   before handing it back to author/wg.

 - Author/editor generates a new revision of the I-D and sends
   it to internet-drafts%ietf.org@localhost who publishes (normal process).
   A diff file will also be generated and be made available to the
   WG, so they can easily see the changes if they want to.

 - WG Last Call is to happen right after copy-editing is complete
   (the previous few steps).

 - The WG and also the authors/editors review the copy-edited 
   document and have the right to undo changes made by the
   rfc-editor. All this happens during the normal 2-week WG
   Last Call. 

   The WG chairs should be involved and agree when:

    - copy-edited text is to be undone.
 
    - suggested replacement copy-editor text is to be selected
      or discarded

   The RFC-Editor/copy-editor is not expected to be consulted 
   for any of this WG review/change process (an occasional
   clarifying question is OK, but we should not end up in 
   negotiations about the text at this time).

 - After WG Last Call completes and WG reaches consensus on the
   document, the WG submits the final I-D for AD review and
   processing. The xml source file of this version (if it is
   changed as a result of WG Last Call) is again checked into
   the source control system, so that easy comparisons can
   be made.

 - Document further follows normal process.
   When the IESG approved document is complete, we again
   check the xml source file into the source code control system.

 - RFC-editor, when document is ready to be EDITED for RFC-publication
   checks the changes/diffs made since they did the copy-edit run.
   If no changes were made, doc is ready for publication; i.e.
   no more extra copy-editing. 
   If doc was changed, only copy-editing takes place in places
   where the changes were made. Any such changes are made in the xml
   source file.

 - The final xml-source file (from which RFC-to-be gets generated)
   will also be checked into the source code control system. 

   Note: the idea is that the source xml file will have all the
         latest copy-edited text, so that we can run diffs to see
         what was changed at the end of the process, so we can
         analyze the results of the experiment more easily).

   Note: this last step is conditional pending fixes to xml2rfc to
         ensure that it can indeed generate the final RFC-text.

 - RFC is published

 Note: the idea of checking the files into a source code control
       system (i'd like to see one either at rfc-editor or on
       ietf system or maybe on a system controlled by the tools 
       team) is that we can later more easily analyze what kind 
       of changes were made between the various steps.

Measurements:

  - RFC-Editor/copy-editor keeps track of time spend on initial
    copy-editing pass (both elapsed and person-hours)
  - RFC-Editor keeps track of time and changes when doc comes
    in normal RFC-Editor queue
  - RFC-Editor keeps track of number of issues that did not
    get resolved in exchanges with author/editors during early
    copy-editing stage.	
  - RFC-Editor keeps track of AUTH48 progress

  - WG chairs keep track of: 
    - how many changes were accepted without discussion
    - how many changes were accepted after discussion
    - how many changes were discarded (and a log why)
    - how many suggested text replacements were accepted;
      same split (with, without discussion and discarded)
    - evaluate if document indeed improved in readability
      (we know this is subjective, but yet may be helpful)
    - collects all the information at the end and
      makes a report (maybe we should have a template)

  - AD (with Bert Wijnen) keeps track of
    - elapsed times for various steps
    - ensures proper checkin into source code
    - manages the process, i.e. hand doc from WG to 
      RFC-Editor, back to WG and further processing
    - collects all the information at the end and
      makes a report (maybe we should have a template)
  
  Maybe some of thses measurements can be automated or done
  by tools. Tools team, any help you can give?

We understand that between now and Vancouver we can probably 
only get the first steps of the process, namely, have the
RFC-Editor do the copy-edit run, have the WG do WG Last Call,
and have the AD process the document through IETF Last Call
and IESG approval. The later steps and data collection
will happen after Vancopuver. But we should have some data
about the initial steps in Vancouver. Based on that info
we can decide if we need to do more experimentation
and if so if we need to change the process/measurements.

Bert