Automated email notifications of NetBSD-current build failures

To: current-users%NetBSD.org@localhost
Subject: Automated email notifications of NetBSD-current build failures
From: Andreas Gustafsson <gson%gson.org@localhost>
Date: Wed, 2 Nov 2011 20:03:18 +0200

Hi all,

As you may have noticed, periodic automated builds and tests of
NetBSD-current are now being run on a dedicated TNF server,
babylon5.NetBSD.org.  Reports from these builds and tests are
published at the following URLs:

  http://releng.NetBSD.org/b5reports/i386/
  http://releng.NetBSD.org/b5reports/amd64/
  http://releng.NetBSD.org/b5reports/sparc/

The scripts generating those web pages also have the capability to
generate email notifications of build failures.  I have been testing
this facility for some two years now by having the notifications sent
only to myself, and I think it is now mature enough for production
use.

This would mean sending the notifications to a public mailing list.
I propose using the current-users list, because there is already an
established practice of manually reporting -current build breakage
there, and the automated notifications are not that different from the
reports that are already being sent by hand, just hopefully more
timely, reliable, and complete.

During the testing phase, the automated reports have typically arrived
in my mailbox many hours before any corresponding manual report on
current-users, and they have had a smaller proportion of false reports
because automated builds eliminate many common human errors that can
cause a build to fail, such as stale files in the build tree.

Nevertheless, there are some limitations to keep in mind:

 - Although false positives have been rare compared to manual reports,
   they do sometimes occur.  In particular, there have been a couple
   of cases where spurious build failures were reported because
   mirroring the CVS repository using rsync can result in a repository
   where only some of the files affected by a multi-file commit have
   been updated.

 - Notifications are generated for build failures only, not for
   failures at the install, boot, or ATF test stage.  The ATF
   tests in particular still have too many non-deterministic
   failures to allow automated reporting.

 - Notifications are currently only generated for a single NetBSD
   port (i386).

 - Notifications are only generated when the build transitions from working
   to broken; additional breakage committed while the build is already
   broken is not automatically reported.

Here is an example of what an automatically generated build failure
report might look like (this one is for the most recent failure at the
time of writing):

  From: NetBSD Test Fixture <bracket%NetBSD.org@localhost>
  To: current-users%NetBSD.org@localhost
  Subject: NetBSD-current/i386 build failure
  Date: Wed,  2 Nov 2011 00:56:11 +0000 (UTC)

  This is an automatically generated notice of a NetBSD-current/i386
  build failure.

  The failure occurred on babylon5.NetBSD.org, a NetBSD/amd64 host,
  using sources from CVS date 2011.11.02.00.10.08.

  An extract from the build.sh output follows:

      --- dependall-mtree ---
      --- mtree.lo ---
      #   compile  mtree/mtree.lo
      cc -O -DNO_MD5 -DNO_RMD160 -DNO_SHA1 -DNO_SHA2 
-I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat 
-I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat/include 
-I/bracket/i386/work/2011.11.02.00.10.08/src/tools/mtree/../compat 
-DHAVE_NBTOOL_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -DMTREE 
-I/bracket/i386/work/2011.11.02.00.10.08/src/sbin/mknod -c -o mtree.lo.o    
/bracket/i386/work/2011.11.02.00.10.08/src/tools/mtree/../../usr.sbin/mtree/mtree.c
      --- dependall-uudecode ---
      #    create  uudecode/uudecode.d
      /bracket/i386/work/2011.11.02.00.10.08/tools/bin/nbhost-mkdep -f 
uudecode.d --     -DNO_BASE64 
-I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat 
-I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat/include 
-I/bracket/i386/work/2011.11.02.00.10.08/src/tools/uudecode/../compat 
-DHAVE_NBTOOL_CONFIG_H=1 -D_FILE_OFFSET_BITS=64  
/bracket/i386/work/2011.11.02.00.10.08/src/tools/uudecode/../../usr.bin/uudecode/uudecode.c
      --- dependall-nbperf ---
      --- nbperf ---
      #      link  nbperf/nbperf
      cc -O -I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat 
-I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat/include 
-I/bracket/i386/work/2011.11.02.00.10.08/src/tools/nbperf/../compat 
-DHAVE_NBTOOL_CONFIG_H=1 -D_FILE_OFFSET_BITS=64   -o nbperf nbperf.lo 
nbperf-bdz.lo nbperf-chm.lo nbperf-chm3.lo graph2.lo graph3.lo 
-L/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat -lnbcompat -lz
      nbperf.lo: In function `main':
      nbperf.c:(.text+0x543): undefined reference to `getline'
      --- dependall-mtree ---
      --- spec.lo ---
      #   compile  mtree/spec.lo
      cc -O -DNO_MD5 -DNO_RMD160 -DNO_SHA1 -DNO_SHA2 
-I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat 
-I/bracket/i386/work/2011.11.02.00.10.08/obj/tools/compat/include 
-I/bracket/i386/work/2011.11.02.00.10.08/src/tools/mtree/../compat 
-DHAVE_NBTOOL_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -DMTREE 
-I/bracket/i386/work/2011.11.02.00.10.08/src/sbin/mknod -c -o spec.lo.o    
/bracket/i386/work/2011.11.02.00.10.08/src/tools/mtree/../../usr.sbin/mtree/spec.c
      --- dependall-nbperf ---

  The following commits were made between the last successful build and the 
failed build:

     2011.11.01.21.16.28 joerg src/tools/Makefile,v 1.150
     2011.11.01.21.16.29 joerg src/share/mk/bsd.own.mk,v 1.690
     2011.11.01.21.16.29 joerg src/tools/nbperf/Makefile,v 1.3
     2011.11.01.21.21.32 joerg src/sys/arch/x86/include/cpu.h,v 1.40
     2011.11.01.21.37.08 uwe src/external/gpl3/gdb/dist/gdb/shnbsd-tdep.c,v 1.9
     2011.11.01.21.47.53 uwe src/external/gpl3/gdb/dist/gdb/sh-tdep.c,v 1.2
     2011.11.01.22.30.32 pgoyette src/usr.sbin/i2cscan/i2cscan.c,v 1.3
     2011.11.01.22.36.53 pgoyette src/tests/lib/libc/net/Makefile,v 1.6
     2011.11.01.22.36.53 pgoyette src/tests/lib/libc/net/gen_ether_subr,v 1.1
     2011.11.01.22.36.53 pgoyette src/tests/lib/libc/net/t_ether_aton.c,v 1.1
     2011.11.01.22.37.42 pgoyette src/distrib/sets/lists/tests/mi,v 1.413
     2011.11.01.23.07.34 pgoyette src/regress/sys/net/Makefile,v 1.3
     2011.11.01.23.07.34 pgoyette 
src/regress/sys/net/ether_aton_r/Attic/Makefile,v 1.3
     2011.11.01.23.07.34 pgoyette 
src/regress/sys/net/ether_aton_r/Attic/autogen,v 1.3
     2011.11.01.23.07.34 pgoyette 
src/regress/sys/net/ether_aton_r/Attic/main.c,v 1.4
     2011.11.01.23.11.55 joerg src/tools/Makefile,v 1.151
     2011.11.01.23.17.59 wiz src/share/man/man9/skpc.9,v 1.2
     2011.11.01.23.18.07 wiz src/share/man/man9/scanc.9,v 1.2
     2011.11.02.00.10.08 uwe src/share/mk/bsd.own.mk,v 1.691

  More log files can be found at:

      
http://releng.NetBSD.org/b5reports/i386/commits-2011.11.html#2011.11.02.00.10.08

Any comments, suggestions, support for, or objections to turning this on?
-- 
Andreas Gustafsson, gson%gson.org@localhost

Follow-Ups:
- Re: Automated email notifications of NetBSD-current build failures
  - From: Bernd Ernesti
- Re: Automated email notifications of NetBSD-current build failures
  - From: Andreas Gustafsson
- Re: Automated email notifications of NetBSD-current build failures
  - From: Thomas Mueller
- Re: Automated email notifications of NetBSD-current build failures
  - From: Matthew Mondor
- Re: Automated email notifications of NetBSD-current build failures
  - From: Greg Troxel
- Re: Automated email notifications of NetBSD-current build failures
  - From: Jukka Ruohonen
- Re: Automated email notifications of NetBSD-current build failures
  - From: Julio Merino
- Re: Automated email notifications of NetBSD-current build failures
  - From: Jeff Rizzo

Prev by Date: Missing getline.c in tools/compat/Makefile
Next by Date: Re: Automated email notifications of NetBSD-current build failures
Previous by Thread: Missing getline.c in tools/compat/Makefile
Next by Thread: Re: Automated email notifications of NetBSD-current build failures
Indexes:

Home | Main Index | Thread Index | Old Index