pkg/39432: Tests performed against installed packages rather than newly-built replacements

[srcshelton%gmail.com@localhost]

>Number:         39432
>Category:       pkg
>Synopsis:       Tests performed against installed packages rather than 
>newly-built replacements
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Aug 29 11:50:01 +0000 2008
>Originator:     Stuart Shelton
>Release:        pkgsrc latest from CVS
>Organization:
>Environment:
IRIX 6.5
>Description:

There seems to be a number of packages, in this case png-1.2.31, where the 
package test-suite is executed against the older installed version of the 
package rather than the newly-built upgrade.

For example, 'bmake clean && bmake test && bmake upgrade' results in:

creating pngtest
/usr/bsd/bin/bmake  check-TESTS
Testing libpng version 1.2.31
   with zlib   version 1.2.3

 libpng version 1.2.30 - August 15, 2008
   Copyright (c) 1998-2008 Glenn Randers-Pehrson
   Copyright (c) 1996-1997 Andreas Dilger
   Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc.
 library (10230): libpng version 1.2.30 - August 15, 2008

 pngtest (10231): libpng version 1.2.31 - August 21, 2008
 sizeof(png_struct)=808, sizeof(png_info)=288
Warning: versions are different between png.h and png.c
  png.h version: 1.2.31
  png.c version: 1.2.30

Testing ./pngtest.png:Testing ./pngtest.png:
 Pass 0: rwrwrwrwrwrwrwrwrw
 Pass 1: rwrwrwrwrwrwrwrwrw
 Pass 2: rwrwrwrwrwrwrwrw
 Pass 3: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
 Pass 4: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
 Pass 5: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
         rwrwrwrw
 Pass 6: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
         rwrwrwrwrwTesting ./pngtest.png:
 PASS (9782 zero samples)
 Filter 0 was used 21 times
 Filter 1 was used 15 times
 Filter 2 was used 52 times
 Filter 3 was used 10 times
 Filter 4 was used 33 times
 tIME = 7 Jun 1996 17:58:08 +0000
libpng FAILS test
FAIL: test-pngtest.sh
========================================================
1 of 1 tests failed
Please report to png-mng-implement%lists.sourceforge.net@localhost
========================================================
*** Error code 1

Stop.

... whereas 'bmake clean && bmake upgrade && bmake test' gives the desired 
result of:

===> test-message [png-1.2.31] ===> Testing for png-1.2.31
/usr/bsd/bin/bmake  pngtest
`pngtest' is up to date.
/usr/bsd/bin/bmake  check-TESTS
Testing libpng version 1.2.31
   with zlib   version 1.2.3

 libpng version 1.2.31 - August 21, 2008
   Copyright (c) 1998-2008 Glenn Randers-Pehrson
   Copyright (c) 1996-1997 Andreas Dilger
   Copyright (c) 1995-1996 Guy Eric Schalnat, Group 42, Inc.
 library (10231): libpng version 1.2.31 - August 21, 2008

 pngtest (10231): libpng version 1.2.31 - August 21, 2008
 sizeof(png_struct)=808, sizeof(png_info)=288
Testing ./pngtest.png:
 Pass 0: rwrwrwrwrwrwrwrwrw
 Pass 1: rwrwrwrwrwrwrwrwrw
 Pass 2: rwrwrwrwrwrwrwrw
 Pass 3: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
 Pass 4: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
 Pass 5: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
         rwrwrwrw
 Pass 6: rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
         rwrwrwrwrw
 PASS (9782 zero samples)
 Filter 0 was used 21 times
 Filter 1 was used 15 times
 Filter 2 was used 52 times
 Filter 3 was used 10 times
 Filter 4 was used 33 times
 tIME = 7 Jun 1996 17:58:08 +0000
libpng passes test
PASS: test-pngtest.sh
==================
All 1 tests passed
==================

>How-To-Repeat:

>Fix:

A test-suite isn't a great deal of use if it can't be executed until after the 
(potentially broken) package is installed...

Even if this problem can't be fixed, surely a warning could be added to 
packages' Makefiles if it is known that their test suite will not complete 
successfully unless the updated package is first installed.