Re: pkg_regress usage and errors

[Richard PALO <richard.palo%free.fr@localhost>]

Le 15/05/12 14:17, Jeremy C. Reed a écrit :
> 1. Running "make" in pkgsrc/regress results in numerous ERRORs.
>
> I see the pkgsrc guide mentions pkg_regress.
>
> We should have a README in pkgsrc/regress/README that explains it also.
> Is it okay to have errors when running make? Can someone please explain
> this in a README (even if it just points to the pkgsrc guide)?
>
> Should we have a make all target in pkgsrc/regress that has
> understandable or informational output?
>
> 2. Running pkg_regress results in:
>
> sunflower:regress$ /usr/pkg/sbin/pkg_regress
> ERROR: Expected "USE_TOOLS\+=perl" in the output, but it is not there.
> ERROR: Expected exit code 0, but got 1.
> ERROR: Expected exit code 0, but got 1.
> ERROR: Expected exit code 0, but got 1.
> ERROR: Expected exit code 0, but got 1.
> ===> Cleaning for compiler-1.1
> ERROR: Expected "^Cplain.*:.*Cplain" in the output, but it is not there.
> ===> Cleaning for compiler-1.1
> ERROR: Expected "Required package bison" in the output, but it is not
> there.
> ERROR: Expected "Circular dependency detected" in the output, but it is
> not there.
> ERROR: Didn't expect "bison-must-be-deinstalled" in the output, but
> found it.
> ERROR: Expected "^The variable MAKE_ENV remains unchanged\.$" in the
> output, but it is not there.
> ERROR: Expected exit code 0, but got 1.
> ERROR: Expected exit code 0, but got 1.
> ERROR: Expected exit code 0, but got 1.
> ERROR: Expected "^@comment Matched test file.$" in the output, but it is
> not there.
> ERROR: Expected "^print-plist/foo$" in the output, but it is not there.
> ERROR: Expected "^@comment Matched test directory.$" in the output, but
> it is not there.
> ERROR: Expected "^@dirrm print-plist$" in the output, but it is not
> there.
> Password:
> Sorry, try again.
> Password:
> ERROR: Expected exit code 0, but got 1.
> Tests failed:  barrier bootstrap-install-sh buildlink-libtool
> buildlink-transform buildlink-unwrap compiler ignore-tools
> make-env-phases make-quoting pkg-options print-plist tools
>
> Statistics:
>    1 passed, 12 failed
>
> Is the above what others are seeing?
>
> pkg_regress needs a manual page.
>
> Is any NetBSD system automated to run the pkg_regress tests?  If not,
> can we do this?
>
> What other regression tests should we add?
>
> I'd like to prove that a selection of packages work to install, run, and
> deinstall.  It would also be nice to simulate some upgrade type
> scenario.

Apparently not much has changed since this posting.
For example, regress/tools seems to only need LANG=C or LANG=posix in 
order to get it to pass directly...
Some errors, such is from regress/make-quoting are actually 
disconcerting if the constructs expected to pass are indeed used:
> ===> Building for regress-make-1.0
> Testing bmake(1) version 20110606
> Running testcase bug2
> Running testcase dblquote
> Running testcase for
> Running testcase ltarget
> Running testcase nestfor
> Running testcase quoting
> Running testcase sglquote
> 1c1
> < \a   b z
> ---
> > \\a   b z
> *** Error code 1
>
> Stop.