Re: PR/60496 CVS commit: src/tests/lib/libc/locale

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Thu, 29 Oct 2026 14:05:35 +0700
> From: Robert Elz <kre%munnari.OZ.AU@localhost>
> 
>     Date:        Wed, 28 Oct 2026 17:52:47 +0000
>     From:        Taylor R Campbell <riastradh%NetBSD.org@localhost>
>     Message-ID:  <20261028175248.A282385606%mail.netbsd.org@localhost>
> 
>   | Verifying that they actually fail is _exactly the point_ of marking
>   | them as xfail:
> 
> I disagree.   I see xfail as being intended for tests that continually
> fail, we know why they fail, but for whatever reason, we're unable or
> unwilling to do anything about it.

No, xfail is intended for tests that are expected to fail, nothing
more, nothing less.

It does not imply anything about duration of failures or about ability
or willingness to fix bugs.  That information may be recorded in the
bug tracker.  (A test that is marked xfail should, however, have an
associated bug report.)

If an xpass test fails or an xfail test passes, that is supposed to be
surprising and to merit an alarm soliciting attention.  You should not
add tests you expect to fail without marking them as `expected to
fail'.

> That leaves the tests that fail as ones that we mostly concentrate on fixing.

These are the ones that need developer attention to analyze and either
fix or at least turn into bug reports with the analysis.

> In any case, I see no reason for the behaviour to be different when a new
> test is added to test that something works as it should, expecting that
> all will be OK, and when a new test is added because it is expected that
> there is a bug, and confirmation of that is required.

You are wasting others' time by creating spurious alarms of new test
failures.  You can get exactly the same confirmation without raising
spurious alarms by simply using xfail markers as intended.

> Overall I see the large number of xfail marked tests, with the number of
> them continually growing, as one of the projects biggest failings - it is
> indicating that we know we have problems, but we're unable/unwilling to
> fix them.    That's bad.   So I simply will not purposely add to that.
> Even temporarily.

The problem isn't the xfail markers; the problem is the bugs!

Adding xfail tests is a step on the way toward fixing the bugs, by
recording executable reproducers.  We could avoid xfail markers by
simply not having tests for the bugs, but that would be worse.  We
could similarly avoid a large number of open bug reports simply by
closing them, but that would be worse.