Re: PR/60496 CVS commit: src/tests/lib/libc/locale

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Wed, 28 Oct 2026 17:52:47 +0000
    From:        Taylor R Campbell <riastradh%NetBSD.org@localhost>
    Message-ID:  <20261028175248.A282385606%mail.netbsd.org@localhost>

I am not including gnats-bugs as a recipient of this message, there
is no point cluttering PRs about real bugs with meta-discussions
on how ATF tests should be handled.

  | Please mark the tests as xfail.

Pointless now, they no should no longer fail.   If they do, that would
be a real (new) bug that needs looking into.

  | Verifying that they actually fail is _exactly the point_ of marking
  | them as xfail:

I disagree.   I see xfail as being intended for tests that continually
fail, we know why they fail, but for whatever reason, we're unable or
unwilling to do anything about it.

That leaves the tests that fail as ones that we mostly concentrate on fixing.

Those in xfail can be because a bug cause just cannot be found - ie: we
know the code isn't acting as it should, but it is all so complex that
no-one can work out what the actual problem is.

Or it can be that we cannot fix a problem, as the fix would cause other
more serious problems (perhaps breaking compat with older code, or similar)
which is considered a bigger issue than fixing what is clearly a bug.

Or it can be that the test is simply broken, and testing for something
either not guaranteed to happen, or which actually would be a bug if it
did -- as in sometimes where we write tests which simply verify that things
keep operating as they always have - and it turns out that the way that
always happened was wrong, and is fixed, but no-one has had the time/energy
to also fix the test.   I have one like that for sh, where fixing the test
would actually be easy (just delete it), I just don't want to, as a lot of
work went into making that test, and testing every conceivable point where
problems could arise - then the whole thing was just rendered pointless.

Or perhaps other reasons.

In any case, I see no reason for the behaviour to be different when a new
test is added to test that something works as it should, expecting that
all will be OK, and when a new test is added because it is expected that
there is a bug, and confirmation of that is required.

One of the tests I just added, which wasn't marked xfail, was actually just
that.  I looked at the code, decided that it wouldn't work, and changed the
code, then added a test to verify that things worked as they should.   The
test (and my local testing of it) verified that the changed code worked, but
nothing had ever verified that the change was actually needed, that is that
the code had been broken before I altered it.

Now in this case, I did actually (after the new test existed in my local
tree) undo the source modification, rebuild, and run the test ... and it
still succeeded.   Now that turned out to be because I just mv'd the fixed
code out of the way, mv'd the original back, and then did an update
build ... easy to see the problem with that technique (with hindsight)
and I eventually did things properly, and the test did fail for me as I
expected it would, but that is beside the point.

Overall I see the large number of xfail marked tests, with the number of
them continually growing, as one of the projects biggest failings - it is
indicating that we know we have problems, but we're unable/unwilling to
fix them.    That's bad.   So I simply will not purposely add to that.
Even temporarily.

A test for something that should work, which doesn't work, should fail.

It really is as simple as that.

kre