Re: tar: Can't translate pathname 'xxx' to UTF-8

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Fri, 25 Sep 2026 20:32:06 +0100
> From: Jonathan Perkin <jperkin%pkgsrc.org@localhost>
> 
> * On 2026-09-25 at 20:11 BST, Taylor R Campbell wrote:
> 
> >You may be thinking of this thread, and of its resolution of setting
> >LC_CTYPE.UTF-8 in EXTRACT_ENV (on NetBSD), after I fixed a bug in
> >NetBSD's libarchive/bsdtar so that it would not silently suppress an
> >underlying error:
> 
> FWIW this is still the cause of the top breakage on at least SmartOS, 
> and presumably will be the same on any platform that doesn't ship a 
> native bsdtar without this handling.
> 
> I don't really understand why it doesn't JFDI and extract the bytes, and 
> am curious about what happens if we kludge LC_CTYPE to UTF-8 and then 
> have an archive which contains filenames that aren't valid UTF-8?

Quoting from the earlier thread
(https://mail-index.netbsd.org/tech-userlevel/2026/08/24/msg015058.html,
https://mail-index.netbsd.org/tech-userlevel/2026/08/24/msg015063.html):

	The archive in question is in POSIX pax interchange format, which has
	_two paths_ for the file in question (written here in vis(3)
	notation):

	- (pax extended header `path' attribute, [in whatever encoding
	  is set by hdrcharset, or UTF-8 by default if not set])

	  meson_python-0.19.0/tests/packages/encoding/\xe3\x83\x86\xe3\x82\xb9\xe3\x83\x88.py

	  Reference:
	  https://pubs.opengroup.org/onlinepubs/9799919799/utilities/pax.html#tag_20_94_13_03

	- (ustar record path field, exactly 100 octets, unspecified encoding)

	  meson_python-0.19.0/tests/packages/encoding/???.py

	  Reference:
	  https://pubs.opengroup.org/onlinepubs/9799919799/utilities/pax.html#tag_20_94_13_06

	POSIX prescribes of pax(1) that, for the `path' extended header
	attribute:

		The pax utility shall translate the pathname of the file from
		the encoding in the header to the character set appropriate
		for the local file system.

To answer your question, there are two scenarios you might be
concerned with:

1. The input hdrcharset is not UTF-8.  In this case, bsdtar will
   translate the path from whatever hdrcharset specifies -- if bsdtar
   knows how -- into UTF-8.  If there is no invertible translation,
   bsdtar will fail noisily.

1. The input _claims_ to be UTF-8, or _makes no claim_ and therefore
   implicitly takes the default of UTF-8, and is invalid UTF-8.  In
   this case, bsdtar will fail noisily.

Caveat: The bsdtar shipped in NetBSD _base_ is happy to do iconv
conversions (if NetBSD knows about them).  The bsdtar shipped in
_pkgsrc_ is built without any iconv support, because it is a
statically linked bootstrap thing and iconv complicates the bootstrap
story yadda yadda yadda.

There is no scenario, however, where bsdtar takes the pax extended
header path bytes verbatim.  The bsdtar/libarchive maintainers might
be willing to contemplate (say) a --pax-extended-header-path-verbatim
option but I don't know; I haven't asked them.

> Why is this suddenly a problem now?

bouyer@'s issue has been happening for three years but also appears to
be a red herring (it is not actually the cause of a build failure --
though is not _harmless_ because it causes warning fatigue and wild
goose chases, and we should maybe track it down and fix it).

wiz@'s issue, and the motivation for EXTRACT_ENV+=LC_CTYPE=UTF-8, is
happening now because libarchive _used to_ have a bug where it
misinterpreted the POSIX standard return value of iconv(3) and
_silently did non-invertible character substitutions_ in the output,
when converting from (e.g.) UTF-8 to US-ASCII.

So when run with LC_CTYPE=C, bsdtar would silently substitute `?' or
similar in paths for every non-ASCII byte.  That is, bsdtar would
silently corrupt the paths, and I think everyone would agree silent
corruption is generally bad.

(This happened on NetBSD but not on GNU systems because GNU iconv(3)
is not POSIX-compliant, and not on FreeBSD because there's an #ifdef
FreeBSD turn_on_incompatible_gnu_iconv() #endif; libarchive assumed
GNU iconv(3) semantics instead of POSIX iconv(3) semantics.)

That bug has now been fixed by making libarchive noisily fail rather
than silently corrupt in this case, under POSIX semantics:

Upstream bug report: https://github.com/libarchive/libarchive/issues/3413
Upstream change: https://github.com/libarchive/libarchive/pull/3416

> Why does using GNU tar work fine?

GNU tar _does_ take the extended header path verbatim, _if_ conversion
to UTF-8 fails.  But I don't think that semantics is reliable: the GNU
tar maintainers appear to consider it a bug, and may intend to fix it
at some point.  Quoting from the earlier mail thread:

	I reviewed the GNU tar code, and I believe that:

	1. this is, in fact, what GNU tar does, but

	2. the GNU tar authors may consider it a bug and intend to fix or at
	   least change it in some way later.

	So I'm not sure the GNU tar behaviour is any more reliable than the
	earlier bsdtar behaviour.


	This is the subroutine in GNU tar that decodes the pax extended header
	`path' attribute (among other things), from src/xheader.c in v1.35
	(commit e545d446dfe6564265cdf4186641ee76f4acc7fa):

	   1024 static void
	   1025 decode_string (char **string, char const *arg)
	   1026 {
	   1027   if (*string)
	   1028     {
	   1029       free (*string);
	   1030       *string = NULL;
	   1031     }
	   1032   if (!utf8_convert (false, arg, string))
	   1033     {
	   1034       /* FIXME: report error and act accordingly to --pax invalid=UTF-8 */
	   1035       assign_string (string, arg);
	   1036     }
	   1037 }

	https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/xheader.c?h=v1.35&id=e545d446dfe6564265cdf4186641ee76f4acc7fa

	(assign_string under the hood boils down to strdup.)

So:

- bsdtar tries to adhere to the letter of the standard for
  interpretation of pax archives, and just doesn't have an option to
  do what you want.

- GNU tar does what you want but the maintainers may view what you
  want as a bug and might fix it later.