kern/58393: psshfs mounted FS malfunctioning

[reinoud%13thmonkey.org@localhost]

>Number:         58393
>Category:       kern
>Synopsis:       psshfs mounted FS malfunctioning
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Jul 03 10:55:00 +0000 2024
>Originator:     Reinoud Zandijk
>Release:        NetBSD-10 from April 20th 2024
>Organization:
NetBSD
	
>Environment:
System: NetBSD gorilla 10.99.10 NetBSD 10.99.10 (GENERIC) #2: Thu Jun  6 21:26:08 CEST 2024
Architecture: x86_64
Machine: amd64
>Description:

When mounting with mount_psshfs the resulting mount is OK but copying small
files fail and files can get corrupted. Directory listings can get odd
results. I am putting them into one PR for I think they are related.

Note that both sftp and scp work fine and don't give errors for file
transfers.

On copying small files, it fails at the same place over and over again. When
corrupting it seems to be related to incremental writing with sometimes the
same bad piece duplicated multiple times. The issues fail at the same place
with a NetBSD server or a Linux server.

Example of a copy action, run as normal user:

# mount_psshfs reinoud@remote: /tmp/mnt
# ktrace -id cp -Rv ~/UDF /tmp/mnt/UDF-cpy
/home/reinoud/UDF -> /tmp/mnt/UDF-cpy/UDF
/home/reinoud/UDF/doc -> /tmp/mnt/UDF-cpy/UDF/doc
/home/reinoud/UDF/doc/dcn-new.doc -> /tmp/mnt/UDF-cpy/UDF/doc/dcn-new.doc
/home/reinoud/UDF/doc/NOTE -> /tmp/mnt/UDF-cpy/UDF/doc/NOTE
/home/reinoud/UDF/doc/OSS_20060428 -> /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428
/home/reinoud/UDF/doc/OSS_20060428/SampleCode -> /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428/SampleCode
...
/home/reinoud/UDF/doc/OSS_20060428/VDD -> /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428/VDD
/home/reinoud/UDF/doc/OSS_20060428/VDD/api -> /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428/VDD/api
cp: /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428/VDD/api/api.bld: Operation not permitted
cp: /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428/VDD/api/api.gpj: Operation not permitted
/home/reinoud/UDF/doc/OSS_20060428/VDD/api/vdd.c -> /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428/VDD/api/vdd.c
/home/reinoud/UDF/doc/OSS_20060428/VDD/api -> /tmp/mnt/UDF-cpy/UDF/doc/OSS_20060428/VDD/api

It always fails with the same files and always with "Operation not permitted"
on a NetBSD server and "Permission denied" on a Linux server. If mounted as
root user it fails exactly the same at the same place. The files in question
are as for as I can see always tiny files:

-r--r--r--  1 reinoud  staff  58 Aug  4  2013 UDF/doc/OSS_20060428/VDD/api/api.bld
-r--r--r--  1 reinoud  staff  75 Aug  4  2013 UDF/doc/OSS_20060428/VDD/api/api.gpj
-rw-r--r--  1 reinoud  staff  26699 Aug  4  2013 /home/reinoud/UDF/doc/OSS_20060428/VDD/api/vdd.c

ktrace shows:
...
 10084  10084 cp       CALL  __stat50(0x1f6c05290,0x7f7fffc47790)
 10084  10084 cp       NAMI  "/tmp/mnt/UDF-cpy/doc/OSS_20060428/VDD/api/api.bld"
 10084  10084 cp       RET   __stat50 -1 errno 2 No such file or directory
 10084  10084 cp       CALL  open(0x78b657dce000,0,0)
 10084  10084 cp       NAMI  "/home/reinoud/UDF/doc/OSS_20060428/VDD/api/api.bld"
 10084  10084 cp       RET   open 3
 10084  10084 cp       CALL  open(0x1f6c05290,0x601,0x8124)
 10084  10084 cp       NAMI  "/tmp/mnt/UDF-cpy/doc/OSS_20060428/VDD/api/api.bld"
 10084  10084 cp       RET   open -1 errno 1 Operation not permitted
 10084  10084 cp       CALL  unlink(0x1f6c05290)
 10084  10084 cp       NAMI  "/tmp/mnt/UDF-cpy/doc/OSS_20060428/VDD/api/api.bld"
 10084  10084 cp       RET   unlink 0
 10084  10084 cp       CALL  open(0x1f6c05290,0x601,0x8124)
 10084  10084 cp       NAMI  "/tmp/mnt/UDF-cpy/doc/OSS_20060428/VDD/api/api.bld"
 10084  10084 cp       RET   open -1 errno 1 Operation not permitted
 10084  10084 cp       CALL  write(2,0x7f7fffc46e60,4)
 10084  10084 cp       GIO   fd 2 wrote 4 bytes
       "cp: "
...

Just listing a directory can give issues too and are repeatable too:

# cd /tmp/mnt/
# ktrace -id ls -al 
total 36028797018964015
ls: ./.: Operation not supported
drwxr-xr-x  10 reinoud     staff         512 Jul  3 11:59 .
drwxrwxrwt  32 root        wheel        1584 Jul  3 11:59 ..
ls: ./UDF Virtual Plug-fest 2009.ods: Operation not supported
-rw-r--r--   1 reinoud     staff       12519 Jul 16  2009 UDF Virtual Plug-fest 2009.ods
..

The kdump shows:
  9135   9135 ls       CALL  write(1,0x78693cbfb000,0x18)
  9135   9135 ls       GIO   fd 1 wrote 24 bytes
       "total 36028797018964015\n"
  9135   9135 ls       RET   write 24/0x18
  9135   9135 ls       CALL  lpathconf(0x7f7fffbdf800,0x11)
  9135   9135 ls       NAMI  "./."
  9135   9135 ls       RET   lpathconf -1 errno 45 Operation not supported

When executing ./configure in the directory it fails but with different errors
each time. Sometimes it does succeed.

...
checking for struct dirent.d_namlen... no
checking for dev/scsipi/scsipi_all.h... yes
configure: creating ./config.status
./config.status: 872: Syntax error: ";;" unexpected (expecting "done")
...

This is due to the config.status file being corrupted. Inspection of the
config.status file can show a repeated block of bad data, sometimes binary.
This file is AFAIK created incementally with small pieces and can thus suffer
from the same small-files issue.

The bad config.status seems to miss several pieces when compared to a
reference one and at times get bad (stale?) data.

	
>How-To-Repeat:
See above; download UDFclient.0.8.9.tgz, unpack it and mount the directory
remotely on a NetBSD machine. Or copy the source to the remote machine.

	
>Fix:

Unknown.
Use sftp or scp to copy files around and only use it to copy big files.

pathconf(2)/lpathconf(2) could be implemenred or fixed for psshfs.