kern/40501: NTFS directory listings missing some files

[jdarrow%netbsd.org@localhost]

>Number:         40501
>Category:       kern
>Synopsis:       NTFS directory listings missing some files
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Jan 28 09:00:01 +0000 2009
>Originator:     John Darrow
>Release:        NetBSD 4.0_STABLE (also seen with NetBSD 5.0_BETA CD)
>Organization:
        Fairly dis-
>Environment:
System: NetBSD jdarrowpiii 4.0_STABLE NetBSD 4.0_STABLE (JDARROW) #0: Sun Aug 3 
18:18:58 CDT 2008 
jjjjjjj%jdarrowcel.jdarrow.dyndns.org@localhost:/var/obj.release-4/i386/sys/arch/i386/compile/JDARROW
 i386
Architecture: i386
Machine: i386
>Description:

While looking at an NTFS partition, a ls in a certain directory did not
show the first (sorted case-insensitively) few files or subdirectories
in that directory; nor does an 'echo *' print those first few entries.
An explicit 'ls entryname' does, however, see the file/subdirectory,
which can also be opened normally.  (The behavior is similar to that seen
when a directory's mode has the x bit set but not the r bit, but only
fir the first few entries; all the later entries are visible normally.)

This behavior was first seen when vnd-mounting a disk image under
4.0_STABLE; however, a NetBSD 5.0_BETA CD also had the same problem
reading from the actual drive in the machine the image had been taken
from.  A Windows XP CD, booted to the "Recovery Console" on that
machine, could see all the entries in that directory normally.
>How-To-Repeat:
[some filenames obscured for privacy]
$ fdisk vnd3
Disk: /dev/rvnd3d
NetBSD disklabel disk geometry:
cylinders: 76319, heads: 64, sectors/track: 32 (2048 sectors/cylinder)
total sectors: 156301488

BIOS disk geometry:
cylinders: 1024, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 156301488

Partition table:
0: Dell PowerEdge Server utilities (sysid 222)
    start 63, size 96327 (47 MB, Cyls 0-5)
1: NTFS, OS/2 HPFS, QNX2 or Advanced UNIX (sysid 7)
    start 96390, size 146769840 (71665 MB, Cyls 6-9141), Active
2: CP/M or Concurrent CP/M or Concurrent DOS or CTOS (sysid 219)
    start 146866230, size 9430155 (4605 MB, Cyls 9142-9728)
3: <UNUSED>
Drive serial number: 1101734678 (0x41ab2316)
$ disklabel vnd3
# /dev/rvnd3d:
type: vnd
disk: vnd
label: fictitious
flags:
bytes/sector: 512
sectors/track: 32
tracks/cylinder: 64
sectors/cylinder: 2048
cylinders: 76319
total sectors: 156301488
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0 

7 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 d: 156301488         0     unused      0     0        # (Cyl.      0 -  76319*)
 e:     96327        63    unknown                     # (Cyl.      0*-     47*)
 f: 146769840     96390       NTFS                     # (Cyl.     47*-  71712*)
 g:   9430155 146866230    unknown                     # (Cyl.  71712*-  76316*)
disklabel: boot block size 0
disklabel: super block size 0
$ mount |grep vnd3
/dev/vnd3f on /ccccc type ntfs (read-only, local)
$ df -k /ccccc
Filesystem  1K-blocks      Used     Avail Capacity  Mounted on
/dev/vnd3f   73384919  68937387   4447532    93%    /ccccc
$ ls -l /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents
total 2915
drwxr-xr-x  0 root  wheel        0 Dec 31 10:22 Desktop Shortcuts 12-31
-rwxr-xr-x  0 root  wheel      788 Jun 14  2007 Direct Loan Interest.lnk
drwxr-xr-x  0 root  wheel        0 Aug 18 18:43 Downloaded Games
drwxr-xr-x  0 root  wheel        0 Jan 21  2008 Downloaded Program Updates
drwxr-xr-x  0 root  wheel        0 Sep 11  2007 Ebay
drwxr-xr-x  0 root  wheel        0 Dec 31 10:20 Essential Files
drwxr-xr-x  0 root  wheel        0 Dec 30 13:17 Excel docs
drwxr-xr-x  0 root  wheel        0 May 14  2008 Family Tree Research
drwxr-xr-x  0 root  wheel        0 Dec 30 20:06 Financial
-rwxr-xr-x  0 root  wheel      581 Sep 28  2006 Graphics projects files.lnk
-rwxr-xr-x  0 root  wheel      541 Jun  8  2006 Install Files.lnk
drwxr-xr-x  0 root  wheel        0 Dec 30 17:37 Insurance & Medical
drwxr-xr-x  0 root  wheel        0 Feb 12  2008 ItsDeductible2006
drwxr-xr-x  0 root  wheel        0 Jul  1  2008 Jjjj
-rwxr-xr-x  0 root  wheel   204800 Dec 29 21:46 LIST.DOC
drwxr-xr-x  0 root  wheel        0 Apr 17  2008 Labels & Signs
drwxr-xr-x  0 root  wheel        0 Jun 27  2008 Letters
drwxr-xr-x  0 root  wheel        0 Oct 31 10:36 Manuals
drwxr-xr-x  0 root  wheel        0 Sep 26 11:54 Map Files
-rwxr-xr-x  0 root  wheel     1099 Dec 17 10:20 Mogul User Guide.lnk
-rwxr-xr-x  0 root  wheel     1371 Dec 15 09:21 Movie List.txt
drwxr-xr-x  0 root  wheel        0 Sep  7 11:01 Moving Information
drwxr-xr-x  0 root  wheel        0 Feb 20  2008 My Data Sources
drwxr-xr-x  0 root  wheel        0 Dec 21 22:37 My Music
drwxr-xr-x  0 root  wheel        0 Dec 30 19:26 My Pictures
drwxr-xr-x  0 root  wheel        0 Dec 27  2005 My Videos
drwxr-xr-x  0 root  wheel        0 Jan 22  2008 My eBooks
drwxr-xr-x  0 root  wheel        0 Dec 29 10:39 Myyyyyy Cccc
drwxr-xr-x  0 root  wheel        0 Jan  2  2008 Network Shortcuts
drwxr-xr-x  0 root  wheel        0 Dec 31 10:18 Office Templates
drwxr-xr-x  0 root  wheel        0 Nov  1 15:29 PDF files
-rwxr-xr-x  0 root  wheel    12480 Nov 16 20:53 POST IT NOTES.txt
drwxr-xr-x  0 root  wheel        0 Dec 31 10:17 PROOFS
drwxr-xr-x  0 root  wheel        0 Dec 31 10:22 PSSS
drwxr-xr-x  0 root  wheel        0 Dec 30 20:08 RESUME
-rwxr-xr-x  0 root  wheel      934 Jun  9  2007 Recipes to Print.doc.lnk
-rwxr-xr-x  0 root  wheel      617 Dec 22  2006 Recipes.lnk
drwxr-xr-x  0 root  wheel        0 Oct  2  2006 Reconstruction
-rwxr-xr-x  0 root  wheel  1158656 Dec  3 14:53 Registry Error Message Oct 
2006.doc
drwxr-xr-x  0 root  wheel        0 Aug 15 09:59 Theatre
-rwxr-xr-x  0 root  wheel      505 Dec 21  2007 Theatre Manuals.lnk
-rwxr-xr-x  0 root  wheel    13824 Apr 17  2008 Thumbs.db
-rwxr-xr-x  0 root  wheel    29696 Dec 18 12:00 To Do for Trips.doc
drwxr-xr-x  0 root  wheel        0 Jul 14  2008 To Print
-rwxr-xr-x  0 root  wheel    29184 Feb 15  2008 To do This Week.doc
drwxr-xr-x  0 root  wheel        0 Mar  3  2008 TurboTax
-rwxr-xr-x  0 root  wheel      737 Nov 10 13:03 Uuuuuuu Rate Increase.rtf
drwxr-xr-x  0 root  wheel        0 Dec 18 09:56 Webshots Data
drwxr-xr-x  0 root  wheel        0 Jan  7  2006 Webshots Imported Collections
drwxr-xr-x  0 root  wheel        0 Oct 10 11:46 Wwwww W
-rwxr-xr-x  0 root  wheel       76 Nov 13  2006 desktop.ini
drwxr-xr-x  0 root  wheel        0 Nov  1 15:32 download
-rwxr-xr-x  0 root  wheel      732 Jan  2  2008 email settings in 
progress.doc.lnk
-rwxr-xr-x  0 root  wheel      496 Nov 17 14:12 ooooo funeral.rtf
drwxr-xr-x  0 root  wheel        0 Dec 20 11:07 webdesgn
$ ls -l /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents/08calendar.pdf
-rwxr-xr-x  2 root  wheel  61239 Aug 12 14:43 /ccccc/Documents and 
Settings/Ccccc/My Documents/08calendar.pdf
$ ls -ld /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents/Archive\ Files
drwxr-xr-x  2 root  wheel  0 Aug 18 18:34 /ccccc/Documents and 
Settings/Ccccc/My Documents/Archive Files
$ ls -l /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents/Archive\ Files
total 256
drwxr-xr-x  0 root  wheel      0 Jun 18  2007 Jr's Documents
-rwxr-xr-x  0 root  wheel  64512 Aug 18 18:33 Phone List WWWW-WWW 2008-2009.xls
-rwxr-xr-x  0 root  wheel  65536 Jan 24  2008 Phone List WWWW-WWW Jan 0.xls
drwxr-xr-x  0 root  wheel      0 Sep 19  2007 PictureGear
drwxr-xr-x  0 root  wheel      0 Jul  6  2007 SSS Wedding
$ ls -l /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents/Christmas\ List\ 
2008.doc
-rwxr-xr-x  0 root  wheel  30208 Dec 12 10:15 /ccccc/Documents and 
Settings/Ccccc/My Documents/Christmas List 2008.doc
$ ls -l /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents/cccccccresume.pdf
-rwxr-xr-x  0 root  wheel  209308 Dec 19 10:20 /ccccc/Documents and 
Settings/Ccccc/My Documents/cccccccresume.pdf
$ file /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents/08calendar.pdf
/ccccc/Documents and Settings/Ccccc/My Documents/08calendar.pdf: PDF document, 
version 1.4
$ cd /ccccc/Documents\ and\ Settings/Ccccc/My\ Documents/
$ echo *
Desktop Shortcuts 12-31 Direct Loan Interest.lnk Downloaded Games Downloaded 
Program Updates Ebay Essential Files Excel docs Family Tree Research Financial 
Graphics projects files.lnk Install Files.lnk Insurance & Medical 
ItsDeductible2006 Jjjj LIST.DOC Labels & Signs Letters Manuals Map Files Mogul 
User Guide.lnk Movie List.txt Moving Information My Data Sources My Music My 
Pictures My Videos My eBooks Myyyyyy Cccc Network Shortcuts Office Templates 
PDF files POST IT NOTES.txt PROOFS PSSS RESUME Recipes to Print.doc.lnk 
Recipes.lnk Reconstruction Registry Error Message Oct 2006.doc Theatre Theatre 
Manuals.lnk Thumbs.db To Do for Trips.doc To Print To do This Week.doc TurboTax 
Uuuuuuu Rate Increase.rtf Webshots Data Webshots Imported Collections Wwwww W 
desktop.ini download email settings in progress.doc.lnk ooooo funeral.rtf 
webdesgn

There are 7 subdirectories and 6 files not showing up, a total of 13
entries.  (Not all have been shown here, only a representative sample.)

Note that this doesn't seem to affect all directories - I've used NTFS
mounts before and this is the first place I've encountered it.

>Fix:
Not yet known.  Since it affects (at least) ls, shell wildcard
expansion, and shell path completion, it's not simply a problem in ls.