10.0_BETA randomly panics on huge I/O on a ffs slice on ccd device

To: tech-net%netbsd.org@localhost
Subject: 10.0_BETA randomly panics on huge I/O on a ffs slice on ccd device
From: BERTRAND Joël <joel.kachelhoffer-bertrand%gmx.fr@localhost>
Date: Sun, 19 Mar 2023 15:01:01 +0100

	Hello,

	My main server runs 10.0_BETA (that fix a lot of iscsi initiator bugs).

[ 456386,139917] panic: ffs_blkfree: bad size: dev = 0xa805, bno =
1128617076 bsize = 16384, size = 16384, fs = /var/squid/cache
[ 456386,139917] cpu1: Begin traceback...
[ 456386,139917] vpanic() at netbsd:vpanic+0x183
[ 456386,139917] panic() at netbsd:panic+0x3c
[ 456386,139917] ffs_check_bad_allocation() at
netbsd:ffs_check_bad_allocation+0xff
[ 456386,149915] ffs_blkfree() at netbsd:ffs_blkfree+0x82
[ 456386,149915] ffs_indirtrunc() at netbsd:ffs_indirtrunc+0x420
[ 456386,149915] ffs_truncate() at netbsd:ffs_truncate+0xedd
[ 456386,149915] ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x58
[ 456386,149915] ufs_inactive() at netbsd:ufs_inactive+0xef
[ 456386,149915] VOP_INACTIVE() at netbsd:VOP_INACTIVE+0x3c
[ 456386,149915] vrelel() at netbsd:vrelel+0x154
[ 456386,149915] ufs_remove() at netbsd:ufs_remove+0x50
[ 456386,149915] VOP_REMOVE() at netbsd:VOP_REMOVE+0x81
[ 456386,159915] do_sys_unlinkat.isra.0() at
netbsd:do_sys_unlinkat.isra.0+0x219
[ 456386,159915] syscall() at netbsd:syscall+0x1fc
[ 456386,159915] --- syscall (number 10) ---
[ 456386,159915] netbsd:syscall+0x1fc:
[ 456386,159915] cpu1: End traceback...

	This server runs squid on a /var/squi/cache, that is a wedge on a ccd
device :

legendre# ccdconfig -g
ccd0            32      0x0     2000408739840   /dev/wd0a /dev/wd1a

	Both wd0 and wd1 run fine. On huge I/O, system randomly panics.
Filesystem is mounted with log option and, when I manually launch fsck
on this slice, fsck finds a lot of errors :

legendre# fsck -y /var/squid/cache
** /dev/rdk5
** File system is clean; not checking
legendre# fsck -yf /var/squid/cache
** /dev/rdk5
** File system is already clean
** Last Mounted on /var/squid/cache
** Phase 1 - Check Blocks and Sizes
PARTIALLY TRUNCATED INODE I=23352
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=23448
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=23457
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=256334
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=256459
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=256612
SALVAGE? yes

2459703844 BAD I=256612
1842251117 BAD I=256612
3528233839 BAD I=256612
1004338244 BAD I=256612
3479643544 BAD I=256612
3253368639 BAD I=256612
4162435013 BAD I=256612
1878650390 BAD I=256612
4113582198 BAD I=256612
898064841 BAD I=256612
PARTIALLY TRUNCATED INODE I=279753
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=279756
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=279759
SALVAGE? yes

2459703844 BAD I=279759
PARTIALLY TRUNCATED INODE I=279760
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=279887
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=302965
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=302971
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=373157
SALVAGE? yes

2459703844 BAD I=373157
PARTIALLY TRUNCATED INODE I=373159
SALVAGE? yes

2459703844 BAD I=373159
1842251117 BAD I=373159
3528233839 BAD I=373159
1004338244 BAD I=373159
3479643544 BAD I=373159
3253368639 BAD I=373159
4162435013 BAD I=373159
1878650390 BAD I=373159
4113582198 BAD I=373159
898064841 BAD I=373159
3995785891 BAD I=373159
EXCESSIVE BAD BLKS I=373159
CONTINUE? yes

INCORRECT BLOCK COUNT I=373159 (928 should be 736)
CORRECT? yes

PARTIALLY TRUNCATED INODE I=373160
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=373162
SALVAGE? yes

PARTIALLY TRUNCATED INODE I=373163
SALVAGE? yes

2459703844 BAD I=373163
1842251117 BAD I=373163
PARTIALLY TRUNCATED INODE I=373165
SALVAGE? yes

2459703844 BAD I=373165
1842251117 BAD I=373165
3528233839 BAD I=373165
1004338244 BAD I=373165
3479643544 BAD I=373165
3253368639 BAD I=373165
4162435013 BAD I=373165
...
DUP/BAD  I=373159  OWNER=1012 MODE=100640
SIZE=451390 MTIME=Dec  4 13:07 2022
FILE=/00/3A/00003AA5

REMOVE? yes

DUP/BAD  I=373163  OWNER=1012 MODE=100640
SIZE=215288 MTIME=Dec  4 13:07 2022
FILE=/00/3A/00003AA9

REMOVE? yes

DUP/BAD  I=373165  OWNER=1012 MODE=100640
SIZE=304488 MTIME=Dec  4 13:07 2022
FILE=/00/3A/00003AAB
...
THE FOLLOWING SECTORS COULD NOT BE WRITTEN: 12513426624, 12513426625,
12513426626, 12513426627, 12513426628, 12513426629, 12513426630,
12513426631, 12513426632, 12513426633, 12513426634, 12513426635,
12513426636, 12513426637, 12513426638, 12513426639, 12513426640,
12513426641, 12513426642, 12513426643, 12513426644, 12513426645,
12513426646, 12513426647, 12513426648, 12513426649, 12513426650,
12513426651, 12513426652, 12513426653, 12513426654, 12513426655,

CANNOT READ: BLK 11777852800
CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 11777852800, 11777852801,
11777852802, 11777852803, 11777852804, 11777852805, 11777852806,
11777852807, 11777852808, 11777852809, 11777852810, 11777852811,
11777852812, 11777852813, 11777852814, 11777852815, 11777852816,
11777852817, 11777852818, 11777852819, 11777852820, 11777852821,
11777852822, 11777852823, 11777852824, 11777852825, 11777852826,
11777852827, 11777852828, 11777852829, 11777852830, 11777852831,
CG 31239: ALLOCBLK: BAD MAGIC NUMBER
WRITING ZERO'ED BLOCK 11777852800 TO DISK

CANNOT WRITE: BLK 11777852800
CONTINUE? yes
...
THE FOLLOWING DISK SECTORS COULD NOT BE READ: 1786339776, 1786339777,
1786339778, 1786339779, 1786339780, 1786339781, 1786339782, 1786339783,
1786339784, 1786339785, 1786339786, 1786339787, 1786339788, 1786339789,
1786339790, 1786339791, 1786339792, 1786339793, 1786339794, 1786339795,
1786339796, 1786339797, 1786339798, 1786339799, 1786339800, 1786339801,
1786339802, 1786339803, 1786339804, 1786339805, 1786339806, 1786339807,
CG 4738: ALLOCBLK: BAD MAGIC NUMBER
** Phase 5 - Check Cyl groups
WRITING ZERO'ED BLOCK 1786339776 TO DISK

CANNOT WRITE: BLK 1786339776
CONTINUE? yes

THE FOLLOWING SECTORS COULD NOT BE WRITTEN: 1786339776, 1786339777,
1786339778, 1786339779, 1786339780, 1786339781, 1786339782, 1786339783,
1786339784, 1786339785, 1786339786, 1786339787, 1786339788, 1786339789,
1786339790, 1786339791, 1786339792, 1786339793, 1786339794, 1786339795,
1786339796, 1786339797, 1786339798, 1786339799, 1786339800, 1786339801,
1786339802, 1786339803, 1786339804, 1786339805, 1786339806, 1786339807,
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes

SUMMARY INFORMATION BAD
SALVAGE? yes

BLK(S) MISSING IN BIT MAPS
SALVAGE? yes

29239 files, 1930742 used, 6326849 free (47249 frags, 784950 blocks,
0.6% fragmentation)

***** FILE SYSTEM WAS MODIFIED *****

	Sometimes I find a lot of files in lost+found. Sometimes not.

	I see this panic only on ccd device. Of course, wd0 and wd1 don't
present hardware issues. Sames disks are exported as iSCSI target device
for swap.

	Best regards,

	JKB

Follow-Ups:
- Re: 10.0_BETA randomly panics on huge I/O on a ffs slice on ccd device
  - From: Martin Husemann
- Re: 10.0_BETA randomly panics on huge I/O on a ffs slice on ccd device
  - From: Michael van Elst

Prev by Date: Re: VPN recommendations
Next by Date: Re: 10.0_BETA randomly panics on huge I/O on a ffs slice on ccd device
Previous by Thread: NFS root doen't work on next68k after NetBSD 9.2
Next by Thread: Re: 10.0_BETA randomly panics on huge I/O on a ffs slice on ccd device
Indexes:

Home | Main Index | Thread Index | Old Index