kern/55714: re(4) and iSCSI (initiator) jumbo frames

[joel.bertrand%systella.fr@localhost]

>Number:         55714
>Category:       kern
>Synopsis:       re(4) and iSCSI (initiator) jumbo frames
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Oct 10 21:35:00 +0000 2020
>Originator:     joel.bertrand%systella.fr@localhost
>Release:        NetBSD 9.0_STABLE
>Organization:
KACHELHOFFER-BERTRAND Joël
>Environment:
	
	
System: NetBSD legendre.systella.fr 9.0_STABLE NetBSD 9.0_STABLE (CUSTOM) #14: Fri Oct 9 12:18:18 CEST 2020 root%legendre.systella.fr@localhost:/usr/src/netbsd-9/obj/sys/arch/amd64/compile/CUSTOM amd64
Architecture: x86_64
Machine: amd64
>Description:

	Hello,

	I have added a PCI (not PCIe) Ethernet adapter on a NetBSD server to
directly connect a NAS (QNAP). Chipset of this adapter is a Realtek 8169S.

	pcictl indicates :
004:00:0: Realtek Semiconductor 8169/8110 10/100/1000 Ethernet (ethernet
network, revision 0x10)

	As NAS is directly connected to this server, I have configred on both sides
of the link jumbo frames (MTU 7418) and set iSCSI segment legnth to 7200.
've got a lot of error in console :

[  8900.078912] sd0d: error writing fsbn 11514444945 of
11514444945-11514444961 (sd0 bn 11514444945; cn 5622287 tn 36 sn 17)
[  8900.078912] sd0d: error writing fsbn 11514444962 of
11514444962-11514445008 (sd0 bn 11514444962; cn 5622287 tn 37 sn 2)
[  8900.078912] sd0d: error writing fsbn 11514445009 of
11514445009-11514445072 (sd0 bn 11514445009; cn 5622287 tn 38 sn 17)
[  8900.078912] sd0d: error writing fsbn 11514445073 of
11514445073-11514445089 (sd0 bn 11514445073; cn 5622287 tn 40 sn 17)
[  8900.078912] sd0d: error writing fsbn 11514445090 of
11514445090-11514445104 (sd0 bn 11514445090; cn 5622287 tn 41 sn 2)
[  8900.078912] sd0d: error writing fsbn 11514445105 of
11514445105-11514445168 (sd0 bn 11514445105; cn 5622287 tn 41 sn 17)
[  8900.078912] sd0d: error writing fsbn 11514445169 (sd0 bn
11514445169; cn 5622287 tn 43 sn 17)

until kernel panic :
[  8909.342671] uvm_fault(0xffffffff8151f760, 0xffffba0020ad0000, 1) -> e
[  8909.342671] fatal page fault in supervisor mode
[  8909.342671] trap type 6 code 0 rip 0xffffffff8026aee8 cs 0x8 rflags
0x10286 cr2 0xffffba0020ad0070 ilevel 0 rsp 0xffffba013cc18b90
[  8909.342671] curlwp 0xffff9974adb6e900 pid 0.175 lowest kstack
0xffffba013cc162c0
[  8909.342671] panic: trap
[  8909.342671] cpu5: Begin traceback...
[  8909.342671] vpanic() at netbsd:vpanic+0x160
[  8909.342671] snprintf() at netbsd:snprintf
[  8909.342671] startlwp() at netbsd:startlwp
[  8909.342671] alltraps() at netbsd:alltraps+0xbb
[  8909.342671] dk_start() at netbsd:dk_start+0x102
[  8909.342671] spec_strategy() at netbsd:spec_strategy+0xa7
[  8909.342671] VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x4c
[  8909.352675] dkstart() at netbsd:dkstart+0x184
[  8909.352675] spec_strategy() at netbsd:spec_strategy+0xa7
[  8909.352675] VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x4c
[  8909.352675] wapbl_buffered_write_async() at
netbsd:wapbl_buffered_write_async+0x7d
[  8909.352675] wapbl_buffered_write() at netbsd:wapbl_buffered_write+0xdf
[  8909.352675] wapbl_circ_write() at netbsd:wapbl_circ_write+0x103
[  8909.352675] wapbl_flush() at netbsd:wapbl_flush+0x26f
[  8909.352675] ffs_sync() at netbsd:ffs_sync+0x20a
[  8909.362679] VFS_SYNC() at netbsd:VFS_SYNC+0x35
[  8909.362679] sched_sync() at netbsd:sched_sync+0x98
[  8909.362679] cpu5: End traceback...

	In a first time, I have reconnected NAS with default segment length (less
than 1500 bytes, I have verified). Kernel panics also, but with some more
readable messages :

[   481.957605] re0: watchdog timeout
[   494.632228] S2C1: *** Connection Error, status=24, logout=-1, state=3
[   503.975605] re0: watchdog timeout
[   519.481139] S2C1: *** Connection Error, status=18, logout=2, state=6
[   525.663327] S2C1: ccb_timeout: num=1 total=1 disp=5
[   525.663327] S2C1: *** Connection Error, status=24, logout=2, state=6
[   525.683334] S2C1: ccb_timeout: num=1 total=1 disp=5
[   525.683334] S2C1: *** Connection Error, status=24, logout=2, state=6
[   525.683334] S2C1: ccb_timeout: num=1 total=1 disp=5
[   525.683334] S2C1: *** Connection Error, status=24, logout=2, state=6
[   526.683688] S2C1: ccb_timeout: num=1 total=1 disp=5
[   526.683688] S2C1: *** Connection Error, status=24, logout=2, state=6
[   526.683688] S2C1: ccb_timeout: num=1 total=1 disp=5
[   526.683688] S2C1: *** Connection Error, status=24, logout=2, state=6
[   526.683688] S2C1: ccb_timeout: num=1 total=1 disp=5
[   526.683688] S2C1: *** Connection Error, status=24, logout=2, state=6
[   528.644381] S2C1: ccb_timeout: num=1 total=1 disp=3
[   528.644381] S2C1: *** Connection Error, status=24, logout=2, state=6
[   530.645090] S2C1: ccb_timeout: num=1 total=1 disp=3
[   530.645090] S2C1: *** Connection Error, status=24, logout=2, state=6
[   531.995569] re0: watchdog timeout
[   532.495745] S2C1: Write failed sock 0xffffa6bf802ca940 (ret: 32,
req: 65584, resid: 16234)
[   532.495745] S2C1: *** Connection Error, status=18, logout=-1, state=6
[   534.506456] S2C1: Resend ccb 0xffffd88020ad1d00 (37) - updating
CmdSN old 92, new 95
[   534.506456] S2C1: Resend ccb 0xffffd88020ad1b88 (37) - updating
CmdSN old 93, new 96
[   534.506456] S2C1: Resend ccb 0xffffd88020ad1a10 (37) - updating
CmdSN old 94, new 97
[   534.506456] S2C1: Connection ReCreated successfully - status 0
[   542.009111] re0: watchdog timeout
[   553.023009] re0: watchdog timeout
[   554.533544] S2C1: *** Connection Error, status=24, logout=-1, state=3
[   561.055852] S2C1: *** Connection Error, status=18, logout=2, state=6
[   563.116581] S2C1: Resend ccb 0xffffd88020ad0870 (37) - updating
CmdSN old 104, new 114
[   563.116581] S2C1: Resend ccb 0xffffd88020ad06f8 (37) - updating
CmdSN old 105, new 115
[   563.116581] S2C1: Resend ccb 0xffffd88020ad0580 (37) - updating
CmdSN old 106, new 116
[   563.116581] S2C1: Resend ccb 0xffffd88020ad0408 (37) - updating
CmdSN old 107, new 117
[   563.116581] S2C1: Resend ccb 0xffffd88020ad0290 (37) - updating
CmdSN old 108, new 118
[   563.116581] S2C1: Resend ccb 0xffffd88020ad3018 (37) - updating
CmdSN old 109, new 119
[   563.116581] S2C1: Resend ccb 0xffffd88020ad2ea0 (37) - updating
CmdSN old 110, new 120
[   563.116581] S2C1: Resend ccb 0xffffd88020ad2d28 (37) - updating
CmdSN old 111, new 121
[   563.116581] S2C1: Resend ccb 0xffffd88020ad2bb0 (37) - updating
CmdSN old 112, new 122
[   563.116581] S2C1: Resend ccb 0xffffd88020ad25d0 (37) - updating
CmdSN old 113, new 123
[   563.116581] S2C1: Connection ReCreated successfully - status 0
[   574.110473] S2C1: *** Connection Error, status=24, logout=-1, state=3
[   580.042571] re0: watchdog timeout
[   590.056116] S2C1: *** Connection Error, status=24, logout=-1, state=6
[   597.188640] S2C1: Resend ccb 0xffffd88020ad0580 (37) - updating
CmdSN old 116, new 126
[   597.188640] S2C1: Resend ccb 0xffffd88020ad0408 (37) - updating
CmdSN old 117, new 127
[   597.188640] S2C1: Resend ccb 0xffffd88020ad0290 (37) - updating
CmdSN old 118, new 128
[   597.188640] S2C1: Resend ccb 0xffffd88020ad3018 (37) - updating
CmdSN old 119, new 129
[   597.188640] S2C1: Resend ccb 0xffffd88020ad2ea0 (37) - updating
CmdSN old 120, new 130
[   597.188640] S2C1: Resend ccb 0xffffd88020ad2d28 (37) - updating
CmdSN old 121, new 131
[   597.188640] S2C1: Resend ccb 0xffffd88020ad2bb0 (37) - updating
CmdSN old 122, new 132
[   597.188640] S2C1: Resend ccb 0xffffd88020ad25d0 (37) - updating
CmdSN old 123, new 133
[   597.198644] S2C1: Resend ccb 0xffffd88020ad15a8 (37) - updating
CmdSN old 124, new 134
[   597.198644] S2C1: Resend ccb 0xffffd88020ad1430 (37) - updating
CmdSN old 125, new 135
[   597.198644] S2C1: Connection ReCreated successfully - status 0
[   617.205725] S2C1: *** Connection Error, status=24, logout=-1, state=3
[   618.066028] re0: watchdog timeout

	In a second time, I have reconfigured Ethernet link to use MTU 1500 on both
sides and iSCSI default segment length. With default values, iSCSI initiator
works as expected on re(4), but runs slowly (80 mbps).

	I suppose there is an issue somewhere in re driver.

	Best regards,

	JKB 

>Fix:

Unknown
>Unformatted: