Subject: Corrupt data when reading filesystems under Linux guest
To: None <port-xen@netbsd.org>
From: None <marcotte@panix.com>
List: port-xen
Date: 06/06/2005 17:08:42
We're having a peculiar problem with Xen. We are finding that a
Linux guest can't properly access more than two partitions.

What's happening is that when reading from any partiion after the
first two, some of the data read is not from the disk, but other
random pages in RAM. For instance, you see data from processes in
RAM like "less" or "hexdump" which I was using to look at the data in
the partition.

We are using:

	Xen 2.0.6 (including xentools)
	Dom0: NetBSD 3 as of June 1
	DomU: The linux distributed with Xen 2.0.6

We've used a couple of different systems for this, but in all cases,
the filesystems sent to the linux guest have been physical partitions
on FibreChannel disks.

If I send the same partition to a NetBSD guest, everything looks fine.

Below is some output from the NetBSD dom0 kernel when accessing one of
these filesystems. This is with XENDEBUG_VBD.

- Brian

----------------------------------

xbdback op 0 req_cons 0xcd40 req_prod 0xcd47/0x7 resp_prod 0xcd40/0x0
xbdback_io domain 2: frame_and_sects[0]=0x27f68000
xbdback_io domain 2: frame_and_sects[1]=0x27f68009
xbdback_io domain 2: frame_and_sects[2]=0x27f68012
xbdback_io domain 2: frame_and_sects[3]=0x27f6801b
xbdback_io domain 2: frame_and_sects[4]=0x27f68024
xbdback_io domain 2: frame_and_sects[5]=0x27f6802d
xbdback_io domain 2: frame_and_sects[6]=0x27f68036
xbdback_io domain 2: frame_and_sects[7]=0x27f6803f
xbdback_io domain 2: frame_and_sects[8]=0x27f67000
xbdback_io domain 2: frame_and_sects[9]=0x27f67009
xbdback_io domain 2: frame_and_sects[10]=0x27f67012
xbdback_io domain 2: start sect 1 start 0 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd41 req_prod 0xcd47/0x7 resp_prod 0xcd40/0x0
xbdback_io domain 2: frame_and_sects[0]=0x27f6701b
xbdback_io domain 2: frame_and_sects[1]=0x27f67024
xbdback_io domain 2: frame_and_sects[2]=0x27f6702d
xbdback_io domain 2: frame_and_sects[3]=0x27f67036
xbdback_io domain 2: frame_and_sects[4]=0x27f6703f
xbdback_io domain 2: frame_and_sects[5]=0x27f66000
xbdback_io domain 2: frame_and_sects[6]=0x27f66009
xbdback_io domain 2: frame_and_sects[7]=0x27f66012
xbdback_io domain 2: frame_and_sects[8]=0x27f6601b
xbdback_io domain 2: frame_and_sects[9]=0x27f66024
xbdback_io domain 2: frame_and_sects[10]=0x27f6602d
xbdback_io domain 2: start sect 12 start 1536 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd42 req_prod 0xcd47/0x7 resp_prod 0xcd40/0x0
xbdback_io domain 2: frame_and_sects[0]=0x27f66036
xbdback_io domain 2: frame_and_sects[1]=0x27f6603f
xbdback_io domain 2: frame_and_sects[2]=0x27f65000
xbdback_io domain 2: frame_and_sects[3]=0x27f65009
xbdback_io domain 2: frame_and_sects[4]=0x27f65012
xbdback_io domain 2: frame_and_sects[5]=0x27f6501b
xbdback_io domain 2: frame_and_sects[6]=0x27f65024
xbdback_io domain 2: frame_and_sects[7]=0x27f6502d
xbdback_io domain 2: frame_and_sects[8]=0x27f65036
xbdback_io domain 2: frame_and_sects[9]=0x27f6503f
xbdback_io domain 2: frame_and_sects[10]=0x27f64000
xbdback_io domain 2: start sect 23 start 3072 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd43 req_prod 0xcd47/0x7 resp_prod 0xcd40/0x0
xbdback_io domain 2: frame_and_sects[0]=0x27f64009
xbdback_io domain 2: start sect 34 start 512 size 512
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd44 req_prod 0xcd47/0x7 resp_prod 0xcd40/0x0
xbdback_io domain 2: frame_and_sects[0]=0x27f64012
xbdback_io domain 2: frame_and_sects[1]=0x27f6401b
xbdback_io domain 2: frame_and_sects[2]=0x27f64024
xbdback_io domain 2: frame_and_sects[3]=0x27f6402d
xbdback_io domain 2: frame_and_sects[4]=0x27f64036
xbdback_io domain 2: frame_and_sects[5]=0x27f6403f
xbdback_io domain 2: frame_and_sects[6]=0x27f63000
xbdback_io domain 2: frame_and_sects[7]=0x27f63009
xbdback_io domain 2: frame_and_sects[8]=0x27f63012
xbdback_io domain 2: frame_and_sects[9]=0x27f6301b
xbdback_io domain 2: frame_and_sects[10]=0x27f63024
xbdback_io domain 2: start sect 35 start 1024 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd45 req_prod 0xcd47/0x7 resp_prod 0xcd40/0x0
xbdback_io domain 2: frame_and_sects[0]=0x27f6302d
xbdback_io domain 2: frame_and_sects[1]=0x27f63036
xbdback_io domain 2: frame_and_sects[2]=0x27f6303f
xbdback_io domain 2: frame_and_sects[3]=0x27f62000
xbdback_io domain 2: frame_and_sects[4]=0x27f62009
xbdback_io domain 2: frame_and_sects[5]=0x27f62012
xbdback_io domain 2: frame_and_sects[6]=0x27f6201b
xbdback_io domain 2: frame_and_sects[7]=0x27f62024
xbdback_io domain 2: frame_and_sects[8]=0x27f6202d
xbdback_io domain 2: frame_and_sects[9]=0x27f62036
xbdback_io domain 2: frame_and_sects[10]=0x27f6203f
xbdback_io domain 2: start sect 46 start 2560 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd46 req_prod 0xcd47/0x7 resp_prod 0xcd40/0x0
xbdback_io domain 2: frame_and_sects[0]=0x27f61000
xbdback_io domain 2: frame_and_sects[1]=0x27f61009
xbdback_io domain 2: frame_and_sects[2]=0x27f61012
xbdback_io domain 2: frame_and_sects[3]=0x27f6101b
xbdback_io domain 2: frame_and_sects[4]=0x27f61024
xbdback_io domain 2: frame_and_sects[5]=0x27f6102d
xbdback_io domain 2: frame_and_sects[6]=0x27f61036
xbdback_io domain 2: frame_and_sects[7]=0x27f6103f
xbdback_io domain 2: start sect 57 start 0 size 4096
xbdback_io domain 2: start request
xbdback_io domain 2: end reqest error=0
xbdback op 0 req_cons 0xcd47 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x27f60000
xbdback_io domain 2: frame_and_sects[1]=0x27f60009
xbdback_io domain 2: frame_and_sects[2]=0x27f60012
xbdback_io domain 2: frame_and_sects[3]=0x27f6001b
xbdback_io domain 2: frame_and_sects[4]=0x27f60024
xbdback_io domain 2: frame_and_sects[5]=0x27f6002d
xbdback_io domain 2: frame_and_sects[6]=0x27f60036
xbdback_io domain 2: frame_and_sects[7]=0x27f6003f
xbdback_io domain 2: frame_and_sects[8]=0x27f5f000
xbdback_io domain 2: frame_and_sects[9]=0x27f5f009
xbdback_io domain 2: frame_and_sects[10]=0x27f5f012
xbdback_io domain 2: start sect 65 start 0 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd48 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x27f5f01b
xbdback_io domain 2: frame_and_sects[1]=0x27f5f024
xbdback_io domain 2: frame_and_sects[2]=0x27f5f02d
xbdback_io domain 2: frame_and_sects[3]=0x27f5f036
xbdback_io domain 2: frame_and_sects[4]=0x27f5f03f
xbdback_io domain 2: frame_and_sects[5]=0x203e6000
xbdback_io domain 2: frame_and_sects[6]=0x203e6009
xbdback_io domain 2: frame_and_sects[7]=0x203e6012
xbdback_io domain 2: frame_and_sects[8]=0x203e601b
xbdback_io domain 2: frame_and_sects[9]=0x203e6024
xbdback_io domain 2: frame_and_sects[10]=0x203e602d
xbdback_io domain 2: start sect 76 start 1536 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd49 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x203e6036
xbdback_io domain 2: frame_and_sects[1]=0x203e603f
xbdback_io domain 2: frame_and_sects[2]=0x203e5000
xbdback_io domain 2: frame_and_sects[3]=0x203e5009
xbdback_io domain 2: frame_and_sects[4]=0x203e5012
xbdback_io domain 2: frame_and_sects[5]=0x203e501b
xbdback_io domain 2: frame_and_sects[6]=0x203e5024
xbdback_io domain 2: frame_and_sects[7]=0x203e502d
xbdback_io domain 2: frame_and_sects[8]=0x203e5036
xbdback_io domain 2: frame_and_sects[9]=0x203e503f
xbdback_io domain 2: frame_and_sects[10]=0x165f8000
xbdback_io domain 2: start sect 87 start 3072 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd4a req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x165f8009
xbdback_io domain 2: start sect 98 start 512 size 512
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd4b req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x165f8012
xbdback_io domain 2: frame_and_sects[1]=0x165f801b
xbdback_io domain 2: frame_and_sects[2]=0x165f8024
xbdback_io domain 2: frame_and_sects[3]=0x165f802d
xbdback_io domain 2: frame_and_sects[4]=0x165f8036
xbdback_io domain 2: frame_and_sects[5]=0x165f803f
xbdback_io domain 2: frame_and_sects[6]=0x165f7000
xbdback_io domain 2: frame_and_sects[7]=0x165f7009
xbdback_io domain 2: frame_and_sects[8]=0x165f7012
xbdback_io domain 2: frame_and_sects[9]=0x165f701b
xbdback_io domain 2: frame_and_sects[10]=0x165f7024
xbdback_io domain 2: start sect 99 start 1024 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd4c req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x165f702d
xbdback_io domain 2: frame_and_sects[1]=0x165f7036
xbdback_io domain 2: frame_and_sects[2]=0x165f703f
xbdback_io domain 2: frame_and_sects[3]=0x16acc000
xbdback_io domain 2: frame_and_sects[4]=0x16acc009
xbdback_io domain 2: frame_and_sects[5]=0x16acc012
xbdback_io domain 2: frame_and_sects[6]=0x16acc01b
xbdback_io domain 2: frame_and_sects[7]=0x16acc024
xbdback_io domain 2: frame_and_sects[8]=0x16acc02d
xbdback_io domain 2: frame_and_sects[9]=0x16acc036
xbdback_io domain 2: frame_and_sects[10]=0x16acc03f
xbdback_io domain 2: start sect 110 start 2560 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd4d req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x16acb000
xbdback_io domain 2: frame_and_sects[1]=0x16acb009
xbdback_io domain 2: frame_and_sects[2]=0x16acb012
xbdback_io domain 2: frame_and_sects[3]=0x16acb01b
xbdback_io domain 2: frame_and_sects[4]=0x16acb024
xbdback_io domain 2: frame_and_sects[5]=0x16acb02d
xbdback_io domain 2: frame_and_sects[6]=0x16acb036
xbdback_io domain 2: frame_and_sects[7]=0x16acb03f
xbdback_io domain 2: frame_and_sects[8]=0x17ca0000
xbdback_io domain 2: frame_and_sects[9]=0x17ca0009
xbdback_io domain 2: frame_and_sects[10]=0x17ca0012
xbdback_io domain 2: start sect 121 start 0 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd4e req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x17ca001b
xbdback_io domain 2: start sect 132 start 1536 size 512
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd4f req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x17ca0024
xbdback_io domain 2: frame_and_sects[1]=0x17ca002d
xbdback_io domain 2: frame_and_sects[2]=0x17ca0036
xbdback_io domain 2: frame_and_sects[3]=0x17ca003f
xbdback_io domain 2: frame_and_sects[4]=0x17c9f000
xbdback_io domain 2: frame_and_sects[5]=0x17c9f009
xbdback_io domain 2: frame_and_sects[6]=0x17c9f012
xbdback_io domain 2: frame_and_sects[7]=0x17c9f01b
xbdback_io domain 2: frame_and_sects[8]=0x17c9f024
xbdback_io domain 2: frame_and_sects[9]=0x17c9f02d
xbdback_io domain 2: frame_and_sects[10]=0x17c9f036
xbdback_io domain 2: start sect 133 start 2048 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd50 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x17c9f03f
xbdback_io domain 2: frame_and_sects[1]=0x18f3c000
xbdback_io domain 2: frame_and_sects[2]=0x18f3c009
xbdback_io domain 2: frame_and_sects[3]=0x18f3c012
xbdback_io domain 2: frame_and_sects[4]=0x18f3c01b
xbdback_io domain 2: frame_and_sects[5]=0x18f3c024
xbdback_io domain 2: frame_and_sects[6]=0x18f3c02d
xbdback_io domain 2: frame_and_sects[7]=0x18f3c036
xbdback_io domain 2: frame_and_sects[8]=0x18f3c03f
xbdback_io domain 2: frame_and_sects[9]=0x18f3b000
xbdback_io domain 2: frame_and_sects[10]=0x18f3b009
xbdback_io domain 2: start sect 144 start 3584 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd51 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x18f3b012
xbdback_io domain 2: frame_and_sects[1]=0x18f3b01b
xbdback_io domain 2: frame_and_sects[2]=0x18f3b024
xbdback_io domain 2: frame_and_sects[3]=0x18f3b02d
xbdback_io domain 2: frame_and_sects[4]=0x18f3b036
xbdback_io domain 2: frame_and_sects[5]=0x18f3b03f
xbdback_io domain 2: frame_and_sects[6]=0x20386000
xbdback_io domain 2: frame_and_sects[7]=0x20386009
xbdback_io domain 2: frame_and_sects[8]=0x20386012
xbdback_io domain 2: frame_and_sects[9]=0x2038601b
xbdback_io domain 2: frame_and_sects[10]=0x20386024
xbdback_io domain 2: start sect 155 start 1024 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd52 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x2038602d
xbdback_io domain 2: start sect 166 start 2560 size 512
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd53 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x20386036
xbdback_io domain 2: frame_and_sects[1]=0x2038603f
xbdback_io domain 2: frame_and_sects[2]=0x20385000
xbdback_io domain 2: frame_and_sects[3]=0x20385009
xbdback_io domain 2: frame_and_sects[4]=0x20385012
xbdback_io domain 2: frame_and_sects[5]=0x2038501b
xbdback_io domain 2: frame_and_sects[6]=0x20385024
xbdback_io domain 2: frame_and_sects[7]=0x2038502d
xbdback_io domain 2: frame_and_sects[8]=0x20385036
xbdback_io domain 2: frame_and_sects[9]=0x2038503f
xbdback_io domain 2: frame_and_sects[10]=0x203a8000
xbdback_io domain 2: start sect 167 start 3072 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd54 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x203a8009
xbdback_io domain 2: frame_and_sects[1]=0x203a8012
xbdback_io domain 2: frame_and_sects[2]=0x203a801b
xbdback_io domain 2: frame_and_sects[3]=0x203a8024
xbdback_io domain 2: frame_and_sects[4]=0x203a802d
xbdback_io domain 2: frame_and_sects[5]=0x203a8036
xbdback_io domain 2: frame_and_sects[6]=0x203a803f
xbdback_io domain 2: frame_and_sects[7]=0x203a7000
xbdback_io domain 2: frame_and_sects[8]=0x203a7009
xbdback_io domain 2: frame_and_sects[9]=0x203a7012
xbdback_io domain 2: frame_and_sects[10]=0x203a701b
xbdback_io domain 2: start sect 178 start 512 size 5632
xbdback_io domain 2: start request
xbdback op 0 req_cons 0xcd55 req_prod 0xcd56/0x16 resp_prod 0xcd41/0x1
xbdback_io domain 2: frame_and_sects[0]=0x203a7024
xbdback_io domain 2: frame_and_sects[1]=0x203a702d
xbdback_io domain 2: frame_and_sects[2]=0x203a7036
xbdback_io domain 2: frame_and_sects[3]=0x203a703f
xbdback_io domain 2: start sect 189 start 2048 size 2048
xbdback_io domain 2: start request
xbdback_io domain 2: end reqest error=0
[ and so on... ]