misc/56368: Using if=virtio,cache=none causes device to fail to read until reboot

[dev%iotib.net@localhost]

>Number:         56368
>Category:       misc
>Synopsis:       Using if=virtio,cache=none causes device to fail to read until reboot
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    misc-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Aug 17 21:05:00 +0000 2021
>Originator:     Charadon
>Release:        9.2
>Organization:
none
>Environment:
NetBSD localhost 9.2 NetBSD 9.2 (GENERIC) #0: Wed May 12 13:15:55 UTC 2021  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
Passing a raw device into qemu using the following:
-drive file=/dev/rwd1,if=none,id=hd0,format=raw

However, if this works the same way as it does on linux, this leads to very bad disk performance. So if I use this:

-drive file=/dev/rwd1,if=virtio,id=hd0,format=raw,cache=none
======================
This leads to an error where the raw device will no longer be readable until reboot is done. Reading times out with this error:

wd1d: device timeout reading fsbn 0 of 0-7 (wd1 bn 0; cn 0 tn 0 sn 0), xfer 38, retry 0
wd1: soft error (corrected) xfer 38
wd1d: device timeout reading fsbn 0 (wd1 bn 0; cn 0 tn 0 sn 0), xfer 38, retry 0
wd1d: device timeout reading fsbn 0 (wd1 bn 0; cn 0 tn 0 sn 0), xfer 38, retry 1
wd1d: device timeout reading fsbn 0 (wd1 bn 0; cn 0 tn 0 sn 0), xfer 38, retry 2
wd1d: device timeout reading fsbn 0 (wd1 bn 0; cn 0 tn 0 sn 0), xfer 38, retry 3
wd1d: device timeout reading fsbn 0 (wd1 bn 0; cn 0 tn 0 sn 0), xfer 38, retry 4
wd1d: device timeout reading fsbn 0 (wd1 bn 0; cn 0 tn 0 sn 0)
======================
And until you reboot, even if you revert the qemu launch options to the ones that work, it will not read.

Here's my full QEMU command:
qemu-system-x86_64 -accel nvmm \
-cpu max -smp cpus=4 -m 6G \
-object rng-random,filename=/dev/urandom,id=viornd0 \
-device virtio-rng-pci,rng=viornd0 \
-drive file=/dev/rwd1,if=virtio,id=hd0,format=raw,cache=none \
-device virtio-blk-pci,drive=hd0 \
-display curses \
#-vga vmware \
-usb -device usb-mouse,bus=usb-bus.0 \
-netdev tap,id=tap0,ifname=tap0,script=no -device virtio-net-pci,netdev=tap0

What i've tried and didn't work:
- Removing: -device virtio-blk-pci,drive=hd0
- Trying if=virtio from boot up (before trying the options that work)
- Use /dev/wd1 instead of /dev/rwd1 (qemu forces /dev/rwd1)

Sidenote: I didn't know what kind of issue this is, so I put it under misc.
>How-To-Repeat:
- Have a 2nd storage device
- Install QEMU
- Load NVMM Module
- Run QEMU with launch option: -drive file=/dev/rwd1,if=virtio,id=hd0,format=raw,cache=none
- Device will fail to run.
>Fix: