kern/56008: Zpool lost when disk is moved to another bus

[greg-org-netbsd%akua.com@localhost]

>Number:         56008
>Category:       kern
>Synopsis:       Zpool lost when disk is moved to another bus
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Feb 23 02:55:00 +0000 2021
>Originator:     Greg Kerr
>Release:        9.1_STABLE
>Organization:
Akua
>Environment:
NetBSD lion.mlb.akua.com 9.1_STABLE NetBSD 9.1_STABLE (Akua Lion lion-90-a $Revision: 1.531.2.10 $) #12: Mon Jan 11 16:27:59 EST 2021  akua%lion.mlb.akua.com@localhost:/usr/src/OUT/netbsd-9/kernel/lion-90-a amd64

>Description:
Created two zpools

One M.2/SATA 1TB SSD
One SATA 2TB SSD

Wanted to replace them
1. Shutdown 
2. Moved to USB enclosures. USB-SATA, USB-M.2-SATA
3. Inserted new M.2 NVMe
4. Booted (from mSATA)
5. One Zpool was there, the other not.


 $ sudo zpool status
  pool: M2
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        M2          ONLINE       0     0     0
          dk9       ONLINE       0     0     0

errors: No known data errors

  pool: QVO
 state: UNAVAIL
status: One or more devices could not be opened.  There are insufficient
        replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-3C
  scan: none requested
config:

        NAME                   STATE     READ WRITE CKSUM
        QVO                    UNAVAIL      0     0     0
          7099944016363518513  UNAVAIL      0     0     0  was /dev/wd1

 $ sudo zpool online QVO 7099944016363518513 sd0
cannot open 'QVO': pool is unavailable

 $ sudo zpool online QVO 7099944016363518513 rsd0
cannot open 'QVO': pool is unavailable

Difference was, M2 is GPT partitioned (created wedges), QVO was whole device. hexdump -C /dev/sd0 shows it is attached and reading fine.

 $ dmesg | grep sd0
[     3.150596] sd0 at scsibus0 target 0 lun 0: <Seagate, BACKUP+, 0304> disk fixed
[     3.190596] sd0: 1863 GB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 3907029167 sectors

 $ dmesg | grep sd1
[     3.240596] sd1 at scsibus1 target 0 lun 0: <X80 XDIS, , 2210> disk fixed
[     3.280596] sd1: fabricating a geometry
[     3.310596] sd1: 931 GB, 953869 cyl, 64 head, 32 sec, 512 bytes/sect x 1953525168 sectors
[     3.350596] sd1: fabricating a geometry
[     3.380596] sd1: GPT GUID: 2da609c5-3876-4e21-98c6-caa3ee7cf210
[     3.390596] dk6 at sd1: "ESP_M2", 614361 blocks at 40, type: msdos
[     3.410596] dk7 at sd1: "NetBSD", 134217728 blocks at 614440, type: ffs
[     3.440596] dk8 at sd1: "swap", 67108864 blocks at 134832200, type: swap
[     3.470596] dk9 at sd1: "NetBSD ZFS", 1751584055 blocks at 201941080, type: <unknown>
[     7.350596] boot device: sd1

>How-To-Repeat:
1. Create whole device zpool
2. Power down
3. Move to another bus
4. Power up
5. zpool status

>Fix:
Move back to original location