The enclosed diff is the final version of the code I'm going to produce 
-- for my limited purposes (upgrading to a larger disk), it's 
sufficient, and it's a good roadmap for anyone else who runs into 
similar problems.  It would work in-tree -- the retry limit is now a 
static variable that can be patched with ddb, and if it's set to zero 
the behavior will be identical (modulo one extra debugging message) to 
today's (incorrect) behavior.

I did, however, encounter another problem that I worked around but 
haven't fixed; this problem requires more thought, which is why I've 
redirected the thread to tech-kern: on a device like this, on the last 
close(), power is removed from the drive before any final writes can 
finish.  Thus, if I unmounted a file system, the superblock wouldn't be 
marked clean, and fsck would run the next time.  Similarly, I had 
trouble with fdisk -- I'd run it, but the old label would still be 
there.  I finally worked around the problem by running

	sleep 99999 </dev/rwd1a

in the background.

What this boils down to, I think, is that we need better support for 
removeable devices.  They'll take longer to be available, since they're 
often powered down to start, and they have to be stopped gently, to 
allow things to flush.  I'm not sure what the best approach is to this.

Index: wdc.c
===================================================================
RCS file: /cvsroot/src/sys/dev/ic/wdc.c,v
retrieving revision 1.172
diff -r1.172 wdc.c
476a477,478
> int wdc_delay_cnt = 5;
> 
484a487
> 	int bcnt;
493,494c496,501
< 		if (wdc != NULL && (wdc->cap & WDC_CAPABILITY_SELECT))
< 			wdc->select(chp,0);
---
> 		/* This loop is a hack; it should be a poll based on
> 		   timer interrupts.  --smb */
> 		bcnt = 0;
> 		do {
> 			if (wdc != NULL && (wdc->cap & WDC_CAPABILITY_SELECT))
> 				wdc->select(chp,0);
496,503c503,507
< 		bus_space_write_1(chp->cmd_iot, chp->cmd_iohs[wd_sdh], 0,
< 		    WDSD_IBM);
< 		delay(10);	/* 400ns delay */
< 		st0 = bus_space_read_1(chp->cmd_iot,
< 		    chp->cmd_iohs[wd_status], 0);
< 		
< 		if (wdc != NULL && (wdc->cap & WDC_CAPABILITY_SELECT))
< 			wdc->select(chp,1);
---
> 			bus_space_write_1(chp->cmd_iot, chp->cmd_iohs[wd_sdh], 0,
> 			    WDSD_IBM);
> 			delay(10);	/* 400ns delay */
> 			st0 = bus_space_read_1(chp->cmd_iot,
> 			    chp->cmd_iohs[wd_status], 0);
505,509c509,524
< 		bus_space_write_1(chp->cmd_iot, chp->cmd_iohs[wd_sdh], 0,
< 		    WDSD_IBM | 0x10);
< 		delay(10);	/* 400ns delay */
< 		st1 = bus_space_read_1(chp->cmd_iot,
< 		    chp->cmd_iohs[wd_status], 0);
---
> 			if (wdc != NULL && (wdc->cap & WDC_CAPABILITY_SELECT))
> 				wdc->select(chp,1);
> 
> 			bus_space_write_1(chp->cmd_iot, chp->cmd_iohs[wd_sdh], 0,
> 			    WDSD_IBM | 0x10);
> 			delay(10);	/* 400ns delay */
> 			st1 = bus_space_read_1(chp->cmd_iot,
> 			    chp->cmd_iohs[wd_status], 0);
> 
> 			if ((st0 & WDCS_BSY) == 0 && (st1 & WDCS_BSY) == 0)
> 				break;
> 			delay(1000000);
> 			WDCDEBUG_PRINT(("%s:%d: 'busy' set (0x%x,0x%x)\n",
> 			    wdc != NULL ? wdc->sc_dev.dv_xname : "wdcprobe",
> 			    chp->ch_channel, st0, st1), DEBUG_PROBE);
> 		} while (bcnt++ < wdc_delay_cnt);


		--Steve Bellovin, http://www.research.att.com/~smb