On Wed, Feb 23, 2005 at 09:26:44AM -0800, Jonathan Stone wrote:
> >> I thought about this more last night. What about an app which writes
> >> into each loaned-out page? Do we take a copy-on-write fault for each
> >> such page?
> >
> >yes.
> >
> >> What do the numbers look like then --- that is, is the
> >> penalty bad enough that it's worth adding a per-fd flag to disable
> >> loanout, for apps which intend to dirty their just-read() pages?
> >
> >I haven't done the experiment, I'll do it soon.
> 
> Thanks. I look forward to seeing the results.


I tried the 8 combinations of:

	(1) large vs. small read size
	(2) touching the pages after reading vs. not
	(3) loaning vs. copying

in all cases, the file was fully cached throughout the test.
the results were:

copying

19 p3:~ # time pt -T -r -s 4096 -c 655360 /usr/file
2684354560 bytes transferred in 8.064 secs (332863768 bytes/sec)
0.119u 7.933s 0:08.06 99.7%     0+0k 0+0io 0pf+0w
20 p3:~ # time pt -r -s 4096 -c 655360 /usr/file
2684354560 bytes transferred in 8.069 secs (332641977 bytes/sec)
0.149u 7.908s 0:08.07 99.6%     0+0k 0+0io 0pf+0w

21 p3:~ # time pt -T -r -s 1048576 -c 2560 /usr/file
2684354560 bytes transferred in 13.339 secs (201227306 bytes/sec)
0.079u 13.239s 0:13.34 99.7%    0+0k 0+0io 0pf+0w
22 p3:~ # time pt -r -s 1048576 -c 2560 /usr/file
2684354560 bytes transferred in 13.302 secs (201790933 bytes/sec)
0.029u 13.251s 0:13.30 99.7%    0+0k 0+0io 0pf+0w


loaning

23 p3:~ # time pt -T -r -s 4096 -c 655360 /usr/file
2684354560 bytes transferred in 10.520 secs (255151964 bytes/sec)
0.549u 9.956s 0:10.52 99.7%     0+0k 0+0io 0pf+0w
24 p3:~ # time pt -r -s 4096 -c 655360 /usr/file
2684354560 bytes transferred in 2.954 secs (908543841 bytes/sec)
0.139u 2.810s 0:02.95 99.6%     0+0k 0+0io 0pf+0w

25 p3:~ # time pt -T -r -s 1048576 -c 2560 /usr/file
2684354560 bytes transferred in 13.632 secs (196914611 bytes/sec)
0.259u 13.348s 0:13.63 99.7%    0+0k 0+0io 0pf+0w
26 p3:~ # time pt -r -s 1048576 -c 2560 /usr/file
2684354560 bytes transferred in 0.974 secs (2754571337 bytes/sec)
0.009u 0.967s 0:00.97 98.9%     0+0k 0+0io 0pf+0w


the test program is at ftp://ftp.netbsd.org/pub/NetBSD/misc/chs/pt.c

so for the modify-after-read case, when a small request size is used,
loaning does turn out to be slower than copying.

however, due to the mysql profiling results that I posted on tech-perform
the other day, I've decided not to pursue loaning for read() or write()
further at this point.  I'll get back to it after 3.0 is branched.
(instead I'm looking at direct I/O, which I'll post more about shortly.)

-Chuck