Re: Serious WAPL performance problems

[Roger Pau Monné <roger.pau%citrix.com@localhost>]

On 25/10/12 14:54, Stephan wrote:
> 2012/10/25 Edgar Fuß <ef%math.uni-bonn.de@localhost>:
>> Now this is getting weird. I have retried the experiment with neither softdep
>> nor log, i.e. with a plain old FFS, and that performs as well as or even
>> outperforms WAPL.
>> With both a 5.1_STABLE and a 6.0_RELEASE kernel, on a 16k fsbsize FFSv2, the
>> svn updates takes around 5 seconds with either log or without.
>> On the 8k fsbsize FFSv1, it takes 7 seconds with WAPL and 5 without!
>> So there seems to be something seriously wrong with WAPL if can degrade fs
>> performance, isn't it?
> 
> 
> I always found FFS being slow when creating or deleting many files.
> For example, on 6.0 with FFSv2 and WAPBL it took 20 sec. to complete
> this:
> 
> 
> time seq 1 30000 | xargs touch

I did this inside of a NetBSD DomU, and the time to perform the
operation when using FFSv2 and WAPL is between 27-28s. Also, when the
operation finishes, and I get the time results, it takes some time for
the shell promtp to appear again, between 5s and 10s I would say. I've
added some debugging to xbd_xenbus, to see if we where issuing a lot of
flushes or requests, and I've found out that most of the time the kernel
seems to be sending spaced read requests, and when jumping into ddb the
kernel is almost always in VOP_LOOKUP or ufs_lookup (forget about
anything above and including the hypervisor_callback call):

Stopped in pid 549.1 (touch) at netbsd:breakpoint+0x5:  leave
breakpoint() at netbsd:breakpoint+0x5
xencons_tty_input() at netbsd:xencons_tty_input+0xc9
xencons_handler() at netbsd:xencons_handler+0x79
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
evtchn_do_event() at netbsd:evtchn_do_event+0x175
call_evtchn_do_event() at netbsd:call_evtchn_do_event+0xd
hypervisor_callback() at netbsd:hypervisor_callback+0x9e
ufs_lookup() at netbsd:ufs_lookup+0x41d
VOP_LOOKUP() at netbsd:VOP_LOOKUP+0x39
lookup_once() at netbsd:lookup_once+0xd2
namei_tryemulroot() at netbsd:namei_tryemulroot+0x263
namei() at netbsd:namei+0x2b
do_sys_stat() at netbsd:do_sys_stat+0x64
sys___stat50() at netbsd:sys___stat50+0x29
syscall() at netbsd:syscall+0x94

I've tried getting the line with addr2line on a kernel compiled with
"-g", but there's seems to be no way to get it, I only get ??:00. Just
as a side note, doing the same inside a Linux DomU with ext3 takes less
than 0.1s.

This is a summary of the number of read/write/flush requests before and
after performing the command:

xbdstart: num_reads: 1434 num_writes: 1219 num_flushes: 46
---
xbdstart: num_reads: 1910 num_writes: 2255 num_flushes: 54

This really degrades NetBSD DomU performance, for example unpacking a
sample src.tar.gz set takes 25min, while doing this in a Linux DomU only
takes 1m and 48s (usign exactly the same Dom0).