Subject: Re: CPU 100% on lock on file write
To: Manuel Bouyer <email@example.com>
From: Andrew Doran <firstname.lastname@example.org>
Date: 02/21/2007 11:16:16
On Tue, Feb 20, 2007 at 09:59:52PM +0100, Manuel Bouyer wrote:
> I've tried to run 2 writes to a FFS1 filesystem mounted without softdeps:
> dd if=/dev/zero of=file1 bs=1024 count=500000000&
> dd if=/dev/urandom of=file2 bs=1024 count=500000000&
> Surprisingly, the writes are quite slow (with the drive busy at 1 to 2%)
> and top show:
> 37 processes: 34 sleeping, 3 on processor
> CPU0 states: 0.0% user, 0.0% nice, 0.5% system, 0.5% interrupt, 99.0% idle
> CPU1 states: 0.5% user, 0.0% nice, 99.5% system, 0.0% interrupt, 0.0% idle
> CPU2 states: 0.0% user, 0.0% nice, 0.5% system, 0.0% interrupt, 99.5% idle
> CPU3 states: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle
> Memory: 2541M Act, 1242M Inact, 580K Wired, 4764K Exec, 3771M File, 1088K Free
> Swap: 128M Total, 128M Free
> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
> 113 root 64 0 48K 576K CPU/3 12:31 100.44% 100.44% dd
> 112 root 64 0 48K 576K CPU/1 12:47 100.39% 100.39% dd
> I think the 2 dd commands are busy on spin locks in the kernel.
At a guess, a lot of that time is being soaked up by rndpool_extract_data().
That will put back pressure on the big lock and keep out the second dd.
It would not be hard to put a spinlock into the rnd driver, and have it drop
the kernel_lock when we want to extract data from the pool. The only issue
is that in order to avoid deadlocking against the kernel lock, the spinlock
would need to be taken at splaudio() on x86, splsched() to be portable.
Alternatively, some of the more expensive steps like the SHA transforms
could be modified to run without holding any locks. I haven't looked too