Re: lockstat from pathological builds Re: high sys time, very very slow builds on new 24-core system

[Andrew Doran <ad%netbsd.org@localhost>]

On Fri, Apr 01, 2011 at 01:05:02PM -0400, Thor Lancelot Simon wrote:

> On Thu, Mar 31, 2011 at 04:32:12PM -0400, Thor Lancelot Simon wrote:
> > On Thu, Mar 24, 2011 at 12:04:02PM +0000, Andrew Doran wrote:
> > > 
> > > Try lockstat as suggested to see if something pathological is going on.  
> > > In
> > > addition to showing lock contention problems it can often highlight a code
> > > path being hit too frequently for some reason.
> > 
> > I have attached build.sh and lockstat output from 24-way and 8-way builds
> > of the amd64 GENERIC kernel on this system.  Also dmesg and mount -v output
> > so you can see what's mounted how, etc.
> 
> Mindaugas spent quite a while looking at this with me.  It appears to
> be largely due to contention on the page queue lock.

I did a few back-of-a-cigarette-packet calculations.

On the 8 CPU system we are burning about 1% of the total CPU time on lock
contention, so not all that significant.

On the 24 CPU system we're burning 27% of the available CPU time.  So
roughly 6.5 cores worth of your machine are doing nothing but spinning on
locks.  So lock contention in the VM system is a problem but it doesn't tell
the full story.

> Some of the contention is caused by the idle loop page zeroing code.
> Mindaugas has dramatically improved its performance on large systems
> but I would still recommend turning it off on systems with more than
> 12 CPUs.
> 
> I would actually like to know for what workload, on what hardware, the
> idle loop zeroer is actually beneficial.  I'm sure there are some
> but clearly they are not workloads I've tried testing recently.

It was a pretty big win on the 8 core system I have.  Clearly it needs
more tuning!

> His improvement shaves about 20 seconds off the build time.  Now we
> scale nearly linearly up to about 12 build jobs and get a very slight
> performance improvement from jobs 12-16.  With vm.idlezero=0 set the
> slight performance improvement is a little less slight.
> 
> I believe he's currently working on hashing the page queue lock to see
> if that gives us some performance back for CPUs 12 through 24.

We'll likely see some contention pushed back to uvm_fpageqlock (the freelist
lock), and will still get murdered on cache line contention from the
active/inactive and free lists.

In the mid term I'm thinking we need to investigate an alternative strategy
such as hashing out the complete paging state, or move away from continual
activation/deactivation of pages, or doing things lazily so we don't need to
have so much massive churn of global state as happens now during build.sh.
The global freelist is another area of concern.