Re: Possible buffer cache race?

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Sun, Oct 23, 2016 at 06:27:09PM +0200, Jarom??r Dole??ek wrote:
> 
> I see interesting thing - periodically, all of the tar processes get
> blocked sleeping on either tstile, biolock or pager_map. All the tar
> processes block. When I just wait they stay blocked. When I call
> sync(8), all of them unblock and continue running, until again they
> all hit the same condition later.  When I keep calling sync,
> eventually all processes finish.
> 
> Most often they block on biolock, then somewhat less frequently
> tstile; pager_map is more rare. It's usually mix of these - most
> processes block on biolock, some tstile and zero/one/two on pager_map.

I have a hunch this might be related to the metadata cache growing and
shrinking (I assume by "buffer cache" you mean the traditional buffer
cache, now used only for metadata -- not the page cache?).

It could also be related to the need to push out dirty buffers in order
to make room for new metadata.  If this happens because the clock ticks
over, it's supposed to be basically asynchronous.  But if it happens
because you need another metadata buffer _right now_ to schedule a
directory read or a write... it is effectively synchronous.

> Any idea where I should try to start poking?

You could just disable the code that tries to dynamically allocate and
free metadata cache buffers, replacing it with a fixed size allocation
at system startup -- try a very large one and see what happens.  If that
looks better, it's a hint that the problem is related to the allocations
used to dynamically grow the metadata cache.  But if it doesn't look
better, the problem may simply be related to buffer recycling to make
room for new metadata for new operations.  In that case, make the fixed
buffer cache size much smaller so recycling's going on all the time,
figure out how to instrument it, and debug...

-- 
  Thor Lancelot Simon	                                     tls%panix.com@localhost

    "The dirtiest word in art is the C-word.  I can't even say 'craft'
     without feeling dirty."	-Chuck Close