NetBSD-Bugs archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: kern/53124 (FFS is slow because pmap_update doesn't scale)

Le 24/03/2018 à 23:07, Michael van Elst a écrit :
On Sat, Mar 24, 2018 at 07:10:03PM +0000, wrote:

My guess is that the 'pmap_update' you're talking about actually
touches a user pmap, and not pmap_kernel.

ubc_alloc touches pmap_kernel when deleting old mappings in the UBC
window. It is this call that is slow:

	 * Mapping must be removed before the list entry,
	 * since there is a race with ubc_purge().
	if (umap->flags & UMAP_MAPPING_CACHED) {
			umap->flags &= ~UMAP_MAPPING_CACHED;
			pmap_remove(pmap_kernel(), va,
				va + ubc_winsize);

Alright, so my initial guess was wrong.

But this guess would have to be verified. You should probably try to
assign your program to a given core - and this, early, _before_ your
program starts doing heavy stuff. schedctl, or pset would be even
better. If I'm right, it should "fix" the slowdown.

Why would the synchronization with other CPUs go away?

Because if my initial guess was correct, by assigning your program to a cpu
you would have guaranteed that your pmap was not loaded on other cpus; and
as a result you wouldn't have had to send IPIs to them when changing a page
in your pmap.

But I may have another guess now. Here the call path is: pmap_remove ->
pmap_remove_ptes -> pmap_remove_pte -> pmap_tlb_shootdown. You can see that
the last one gets called only when ((opte & PG_U) != 0). That is, only when
the page has already been used (read/write) by a core on the machine.

It would be nice if you could verify, somehow, two things:

 o Does the 'UMAP_MAPPING_CACHED' branch get taken the same way with and
   without your N-1 user threads. If there is a clear difference here, then
   it means the problem is that UBC does not scale. Otherwise:

 o When the 'UMAP_MAPPING_CACHED' branch gets taken, does the 'PG_U' one get
   taken too afterwards with and without your N-1 user threads. If there is a
   clear difference here, then it means that for some reason an idle kernel
   thread on a remote cpu decided to touch the page before you did yourself,
   which resulted in PG_U being set, and the IPIs being sent to everybody.


Home | Main Index | Thread Index | Old Index