Re: kern/53124 (FFS is slow because pmap_update doesn't scale)

[Maxime Villard <max%m00nbsd.net@localhost>]

The following reply was made to PR kern/53124; it has been noted by GNATS.

From: Maxime Villard <max%m00nbsd.net@localhost>
To: Michael van Elst <mlelstv%serpens.de@localhost>, gnats-bugs%NetBSD.org@localhost
Cc: kern-bug-people%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost, gnats-admin%netbsd.org@localhost
Subject: Re: kern/53124 (FFS is slow because pmap_update doesn't scale)
Date: Sun, 25 Mar 2018 10:00:21 +0200

 Le 24/03/2018 à 23:07, Michael van Elst a écrit :
 > On Sat, Mar 24, 2018 at 07:10:03PM +0000, maxv%NetBSD.org@localhost wrote:
 > 
 >> My guess is that the 'pmap_update' you're talking about actually
 >> touches a user pmap, and not pmap_kernel.
 > 
 > ubc_alloc touches pmap_kernel when deleting old mappings in the UBC
 > window. It is this call that is slow:
 > 
 > 	/*
 > 	 * Mapping must be removed before the list entry,
 > 	 * since there is a race with ubc_purge().
 > 	 */
 > 	if (umap->flags & UMAP_MAPPING_CACHED) {
 > 			umap->flags &= ~UMAP_MAPPING_CACHED;
 > 			mutex_enter(oobj->vmobjlock);
 > 			pmap_remove(pmap_kernel(), va,
 > 				va + ubc_winsize);
 > 			pmap_update(pmap_kernel());
 > 			mutex_exit(oobj->vmobjlock);
 > 	}
 
 Alright, so my initial guess was wrong.
 
 > [...]
 >> But this guess would have to be verified. You should probably try to
 >> assign your program to a given core - and this, early, _before_ your
 >> program starts doing heavy stuff. schedctl, or pset would be even
 >> better. If I'm right, it should "fix" the slowdown.
 > 
 > Why would the synchronization with other CPUs go away?
 
 Because if my initial guess was correct, by assigning your program to a cpu
 you would have guaranteed that your pmap was not loaded on other cpus; and
 as a result you wouldn't have had to send IPIs to them when changing a page
 in your pmap.
 
 But I may have another guess now. Here the call path is: pmap_remove ->
 pmap_remove_ptes -> pmap_remove_pte -> pmap_tlb_shootdown. You can see that
 the last one gets called only when ((opte & PG_U) != 0). That is, only when
 the page has already been used (read/write) by a core on the machine.
 
 It would be nice if you could verify, somehow, two things:
 
   o Does the 'UMAP_MAPPING_CACHED' branch get taken the same way with and
     without your N-1 user threads. If there is a clear difference here, then
     it means the problem is that UBC does not scale. Otherwise:
 
   o When the 'UMAP_MAPPING_CACHED' branch gets taken, does the 'PG_U' one get
     taken too afterwards with and without your N-1 user threads. If there is a
     clear difference here, then it means that for some reason an idle kernel
     thread on a remote cpu decided to touch the page before you did yourself,
     which resulted in PG_U being set, and the IPIs being sent to everybody.
 
 Maxime