pool_cache_invalidate(9) (wrong?) semantic -- enable xcall invalidation

[Jean-Yves Migeon <jeanyves.migeon%free.fr@localhost>]

I would like to turn back on the xcall(9) block found in 
pool_cache_invalidate(), now that rmind@ has implemented high priority 
cross calls [1].

Reasons are following:

A while back I had to find a way to invalidate all allocations done in a 
pool_cache(9). Unfortunately, the pool_cache_invalidate(9) function is 
misleading: it only invalidates and reclaim objects cached in the 
"global" lists, and not those cache in the per-cpu ones.

This has unexpected consequences; all consumers of the pool_cache(9) API 
have to ensure that the objects they get out of it are still valid (see 
pmap allocation in x86 pmap_create() as an example).

In my case (PAE support for Xen save/restore), page directories may be 
cached in a per-cpu pool; not invalidating them upon suspend will lead 
to wrong type pinning and completely break restore.

I had to introduce pool_cache_invalidate_local(9) function back then to 
permit each CPU to deplete their pool and destroy all cached objects. 
This made the API abstraction leaky, and was quickly backed out [2].

Unfortunately the cross-call block had to be commented out later on, as 
it only allowed pools with low IPL to be managed [3]. Now that this is 
fixed, I'd like to enable it again. XC_HIGHPRI makes cross calls (for 
x86) via IPIs combined with a softint that runs on the targeted CPUs. So 
I would expect pool_cache_invalidate() to be safe for pools used in 
interrupt context too.

Opinions?

[1] http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/subr_xcall.c#rev1.12

[2] http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/subr_pool.c#rev1.176

[3] http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/subr_pool.c#rev1.182

--
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost