Re: kmem pool for half pagesize is very wasteful

[Chuck Silvers <chuq%chuq.com@localhost>]

On Tue, Feb 23, 2021 at 06:25:50PM +1100, matthew green wrote:
> hi folks.
> 
> 
> while we were debugging some memory starvation issues i noticed that
> the "kmem-02048" pool only has 1 item per page on a system with 4KiB
> pages, same similarly "kmem-04096" on 8KiB page systems.  i assume
> this also occurs on 16KiB page systems for the "kmem-08192" pool.
> 
> this happens because the pool redzone increases the size from 2048
> bytes to 2048+CACHE_LINE_SIZE bytes.
> 
> this feels extremely wasteful to me.  for the common 4K page size
> case, it's 2048+64 bytes of header of useful data, plus the 64 bytes
> lost for the redzone alignment, and thus 1920 bytes of lost space.
> 
> the lost space is similarly just under 1/2 for large page size
> systems.
> 
> for the smaller kmem pools, the lost space seems not great, but
> significantly less than the previous (1024*3 + 2*64*3 = 3456 bytes
> of useful info).  46.8% lost vs 15.6% lost, while not great, seems
> like a reasonable compromise to almost half the memory required
> for the kmem-02048 pool.
> 
> this patch avoids this problem:
> 
>    https://www.netbsd.org/~mrg/poolwaste.diff
> 
> comments?

this patch is fine for now, please commit.

in the longer term, I think it would be good to use even larger pool pages
for large pool objects on systems that have relatively large amount of memory.
even with your patch, a 1k pool object on a system with a 4k VM page size
still has 33% overhead for the redzone, which is a lot for something that
is enabled by DIAGNOSTIC and is thus supposed to be "inexpensive".
there's a tradeoff here in that using a pool page size that matches the
VM page size allows us to use the direct map, whereas with a larger
pool page size we can't use the direct map (at least effectively can't today),
but for pools that already use a pool page size that is larger than
the VM page size (eg. buf16k using a 64k pool page size) we already
aren't using the direct map, so there's no real penalty for increasing
the pool page size even further, as long as the larger pool page size
is still a tiny percentage of RAM and KVA.  we can choose the pool page size
such that the overhead of the redzone is bounded by whatever percentage
we would like.  this way we can use a redzone for most pools while
still keeping the overhead down to a reasonable level.

-Chuck