kmem pool for half pagesize is very wasteful

[matthew green <mrg%eterna.com.au@localhost>]

hi folks.


while we were debugging some memory starvation issues i noticed that
the "kmem-02048" pool only has 1 item per page on a system with 4KiB
pages, same similarly "kmem-04096" on 8KiB page systems.  i assume
this also occurs on 16KiB page systems for the "kmem-08192" pool.

this happens because the pool redzone increases the size from 2048
bytes to 2048+CACHE_LINE_SIZE bytes.

this feels extremely wasteful to me.  for the common 4K page size
case, it's 2048+64 bytes of header of useful data, plus the 64 bytes
lost for the redzone alignment, and thus 1920 bytes of lost space.

the lost space is similarly just under 1/2 for large page size
systems.

for the smaller kmem pools, the lost space seems not great, but
significantly less than the previous (1024*3 + 2*64*3 = 3456 bytes
of useful info).  46.8% lost vs 15.6% lost, while not great, seems
like a reasonable compromise to almost half the memory required
for the kmem-02048 pool.

this patch avoids this problem:

   https://www.netbsd.org/~mrg/poolwaste.diff

comments?


.mrg.