Proposal: Restore malloc(9) interface

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

I propose to deprecate the kmem(9) interface and go back to the
malloc(9) interface.


1. The main difference is that the malloc(9) interface enables
   attribution of memory usage: how many bytes have been used for this
   purpose vs that purpose, partitioned by named malloc tags like
   M_MBUF or M_ACPI.  The conversion to the kmem(9) interface lost all
   this valuable diagnostic information.

   I've personally spent probably dozens of hours over the last year
   or two puzzling over `vmstat -m' output to guess which subsystem
   might be leaking memory based on allocation sizes and which
   kmem-NNNNN pool looks fishy.  This is extremely frustrating and a
   huge waste of time to recover information we used to gather and
   report systematically.

2. A secondary difference is reduced diffs from FreeBSD and OpenBSD
   drivers if we use malloc(9).

3. A small difference is that kmem(9) distinguishes legacy allocation
   from interrupt context, kmem_intr_alloc/free, from front ends that
   forbid that, kmem_alloc/free.

   I'm not sure this has provided much valuable diagnostic
   information, but it has provided a lot of frustrating crashes.  If
   we want the same frustrating crashes we could introduce an M_INTR
   flag which is mandatory when calling malloc from interrupt context.

Note: I am NOT proposing any substantive changes to the implementation
of the allocator -- I'm just proposing that we go back to the old
_interface_, using the new pool-cache-based _implementation_, and to
add lightweight per-CPU, per-tag usage counting to the malloc and free
paths.

Nor am I suggesting changing anything about uvm_km(9), pool_cache(9),
or anything else -- just changing kmem_alloc(N, KM_[NO]SLEEP) back to
malloc(N, T, M_NOWAIT/WAITOK) and kmem_free(P, N) back to free(P, T),
or possibly free(P, T, N) like OpenBSD does.

Thoughts?


I asked for rationale for the kmem(9) interface last year, and none of
the answers gave any compelling reason to have changed interfaces in
the first place or to finish the conversion now:

https://mail-index.netbsd.org/tech-kern/2022/10/29/msg028498.html

As far as I can tell, we just spent umpteen hundreds of hours on
engineering effort over the last decade to convert various drivers and
subsystems from malloc(9) to kmem(9), in exchange for the loss of
valuable diagnostic information about leaks, for increased cost to
porting drivers, and for crashes when old subsystems newly converted
to kmem(9) still allocate from interrupt context.