I need some references on how the VM system of NetBSD works.

It's mostly the Mach 2.5 VM system, with some changes from Mach 3.0
merged in.  The most useful references are probably the original Mach
papers.

There are aff few convenient ways to hook a compressor into the VM
system:

* Put it in the swap pager, and have pages compressed and decompressed
only as they're stored on or fetched from disk.

* Put it in the swap pager, have some reserved chunk of memory for
holding compressed pages, and `stage' pages to be swapped out through
the reserved area.  This gives you a 3-level system: RAM, compressed
RAM, and disk.

* Similar to the previous, but do the compression preemptively from
another system process, keeping statistics on how long a page has been
inactive, and perhaps the ratio of active/inactive time.

I think this should actually be fairly straightforward, but since I
haven't done it, you should take that with a few grains of salt.