Re: Portmasters alert: was => Re: uvm physseg - put the lid on it.

[Cherry G. Mathew <cherry%zyx.in@localhost>]

>>>>> "Cherry" == Cherry G Mathew <cherry%zyx.in@localhost> writes:

>>>>> "Cherry" == Cherry G Mathew <cherry%zyx.in@localhost> writes:
>>>>> "Cherry" == Cherry G Mathew <cherry%zyx.in@localhost> writes:
>>>>> "Cherry" == Cherry G Mathew <cherry%zyx.in@localhost> writes:
    Cherry> Hi Everyone, Here's a patch to put the vm physseg stuff
    Cherry> behind bars.

    Cherry> I've made some pretty intrusive changes, esp. to the boot
    Cherry> time msgbuf stuff, so if portmasters could have a quick look
    Cherry> at this, I'll be able to go ahead and change the equivalents
    Cherry> on other ports.

    Cherry> I have a bunch of tests that actually test these functions
    Cherry> in userland via rump. It's pretty cool :-)

    Cherry> Ultimately, I want to change the backing implementation to
    Cherry> use an rb tree rather than a static array, so that the
    Cherry> plug/unplug is dynamic. Obviously, this patch is not any
    Cherry> attempt at an "API" - it's just code reorg, to make the next
    Cherry> steps easier.

    >>>> PS: Please reply everyone for public replies, since some on the
    >>>> Cc: list may not be subscribed to tech-kern@

    >>>> Hi Everyone,

    >>>> Thanks for the feedback to folks who sent them in.

    >>>> Here's some further patches. I've broken them out for your
    >>>> convenience:

    >>>> i) atf(7) tests for the new consolidated "API". These are with
    >>>> the caveat that they will change along with the implementation
    >>>> code, when the actual unplug API hits the road. Santhosh (Cc:ed)
    >>>> collaborated closely with me for these.

    >>>> ftp://ftp.netbsd.org/pub/NetBSD/misc/cherry/uvm_physmem/cvs.tests.sys.uvm.diff

    >>>> ii) uvm(9) changes. The #ifdef _KERNEL_OPT is required for the
    >>>> rump(9) testing code (see (ii) below:

    >>>> ftp://ftp.netbsd.org/pub/NetBSD/misc/cherry/uvm_physmem/cvs.sys.uvm.diff

    >>>> iii) port specific changes. I have only tested i386 and amd64. I
    >>>> have compiled all ports in these diffs.

    >>>> *Portmasters*, it would be immensely helpfull if you could test
    >>>> them on your respective ports please.

    >>>> ftp://ftp.netbsd.org/pub/NetBSD/misc/cherry/uvm_physmem/cvs.sys.arch.diff

    >>>> Please, no KNF comments yet - I will clean up in a final pass -
    >>>> I'm really looking for testing and reporting on function (or
    >>>> breakage!)

    >>>> I will aim to cut a branch in CVS RSN.

    >>>> Many Thanks, -- ~cherry

    Cherry> Have updated the patches with some more stuff:

    Cherry> i) Reverted to the current (imho broken) use of
    Cherry> start/avail_start,end/avail_end in boot time segment
    Cherry> "stealing".  This was to minimise divergence with the
    Cherry> current implementation.

    Cherry> ii) Changed the assumption of "handles" to memory segments
    Cherry> from mutable to immutable. A corresponding test was added
    Cherry> (which fails). This is in preparation for changing from the
    Cherry> current array based implementation to a more dynamic
    Cherry> implementation where the handle pointing to backing nodes
    Cherry> will be immutable.

    Cherry> iii) Introduced specific error constants for specific error
    Cherry> conditions in segment node handle traversal.

    Cherry> iv) Fixed a bug introduced in calculating the number of
    Cherry> backing struct vm_page;s allocated per segment.


    Cherry> I've changed my mind about cutting a new branch. The rbtree
    Cherry> backed node management code is wip (it passes all the
    Cherry> userland tests, but needs boot time memory alloc sorted ). I
    Cherry> will cut the new branch with the rbtree code. This set of
    Cherry> patches will have tests which will very likely remain
    Cherry> unchanged wrt the newer implementation with rbtrees.

    Cherry> Santhosh is now working on managing backing pgs[] in the
    Cherry> case of unplug(). This may involve some changes to
    Cherry> extent(9). We're gravitating towards using extent(9) instead
    Cherry> of vmem(9) since it provides a much more flexible backend
    Cherry> memory allocation (the caveat of alloc on fragmenting
    Cherry> free()s can be addressed by using pool(9), iiuc right now -
    Cherry> still under design)

    Cherry> Those are my unfinished thoughts for now.

    Cherry> Updated with the rbtree implementation for segment nodes
    Cherry> (instead of an array of segments). Thank you Santhosh
    Cherry> (Cc:ed) for a tight, simple implementation with excellent
    Cherry> testing coverage - the native build booted straight off with
    Cherry> no debugging required.

    Cherry> I've added a little hack for boot time, which uses a static
    Cherry> array identical to the original implementation only at boot
    Cherry> time. Once kmem(9) is up, it uses regular kernel
    Cherry> memory. free()s back to the array are ignored (and therefore
    Cherry> leaky) - this is tested in the test cases.

    Cherry> I've also updated all ports with the init call
    Cherry> uvm_physmem_init(), with absolutely no idea about the
    Cherry> appropriateness of its location.

    Cherry> ftp://ftp.netbsd.org/pub/NetBSD/misc/cherry/uvm_physmem

    Cherry> Please note that the diffs themselves are versioned via RCS
    Cherry> - so you can look at previous versions of the diffs, if you
    Cherry> like.

    Cherry> Have fun!

    Cherry> PS: What I'd love to have is people trying these patches on
    Cherry> ports other than amd64/i386 and reporting back on breakage
    Cherry> etc. (or fixing things!)

    Cherry> Next stop, seg->pgs[] backing management. This is a bit
    Cherry> trickier (as mentioned in my previous email).


    Cherry> Hi Everyone,

    Cherry> I've managed to implement the backing pgs[] array management using
    Cherry> extent(9).

    Cherry> Please refresh your copy of:
    Cherry> ftp://ftp.netbsd.org/pub/NetBSD/misc/cherry/uvm_physmem

    Cherry> Special thanks to all the people who helped with testing - and skrll@
    Cherry> for patches.

    Cherry> More testing appreciated. 

    Cherry> This set of patches now includes a usecase for the balloon(4) driver to
    Cherry> hotplug in memory from the hypervisor. This is functional with caveats
    Cherry> that I believe are specific to Xen (the dom0 xenstore goes crazy after a
    Cherry> couple of hotplugs).

    Cherry> I believe I now understand what the original intent of
    Cherry> avail_start/avail_end was. This may be relevant to exploit the hotplug
    Cherry> architecture fully. iiuc, (avail_start,avail_end) tuples marked the
    Cherry> range within a segment, where physical memory was actually registered to
    Cherry> uvm(9) via uvm_pagefree(). The memory in between, ie; 
    Cherry> (start, avail_start) and (avail_end, end) are used to indicate physical
    Cherry> memory that is in use, but not used by uvm(9). This is abused by MD
    Cherry> code, to "steal" memory at boot time.

    Cherry> I believe, a more formal way to do this, would be to use the extent(9)
    Cherry> API to manage all of the paddr_t space, thus formalising this
    Cherry> relationship with consumers. I'm not inclined to look at this in the
    Cherry> context of hotplug, due to "mission creep".

    Cherry> Anyway, thoughts/testing welcome, and here's the usual threats about
    Cherry> cutting a branch.

Committed a preview on -current:
http://mail-index.netbsd.org/source-changes/2016/12/19/msg080026.html

Here's the "switch over" patch:
http://ftp.netbsd.org/pub/NetBSD/misc/cherry/uvm_physmem/src.diff

These have been tested on various architectures by a few people, so
unless there's vigorous disagreement, I will commit this once -current
is stable again.

Cheers,
-- 
~cherry