Re: Potential Improvements on Lua Support?

[Qingyao Sun <sunqingyao19970825%icloud.com@localhost>]

Dear List,

I forget to CC the mailing list while replying to kre. Sorry for the confusion! Also I am using the @icloud address hereafter as per kre’s preference.

It just came to me that we can do in-kernel machine learning by making the kernel talk to a userland process capable of doing the heavy lifting with BLAS/LAPACK or even mlpack. I don’t know much about OS, but my understanding is that kernel/userland context switches are expansive, and that the whole point of having Lua in kernel (as opposed to implementing the heuristics with a userland Lua interpreter that calls into the kernel) is to eliminate that overhead, so I would suggest leaving machine learning for now and focusing on implementing a ring-buffer-analyzing toy example first.


Bests,
Qingyao

> On Mar 1, 2023, at 10:27, Qingyao Sun <sunqingyao19970825%gmail.com@localhost> wrote:
> 
> Dear kre,
> 
> Thanks for the reply! See inline below:
> 
> > On Wed, Mar 1, 2023 at 7:23 AM Robert Elz <kre%netbsd.org@localhost> wrote:
> 
> 
> > I like this idea a lot more than the inetd/rc.d idea.   I am not sure that
> "Improvements on Lua Support" is a good title, unless you're actually planning
> on working on the kernel Lua implementation, and it doesn't sound like that.
> 
> I also find this idea more interesting. People seem to have different preferences/opinions on the init system, so I guess it would be hard for me to propose something everybody is happy with. You are right that I don't plan to work on the kernel Lua implementation itself, but I would need to expose some kernel functionalities to Lua for the kernel heuristics to use, or perhaps even modify it to add floating-point support (see below).
>  
> 
> > I'd suggest something more like "Using Lua scripts to improve kernel heuristics"
> or something like that.   That is, if that is what you are really proposing,
> using your machine learning knowledge in areas where it might benefit the kernel.
> 
> The title you are suggesting is definitely more suitable. I'm not sure whether we should use machine learning though. AFAIK the in-kernel Lua interpreter does not support floating-point arithmetic because context switches of the floating-point unit are overly expensive, without which we cannot even do linear regression. Moreover, ML algorithms tend to be computationally extensive, but of course, some benchmarking is required. I was thinking something like collecting some historical data into a ring buffer and checking it periodically to see if we are too optimistic or pessimistic. Anyway, I would say the core idea behind this proposal is to make the kernel heuristics more sophisticated with the expressiveness of Lua.
> 
> 
> > I cannot help with Lua, I know it exists, but that's the extent of my knowledge
> on that front.  I know something about filesystems, but not the details of our
> current implementation.  I would however like to see this idea applied to CPU
> scheduelling, particularly when there are different classes of CPUs available
> (ie: they're not all identical to each other - while any can do any work, they
> don't all do it equally well).    I think we're currently lacking in that area.
> 
> That sounds interesting as well. Many ARM chips seem to have that big.LITTLE technology, including Apple Silicon. They can definitely benefit from this improvement.
>  
> 
> > For filesystems, my gut feeling is that a better task to take on might be
> working out which cached data ought be ejected, and which retained, rather
> than (or perhaps just before) working on prefetching algorithms (particularly
> as "drives" get much faster, and delays to fetch blocks are reducing considerably).
> 
> That makes sense. Just for the record, we can improve kernel heuristics in other areas like I/O latency prediction and page warmth classification (see the LAKE paper I linked in OP). I don't have a strong opinion on which specific subsystem to work on.
>  
> 
> > So, I'm prepared to help with this, with two caveats.   First, you need to also
> find someone knowledgable (enough) with the kernel Lua implementation and use
> to assist with that (and Lua scripting) who is willing, and available, to assist
> with this.  Some occasional filesystem/uvm and schedueller assistance might be
> useful as well.  And second, you need to receive e-mail somewhere other than at
> gmail - gmail refuses mail from me (mail I normally send, rather than mail this
> way, from a NetBSD host, which is not at all convenient to send), and even forwarding
> mail from me to them from some other mailbox is likely to fail.   I have zero interest
> in bowing to gmail's stupid requirements to avoid that problem, so if I am to assist
> you need to be able to receive mail somewhere else.   Probably at UT I'd guess.
> 
> Thanks for your help! Let's wait for some Lua wizard to jump in then. Regarding email, I will probably receive a @utexas.edu email address in a few months (I am enrolling in 2023 fall), and I would be happy to use that. Meanwhile, we can communicate through sunqingyao19970825%icloud.com@localhost if gmail does not work for you.
>  
> 
> > kre