Re: fs-independent quotas

[David Holland <dholland-tech%netbsd.org@localhost>]

On Thu, Oct 20, 2011 at 06:00:28PM +0200, Manuel Bouyer wrote:
 > >  > It's certainly less trouble to send back to userland the whole set of
 > >  > data - especially if what userland wants is the whole set of data
 > >  > (I can't see what a partial read of quota would be usefull for).
 > > 
 > > No, no it really isn't. Suppose there are, say, 50,000 users, so to
 > > send back the whole works you have to accumulate 100,000 quota entries
 > > in a gigantic blob... a machine with 50,000 users will have enough RAM
 > > for this but that doesn't mean that allocating a contiguous chunk of
 > > kernel memory that large is easy or desirable. Far better to read it
 > > out a couple hundred at a time.
 > 
 > We're talking a few MB of ram here, isn't it ? the kernel can certainly
 > allocate this without troubles (other subsystems do).

The proplib'd and XMLified complete dump for 50,000 users will
probably make a blob of between 10 and 20 MB. (Note: this is an
estimate; I haven't checked the size by trying it. It might be larger.
I'd be surprised if it were much smaller.)

I don't see why it's desirable to manifest such large objects when
it's easily avoidable.

 > > There are two design truisms for database stuff that apply here:
 > > first, you always end up wanting cursors, and second, you always end
 > > up wanting bulk get (and not just single get) from those cursors. So
 > > it's usually a good idea to anticipate this and design it all in up
 > > front.
 > 
 > Maybe ... I know that in the end I want the whole set of data and not
 > just a part of it.

Yes, probably. The cursor API I've floated so far is not general
enough to support much else. Although it could be made more general.

 > But if you believe it's needed this can easily be added to the
 > existing quotactl(2) (it would just be a new command).

Yes, perhaps it could... but why? What's to be gained by using a
baroque proplib encoding of what can otherwise be handled as an array
of simple structs?

I remember asking this question when you first proposed the proplib
interface last spring, and never really got a clear answer.

 > >  > > The reason to wrap the position in a cursor abstraction is to allow
 > >  > > flexibility about how the position is represented.
 > >  > 
 > >  > But then the cursor would still be stored in userland ?
 > > 
 > > That's the idea, like reading a file with pread().
 > > 
 > > I think the kernel should know, or at least be able to know, how many
 > > cursors are currently open; but I don't think there's any need to keep
 > > the cursor state itself in the kernel.
 > 
 > So you want a quotaopen/quotaclose, with a file descriptor (or something
 > similar) ?

The proposed API already has explicit open and close for cursors; what
I'm saying is that this should be exposed to the kernel. (Open already
has to be, to initialize the cursor position; close should be, so the
filesystem can if necessary know if there are cursors open at any
given time. Otherwise you can get into trouble; see for example nfsd
and readdir.)

-- 
David A. Holland
dholland%netbsd.org@localhost