Re: libquota proposal

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

[ replying to 4 messages in one, hoping to reduce the number of branches
  in this thread ...]

On Thu, Mar 24, 2011 at 12:19:23AM +0000, David Holland wrote:
> On Wed, Mar 23, 2011 at 05:10:08PM +0100, Manuel Bouyer wrote:
>  > > > > No, it doesn't. Even before you touched anything, they were only
>  > > > > scribbling directly as a fallback if the kernel operations failed.
>  > > > > The kernel operations should not fail in any case where scribbling
>  > > > > directly makes sense; furthermore there's no need at all to deal with
>  > > > > the case where the fs isn't mounted.
>  > > > 
>  > > > repquota at last needs them: it doesn't have any way to get a list
>  > > > of quotas otherwise
>  > > 
>  > > That sounds like a bug.
>  > 
>  > it's the way it works now ~forever ...
> 
> Yes, but wasn't one of the purposes of using plists to make that
> possible?

Yes, but I didn't say it would be mandatory :)

> 
> And if not, it should have been regardless, as the interface to quotas
> needs to be fs- and format-independent.

quotactl(2) is, as you can pass plists down to fs-specific parsers.
The userland tools are very tied to the UFS quota semantic, if something
else ever appears some major work will need to be done (including
redesigning the user interface)


> 
>  > >  > (and it's also part of the migration to quota2,
>  > >  > with repquota -x).
>  > > 
>  > > ...wait, we're exposing the plists directly to the user?
>  > 
>  > yes of course. that's one of the points of using plists, isn't it ?
> 
> ...no, I wouldn't say so. Things the user is meant to look at it
> shouldn't generally be plists,

of course, but he can get a plist if he wants it (and for this specific case,
he doesn't have to look at the plist, just feed it to quotactl(8))

> as plists weren't ever meant to be
> human-readable and aren't very; meanwhile, things that are meant to be
> fed to shell scripts are much better off being tables.
> 
>  > right now I have a perl script which parses repquota output, with all the
>  > bugs/misfunctions this implies, and then calls edquota.  With the plist
>  > interface, I can have my perl script parse plists instead, and generate
>  > plists to send to the kernel.
> 
> That 

I don't see the point to reinvent a new machine-parsable text format,
when we have one, with existing libraries to handle them (there
are perl modules around to deal with plists - I guess there are modules
for other scripting languages too).

> 
>  > > Shouldn't the migration be a single transparent tunefs operation?
>  > 
>  > it's hard to have it fully automated.
> 
> er, why?

because there a different senarii: you may want to convert an existing
filesystem, or you may want to move the quota along with datas when
migrating to a new storage device (or possibly to a new system,
of different architecture, or maybe one day to a different filesystem - 
I'm sure I'm missing use cases here).
Having one tool export the limits in a text format, and another one
importing them from this text format is the most flexible way of doing it
(it also allows you to do automated transforms on the datas in the
process). To me it looks like the unix way or doing it.

> 
>  > >  > > In the new world order all userland quota operations go through the
>  > >  > > kernel interface so they can interact successfully with filesystems
>  > >  > > using either the old or new quota layouts, or with new filesystems
>  > >  > > that may have their own different quota layouts, like zfs or 
> whatever
>  > >  > > else. Right?
>  > >  > 
>  > >  > right. Exept that the "getall" command is not supported for quota1,
>  > >  > repquota does the job itself.
>  > > 
>  > > uh, why not? that *is* a bug.
>  > 
>  > because to get the whole set of quota with quota1, you have to read
>  > all of the (potentially sparse) quota file (it can be up to 128GB
>  > if you use uid -1).  I don't want to do this in the kernel ...
> 
> Much better to do it from inside the FS, where you can pretty easily
> skip over nonmaterialized blocks, than to violate modularity and
> abstraction boundaries so you can stream all 128GB of zeros to
> userland.

I don't know how to do this. Now if someone wants to do it I won't mind :)

On Thu, Mar 24, 2011 at 12:22:34AM +0000, David Holland wrote:
> On Wed, Mar 23, 2011 at 05:11:21PM +0100, Manuel Bouyer wrote:
>  >>>>>>>> (also, edquota and repquota seem fs-independent to me...)
>  >>>>>>> 
>  >>>>>>> no, they're not: they can directly the quota1 file specified in the
>  >>>>>>> fstab if quotactl fails or the filesystem is not mounted.
>  >>>>>> 
>  >>>>>> That's a bug, or more accurately legacy behavior that doesn't need to
>  >>>>>> be supported. 
>  >>>>> 
>  >>>>> of course it's not nice. But we're talking about existing code calling 
> the
>  >>>>> legacy quotactl. If we're going to change it to not check the fstab
>  >>>>> options any more, we may as well change it to use libquota.
>  >>>> 
>  >>>> I don't understand - surely edquota and repquota go through your
>  >>>> proplib interface now?
>  >>> 
>  >>> We were talking about code like netatalk, which is why I propose
>  >>> a public library for this.
>  >>
>  >> Uh, now I really don't understand.
>  > 
>  > what don't you understand ? 
> 
> You said that edquota and repquota aren't fs-independent; I asked why,
> given that they should be going through the new quota API, and you
> suddenly started talking about legacy code and netatalk.

OK, you were talking specifically about the tools, I was talking about
existing userland code which check for fstab options becore calling quotactl
(this is not limited to our tools).


On Thu, Mar 24, 2011 at 12:28:00AM +0000, David Holland wrote:
> On Sat, Mar 19, 2011 at 02:07:50PM -0400, Christos Zoulas wrote:
>  > | At this point, in the source 'quota1' is used for the old
>  > | quota format, 'quota2' for the new one and 'quota' for the few things
>  > | that are common.
>  > 
>  > Are we planning to keep quota1 around for more than 6.0? If not, it will
>  > look funny to have quota2 in the future...
> 
> We have to keep it, because for now at least lfs doesn't support the
> quota2 format. (And, does ffs without WAPBL?) Plus we should retain
> support for historic/portable ffs.

AFAIK lfs has never supported quota1 either (quotacheck has no support
for it, nor quotaon). Adding quota2 support for lfs is probably easy for
someone familiar with LFS (what's missing is probably only
a lfs_quota2_mount and associated data in the equivalent of ffs
superblock - it seems the calls to quota check functions are already there),
probably easier than adding lfs support to quotacheck.
yes, quota2 works on ffs without wapbl too.

> 
> ...however, the material involved really ought to be isolated within
> src/sys/ufs. That doesn't appear to be the case now....

well, some userland tools can't avoid it if we want to keep
support (e.g quotacheck).

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--