Re: summer of code - scrub feature

[Alistair Crooks <agc%pkgsrc.org@localhost>]

On Sun, Mar 22, 2009 at 09:20:47PM -0400, Thor Lancelot Simon wrote:
> On Mon, Mar 23, 2009 at 01:13:16AM +0000, Alistair Crooks wrote:
> > On Sun, Mar 22, 2009 at 08:07:06PM -0400, Thor Lancelot Simon wrote:
> >  
> > > The other problem is that the WAPBL journal will have pieces of file
> > > data in it.  Aggressively overwriting the log after transactions have
> > > been committed will _murder_ performance.  The best solution to all this
> > > is to just use cgd!
> > 
> > Requiring the use of cgd is certainly one way of addressing the
> > problem.  Another way would be to give up using computers completely
> > since we can never be sure that data will not be copied without our
> > prior knowledge and sent back to the HQ of the cabal's mothership.
> 
> Let me try to state the actual performance constraints as I understand
> them, more succinctly:
> 
>       1) Overwrite-on-erase for every erase will cost more than
>          encrypt-on-write for every write, on almost any modern
>          platform and for almost any workload.

A mount option would do it for the last unlink, and every truncate.
Per-file system and user flags are a bit finer-grained. For them,
no-one is talking about "every erase". Like I said before, if I
accidentally remove my .ssh or .gnupg directory, it's too late.
Having system and user scrub flags for the files on there means
taking the worry out of that kind of thing.
 
>       2) WAPBL makes this even more so.  Much more so.

Absolutely.
 
>       3) We expect that almost all users will use WAPBL for almost
>          all workloads.

Your (3) is really (2) with energetic hands, but whatever...  WAPBL is
experimental right now, although it's heaps better than it used to be. 
(Many thanks to Andrew, pooka and all the others who worked on it). We
are all behind the WAPBL work, but relying on it solely post NetBSD-6.0
is not really what I had in mind. I think we're going to see more file
systems in the medium term - but I do take your point - any file system
with journalling will have to address this aspect. For now, it's not
part of the scope of the SoC project.
 
> This suggests to me that cgd is always a better solution to the problem
> we are trying to solve here, that data might be read after files are
> erased.
> 
> Morever, cgd ensures that you don't even have to worry about the journal
> or spared sectors.  It is a strictly better solution.

I agree that it's a better solution, but it's one that may not be
applicable in all cases.

Such cases could be:  because the file system already exists and can't
be changed for some reason; because it's the root filesystem, and
no-one has committed the bits to make a cgd / viable, workable and
usable without extreme discomfort; for travelling to regions and areas
where cgd cannot legally be used; there may be more, I don't know.

If you're going down this route, you should also be encrypting any
swap partitions, of course, using tempested hardware, and wearing tin
foil on your head.  As ever, this is a question of what's possible,
and of securing yourself as much as is economically and comfortably
possible.

It's also a good way for a student to start to learn about some of these
issues, and to explore some areas of the kernel where we've been lacking
over the last few years. As a gentle introduction to that, I believe this
is a good project for a student to gain good experience, and for the
NetBSD project to gain some interesting insight and experience. Dancing
up and down with the mantra "but it's not a complete solution" might be
true, and entertaining for us, but not, ultimately, that useful.

Regards,
Alistair