Re: file-backed cgd backup question

[Paul Ripke <stix%stix.id.au@localhost>]

On Sun, Oct 22, 2023 at 09:40:53AM -0400, Greg Troxel wrote:
> mlelstv%serpens.de@localhost (Michael van Elst) writes:
> 
> > gdt%lexort.com@localhost (Greg Troxel) writes:
> >
> >>> vnd opens the backing file when the unit is created and closes
> >>> the backing file when the unit is destroyed. Then you can access
> >>> the file again.
> >
> >>Is there a guarantee of cache consistency for writes before and reads
> >>after?
> >
> > Before the unit is created you can access the file and after the
> > unit is destroyed you can access the file. That's always safe.
> 
> Sorry if I'm failing to understand something obvious, but with a caching
> layer that has file contents, how are the cache contents invalidated?
> 
> Specifically (but loosely in commands)
> 
>   let's assume the vnd is small and there is a lot of RAM available
> 
>   process opens the file and reads it
> 
>   vnconfig
> 
>   mount vnd0 /mnt
> 
>   date > /mnt/somefile
> 
>   umount /mnt
> 
>   vnconfig -u
> 
>   process opens the file and reads it
> 
> Without fs cache invalidation, stale data can be returned.
> 
> If there is explicit invalidation, it would be nice to say that
> precisely but I am not understanding that it is there.  Reading vnd.c, I
> don't see any cache invalidation on detach.   The only explicit
> invalidation I find is in setcred from VNDIOCSET.
> 
> I guess that prevents the above, but doesn't prevent
> 
>   vnconfig
> 
>   mount
> 
>   read backing file
> 
>   write to mount
> 
>   unmount
> 
>   detach
> 
>   read backing file
> 
> so maybe we need a vinvalbuf on detach?
> 
> > I also think that when the unit is configured but not opened
> > (by device access or mounts) it is safe to access the file.
> 
> As I read the code, reads are ok but will leave possibly stale data in
> the cache for post-close.
> 
> >>> The data is written directly to the allocated blocks of the file.
> >>> So exclusively opening  the backing file _or_ the vnd unit should
> >>> also be safe. But that's not much different from accessing any file
> >>> concurrently, which also leads to "corrupt", inconsistent backups.
> >
> >>That's a different kind of corrupt.
> >
> > Yes, but in the end it's the same, the "backup" isn't usuable.
> 
> I am expecting that after deconfiguring, a read of the entire file is
> guaranteed consistent, but I think we're missing invalidate on close.
> 
> > You cannot access the backing file to get a consistent state of the
> > data while a unit is in use. And that's independent of how vnd accesses
> > the bits.
> 
> Agreed; that's more or less like using a backup program on database
> files while the database is running.
> 
> > N.B. if you want to talk about dangers, think about fdiscard(). I
> > doubt that it is safe in the context of the vnd optimization.
> 
> It seems clear that pretty much any file operations are unsafe while the
> vnd is configured.  That seems like an entirely reasonable situation and
> if that's the rule, easy to document.
> 
> I wrote a test script and it shows that stale reads happen.  When I run
> this on UFS2 (netbsd-10), I find that all 4 files are all zero.  When I
> run it on zfs (also netbsd-10), I find that 000 and 001 are all zero and
> 002 and 003 are the same.  (I am guessing that zfs doesn't use the
> direct operations, or caches differently; here I haven't the slightest
> idea what is happening.)
> 
> 10 minutes later, reading VND is still all zeros.  With a new vnconfig,
> it still reads as all zeros.
> 
> 
> #!/bin/sh
> 
> dd if=/dev/zero of=VND bs=1m count=1
> cat VND > VND.000
> vnconfig vnd0 VND
> cat VND > VND.001
> newfs /dev/rvnd0a
> cat VND > VND.002
> vnconfig -u vnd0
> cat VND > VND.003

At least this DTRT:

dd if=VND of=VND.004 iflag=direct

This reminds me of the way AIX handled O_DIRECT vs mmap, etc. From
https://www.ibm.com/docs/en/aix/7.2?topic=tuning-direct-io:

 To avoid consistency issues, if there are multiple calls to open a file
 and one or more of the calls did not specify O_DIRECT and another open
 specified O_DIRECT, the file stays in the normal cached I/O mode.
 Similarly, if the file is mapped into memory through the shmat() or
 mmap() system calls, it stays in normal cached mode. If the last
 conflicting, non-direct access is eliminated, then the file system will
 move the file into direct I/O mode (either by using the close(),
 munmap(), or shmdt() subroutines). Changing from normal mode to direct
 I/O mode can be expensive because all modified pages in memory will have
 to be flushed to disk at that point.

An elegant, albeit complex, solution.

-- 
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
 discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.