Current-Users archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: file-backed cgd backup question
On Sun, Oct 22, 2023 at 09:40:53AM -0400, Greg Troxel wrote:
> mlelstv%serpens.de@localhost (Michael van Elst) writes:
>
> > gdt%lexort.com@localhost (Greg Troxel) writes:
> >
> >>> vnd opens the backing file when the unit is created and closes
> >>> the backing file when the unit is destroyed. Then you can access
> >>> the file again.
> >
> >>Is there a guarantee of cache consistency for writes before and reads
> >>after?
> >
> > Before the unit is created you can access the file and after the
> > unit is destroyed you can access the file. That's always safe.
>
> Sorry if I'm failing to understand something obvious, but with a caching
> layer that has file contents, how are the cache contents invalidated?
>
> Specifically (but loosely in commands)
>
> let's assume the vnd is small and there is a lot of RAM available
>
> process opens the file and reads it
>
> vnconfig
>
> mount vnd0 /mnt
>
> date > /mnt/somefile
>
> umount /mnt
>
> vnconfig -u
>
> process opens the file and reads it
>
> Without fs cache invalidation, stale data can be returned.
>
> If there is explicit invalidation, it would be nice to say that
> precisely but I am not understanding that it is there. Reading vnd.c, I
> don't see any cache invalidation on detach. The only explicit
> invalidation I find is in setcred from VNDIOCSET.
>
> I guess that prevents the above, but doesn't prevent
>
> vnconfig
>
> mount
>
> read backing file
>
> write to mount
>
> unmount
>
> detach
>
> read backing file
>
> so maybe we need a vinvalbuf on detach?
>
> > I also think that when the unit is configured but not opened
> > (by device access or mounts) it is safe to access the file.
>
> As I read the code, reads are ok but will leave possibly stale data in
> the cache for post-close.
>
> >>> The data is written directly to the allocated blocks of the file.
> >>> So exclusively opening the backing file _or_ the vnd unit should
> >>> also be safe. But that's not much different from accessing any file
> >>> concurrently, which also leads to "corrupt", inconsistent backups.
> >
> >>That's a different kind of corrupt.
> >
> > Yes, but in the end it's the same, the "backup" isn't usuable.
>
> I am expecting that after deconfiguring, a read of the entire file is
> guaranteed consistent, but I think we're missing invalidate on close.
>
> > You cannot access the backing file to get a consistent state of the
> > data while a unit is in use. And that's independent of how vnd accesses
> > the bits.
>
> Agreed; that's more or less like using a backup program on database
> files while the database is running.
>
> > N.B. if you want to talk about dangers, think about fdiscard(). I
> > doubt that it is safe in the context of the vnd optimization.
>
> It seems clear that pretty much any file operations are unsafe while the
> vnd is configured. That seems like an entirely reasonable situation and
> if that's the rule, easy to document.
>
> I wrote a test script and it shows that stale reads happen. When I run
> this on UFS2 (netbsd-10), I find that all 4 files are all zero. When I
> run it on zfs (also netbsd-10), I find that 000 and 001 are all zero and
> 002 and 003 are the same. (I am guessing that zfs doesn't use the
> direct operations, or caches differently; here I haven't the slightest
> idea what is happening.)
>
> 10 minutes later, reading VND is still all zeros. With a new vnconfig,
> it still reads as all zeros.
>
>
> #!/bin/sh
>
> dd if=/dev/zero of=VND bs=1m count=1
> cat VND > VND.000
> vnconfig vnd0 VND
> cat VND > VND.001
> newfs /dev/rvnd0a
> cat VND > VND.002
> vnconfig -u vnd0
> cat VND > VND.003
At least this DTRT:
dd if=VND of=VND.004 iflag=direct
This reminds me of the way AIX handled O_DIRECT vs mmap, etc. From
https://www.ibm.com/docs/en/aix/7.2?topic=tuning-direct-io:
To avoid consistency issues, if there are multiple calls to open a file
and one or more of the calls did not specify O_DIRECT and another open
specified O_DIRECT, the file stays in the normal cached I/O mode.
Similarly, if the file is mapped into memory through the shmat() or
mmap() system calls, it stays in normal cached mode. If the last
conflicting, non-direct access is eliminated, then the file system will
move the file into direct I/O mode (either by using the close(),
munmap(), or shmdt() subroutines). Changing from normal mode to direct
I/O mode can be expensive because all modified pages in memory will have
to be flushed to disk at that point.
An elegant, albeit complex, solution.
--
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.
Home |
Main Index |
Thread Index |
Old Index