Re: wip/glusterfs to pkgsrc filesystems/glusterfs

[Alexander Schreiber <als%thangorodrim.ch@localhost>]

On Fri, Mar 03, 2023 at 08:48:44PM -0500, Greg Troxel wrote:
> Alexander Schreiber <als%thangorodrim.ch@localhost> writes:
> 
> >> Is this new in glusterfs, that it doesn't work on i386, or just new that
> >> you figured it out and are documenting it?  For distributed/remote
> >> filesystems, I tend to feel its important for them to run everywhere
> >> since in my view that's a major part of suitability -- but I realize
> >> that if upstream is 64-bit only, it's next to impossible to tilt against
> >> that.
> >
> > Support for 32bit was effectively dropped by glusterfs a while ago,
> > there is https://github.com/gluster/glusterfs/issues/702 from 2019 that
> > starts proposing this. I know that glusterfs 8.2 (current pkgsrc) works
> > on NetBSD/i386, but somewhere between 8.2 and 10.3 (the one I'm working
> > on for pkgsrc), 32bit support effectively disappeared. You can still
> > _build_ it on 32bit, but it will die with SIGILL very early after
> > startup due to
> > https://github.com/gluster/glusterfs/issues/3911#issuecomment-1451711686
> > (found via trawling through backtraces and then through left behind
> > WRKOBJDIR contents due to the way userspace-rcu is integrated at
> > build time).
> >
> > There are some sporadic discussions about "hey, why does this crash on
> > 32bit?", but I do not expect anyone to actually put any work into
> > making it work on !64bit again, partially because that would involve
> > some ugly hackery (a bunch of internal things went from 32 to 64 bit
> > and they do atomic updates on those).
> >
> > Even so, there were reports (from older versions) that 32bit glusterfs
> > appears noticeably slower than 64bit.
> 
> I do wonder if it's possible to make userspace-rcu work on 32 bit
> processors via using libatomic64, even if it's slower.

No idea honestly, but even if, that presumably would require a reasonable
amount of familiarity with the glusterfs codebase. Which I definitely
don't have.

> > My workaround for 32bit machines is to mount the glusterfs filesystem
> > from a 64bit machine via sshfs from the 32bit machine. Not fast either,
> > but at last it works at all (yes, I still have some 32bit hardware in
> > active use with both NetBSD and Linux).
> 
> I have quite a few 32-bit machines running, 2 RPI3 (yes I know I could
> run 64 on that hardware), 2 desktops, and a notebook that does builds
> for the desktop.  To me, if I have to use ssh, then I might as well just
> use ssh.
> 
> So I guess we have two paths:
> 
>   Update and accept that anybody using gluster on 32-bit CPUs is going to
>   be broken.
> 
>   Version gluster and add gluster8 with the old code and make the new
>   code gluster10.

Having a single NetBSD/i386 machine where I occasionally use glusterfs,
I'm open to both approaches. Whoever ends up committing my updates
to pkgsrc itself is however the person to decide this.

> (In either case the new code package's DESCR needs a note that it is
> unusuable on 32-bit systems, so that people don't decide to rely on the
> marketing copy and later find out it doesn't work.)

They will find out the moment they try to build, as it is marked broken
on LP32PLATFORMS in the Makefile and e.g. attempting to build on i386
will tell you that (just verified). And if they look at the Makefile,
they'll find the comment describing why.

> Do we know if anyone else is using gluster on 32-bit CPUs?
> 
> (Personally, this causes me to prune gluster from my TODO list, as
> desupporting 32-bit CPUs does not seem reasonable.)

Frankly, I suspect that most installations using large datastores like
glusterfs have been running on nothing but 64bit for quite some time.

The only 32bit system with glusterfs I use is a NetBSD/i386 machine
and even there it is used only every few months to dump an updated
system image backup into glusterfs. And while there are occasional
complaints in the general direction of the glusterfs project about
"broken on 32bit", their volume seems to be rather to low to initiate
any serious action - which doesn't surprise me.

Kind regards,
           Alex.
-- 
"Opportunity is missed by most people because it is dressed in overalls and
 looks like work."                                      -- Thomas A. Edison