Re: Very slow file write/create operations

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Tue, 7 Apr 2026 17:19:26 +0200
    From:        Frank Wille <frank%phoenix.owl.de@localhost>
    Message-ID:  <ecccf15e-164d-4f62-9a12-ed468be0c839%phoenix.owl.de@localhost>


  | 2. Shouldn't the user- and sys-times more or less add into "real"? Where
  | did the system waste all that time?

Waiting for something to finish.   With your tests, I'd guess that is
synchronous writes to the ciss raid array - which are (by default) used
for all metadata operations - file creation (adding entries to a directory)
being the likely issue here (/etc is lots of small files).

  | On the other hand, creating a single 100MB file is quick as expected:
  | # dd if=/dev/zero of=/root/dummy bs=1m count=100
  | 100+0 records in
  | 100+0 records out
  | 104857600 bytes transferred in 0.116 secs (903944827 bytes/sec)

That's a completely meaningless test, writing 100MB on a 128GB RAM
system is doing (almost) no more than clearing 100MB of ram.   When that
test finishes, almost nothing will have actually been written, it is
all just sitting in memory, waiting.   There will have been one sync
operation (creating the output file) and maybe one or two more, allocating
space for the output on the filesystem (which involves cylinder group updates,
or probably would) but so few of those that you'd barely notice them,
even if slow.

  | Also reading is fast and writing to tmpfs:
  | # time cp -Rp /etc /tmp/
  |          0.03 real         0.00 user         0.03 sys

That might be doing no more than reading the files from /etc that your
previous attempt loaded into ram in the process of writing them to /root
so you might not even be measuring the filesystem read speed there - though
there are essentially no meta-data updates on file reads, so if it were
reading the filesystem it should be fast.

/tmp is just RAM, so the writes there are just copying one ram block to
another - but that will be most of the time (and why it is all sys time).

  | # time rm -rf /tmp/etc
  |          0.02 real         0.00 user         0.02 sys

Again, just manipulating ram, no i/o involved at all.

  | What is wrong here? Does anybody have an idea what to check?

If you can make a new filesystem on the raid (it should not need to be
very big, use twice the size of du /etc (get the units correct) as a rough
estimate), and mount that one -o async and repeat the test copying /etc
onto that (new) filesystem.   If it is async operations that are the
issue, that should go fairly fast - but include the time needed to umount
the filesystem (which requires flushing the cached writeback blocks,
which would be almost all that will be created by the cp -R) in the
overall results (so all the I/O is known to have happened).

If you can determine that is the issue, then you have something to investigate
further, with the driver, raid implementation, controller, and drives.

kre

ps: having lots of RAM is wonderful in general, but it makes ad-hoc benchmarks
very difficult to design to give any kind of meaningful results.