Re: read/write slow on nvme drive - Supermicro X10DRL-CT

[mlelstv%serpens.de@localhost (Michael van Elst)]

derrick.lobo%givex.com@localhost (Derrick Lobo) writes:

>I have not had issues with the (sata) SSD either.... the nvme is really
>slow for me its 6 ties slower doing basic operation like untaring a file..
>and im not sure if theres any config needed to make this work.  


Here is some data from a cheap Samsung NVME card (about EUR100,- / 500GB).

It's formatted as simple ffs, no logging with bsize = 32k, fsize = 4k.

/dev/dk0 on /mnt type ffs (local)

Reading from raw device gives about 2.3GB/s.

% dd if=/dev/rdk0 bs=1024k count=10240 of=/dev/null
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 4.489 secs (2391939906 bytes/sec)

Writing a large file to the filesystem is about 1.2GB/s.

% dd if=/dev/zero bs=1024k count=10240 of=testfile
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 8.268 secs (1298671775 bytes/sec)

Unpacking a tarfile with little files reads 23MB/s on average
but the write load is of course much higher.

% ls -l src.tar
-rw-r--r--  1 mlelstv  staff  1582223360 May 15 00:31 src.tar
% time tar xf src.tar
1.181u 24.089s 1:07.88 37.2%    0+0k 90+1578062io 0pf+0w

While tar is running, iostat reveals it's writing at 381MB/s.

device  read KB/t    r/s   time     MB/s write KB/t    w/s   time     MB/s
ld0         64.00    159   0.60     9.93      14.28  27350   0.60   381.29
dk0         64.00    159   0.63     9.93      14.28  27350   0.63   381.29

The difference between raw I/O and untaring a file is filesystem
and virtual memory overhead.

Taring it up again is of course much faster (read from cache, partially
async write), but the destination is written at about 210MB/s.

% time tar cf src2.tar src
0.529u 6.319s 0:07.11 96.0%     0+0k 0+40io 0pf+0w

From a freshly mounted filesystem you see the real I/O rates:

% time tar cf src2.tar src
0.525u 7.476s 0:24.64 32.4%     0+0k 63369+49io 0pf+0w

device  read KB/t    r/s   time     MB/s write KB/t    w/s   time     MB/s
ld0          6.34  17370   0.70   107.63      63.91   1008   0.70    62.90
dk0          6.34  17370   0.73   107.63      63.91   1008   0.73    62.90

Write rate is down to 62MB/s, even the total I/O is less than half of
the in-cache case. This difference however is no software overhead,
the CPU is sleeping




Your system should show similar performance. However, your Intel NVME
card is a bit older. The only guess I have is that it reaches its speed
when you use multiple queues in parallel. But our driver currently only
handles a single queue.


-- 
-- 
                                Michael van Elst
Internet: mlelstv%serpens.de@localhost
                                "A potential Snark may lurk in every tree."