Re: Network very very slow... was iSCSI and jumbo frames

[BERTRAND Joël <joel.bertrand%systella.fr@localhost>]

RVP a écrit :
> On Sun, 7 Feb 2021, BERTRAND Joël wrote:
> 
>>>    c) Does squid do anything there?
>>
>>     Squid is used for proxy as my DSL connection is very poor.
> 
> Does it store its cache on /opt?

	No. Squid's cache is on /var (a local volume in Raid1).

>>> 2. Please run these commands as root and post output:
>>>    fstat -f /opt/some_existing_file
>>
>> legendre# fstat -f Full-0024
>> USER     CMD          PID   FD MOUNT       INUM MODE         SZ|DV R/W
>> root     fstat      22591   wd  /opt      114683904 drwxrwxr-x    6144 r
>> root     sh         28130   wd  /opt      114683904 drwxrwxr-x    6144 r
>> legendre#
>>
>>>    pstat -v
>> legendre# pstat -v
>> ...
>> *** MOUNT ffs /dev/dk5 on /opt (log,local)
>> ADDR             TYP VFLAG  USE HOLD TAG NPAGE FILEID IFLAG RDEV|SZ
>> ffffd45932b63428 dir     R    1    1   1     0      2     -     512
>> ffffd4577b884580 dir          3    1   1     0 114683904  -    6144
>> ffffd45873469818 dir          1    0   1     0 697238784  -     512
>> ffffd457232f36d0 reg          1    0   1     0 114683928  - 53687079457
>> legendre#
>>
> 
> Can you enable `noatime' and disable `log' when mounting /opt and
> then run a dd read test on any 5GB+ file there (make sure the dd
> cmd.  has `iflag=direct oflag=direct'). While the test is running,
> do:

	I don't undestand how dd returns "no such file or directory" when I try :

legendre# dd if=/dev/zero of=/opt/test iflag=direct oflag=direct
count=5000 bs=1m
dd: /opt/test: No such file or directory

	I cannot remount /opt without atime and log until next thursday. Bacula
full process is in progress.

>    fstat -f /opt

legendre# fstat -f /opt
USER     CMD          PID   FD MOUNT       INUM MODE         SZ|DV R/W
bacula-s bacula-sd   3861    7  /opt      114683956 -rw-r-----
50745719346 rw
legendre#

>    pstat -v | sed -n '/^\*\*\* MOUNT .*dk5/,/^\*\*\* MOUNT /p'


> I want to see what other processes show an undue interest in
> files being read/written in /opt. For the `pstat -v' command,
> keep an eye on those with a USE count >1. One of them will be
> the file being read/written.

legendre# pstat -v | sed -n '/^\*\*\* MOUNT .*dk5/,/^\*\*\* MOUNT /p'
*** MOUNT ffs /dev/dk5 on /opt (log,local)
ADDR             TYP VFLAG  USE HOLD TAG NPAGE FILEID IFLAG RDEV|SZ
ffffd45812906ac0 reg          1  319   1     0 114683952  - 53687079313
ffffd459b0fab970 dir     R    1    1   1     0      2     -     512
ffffd45628c05188 dir          1    1   1     0 114683904  -    6144
ffffd45845df8ab8 reg          1  401   1     0 114683953  - 53687079313
ffffd45737892970 reg          1  401   1     0 114683954  - 53687079692
ffffd459328b32e0 reg     O    3  381   1 2258960 114683956 - 50745719346
ffffd458711ed970 dir          1    0   1     0 697238784  -     512
legendre#

	I understand, but please note that even if bacula is stopped, a simple
dd doesn't exceed 10 MB/s on euclide.

>>> 3. These config. files (redact any sensitive data if needed).
>>>    euclide:
>>>    /etc/config/iscsi/sbin/iscsid.c
>>
>>     I suppose you want iscsi_trgt.conf
>>
> 
> It's the filename from the `ps -aux' output on euclide--the filename
> might have been truncated, but you should be able to locate it
> easily.
> 
>>     Please note than even with dd (and with bacula stopped, througput
>> doesn't exceed 10 MB/s between NetBSD initiator anq qNAP target).
>> legendre# cat bacula-dir.conf
>>
> 
> If bacula is stopped can you do the i3perf test between euclide
> and legendre with the server port on euclide set to the iscsi-target
> port? I want to make sure that the iscsi data isn't getting diverted
> and filtered at some end, slowing down the throughput.

	I have done these tests without any transaction on iSCSI daemon. I
don't remember if bacula was stopped, but I'm sure it was inactive.

>> legendre# cat bacula-sd.conf
>> Storage {                             # definition of myself
>>  Name = legendre.systella.fr-sd
>>  SDPort = 9103                  # Director's port
>>  WorkingDirectory = "/var/spool/bacula"
>>  Pid Directory = "/var/run"
>>  Plugin Directory = "/usr/pkg/lib"
>>  Maximum Concurrent Jobs = 20
>> }
>>
>> Device {
>>  Name = FileStorage1
>>  Media Type = File
>>  Archive Device = /opt/bacula
>>  LabelMedia = yes;                   # lets Bacula label unlabeled media
>>  Random Access = Yes;
>>  AutomaticMount = yes;               # when device opened, read it
>>  RemovableMedia = no;
>>  AlwaysOpen = no;
>>  Maximum Concurrent Jobs = 5
>> }
>>
> 
> As far as I can tell from your configs., bacula isn't unnecessarily
> accessing files in /opt. Just the standard stuff here.
> 
>>     I can open for you ssh access on legendre if you want.
>>
> 
> Even if you do, I'll just be poking around to get a feel for your
> setup as I don't have any clear idea of what to test yet :-(. The
> only thing I can think of now is doing a tcpdump while a dd test
> is going on.
> 
> Let me think over this for a day or two. I'll look over the emails
> of the past week and post up a summary of this problem and the things
> we've tried. Let's see if other folks have any better ideas.

	I suspect a latency issue somewhere in iscsid. I cannot test until next
thirsday, but I think we should compare two tcpdump outputs. The first
one between NetBSD initiator and euclide, the second one between Linux
initiator and euclide.

	Best regards,

	JKB