XEN 4.11 and 9.99.48 DOMU performance

[Frank Kardel <kardel%kardel.name@localhost>]

This is my first XEN setup so I may have misconfigured something:

I have a 4G DOM0 on a 512G System with a EPYC 7302P 16-Core Processor.

On that I configured a 400G DOMU with 12 vcpus. like this:

name = "system"
kernel = "/netbsd-XEN3_DOMU.gz"
memory = 400000
cpus="all"
vcpus=4
maxvcpus=12
vif = [ 'mac=aa:00:00:d1:00:01,bridge=bridge0',
        'mac=aa:00:00:d1:00:02,bridge=bridge1' ]
disk = [ 'file:/data0/xen-roots/root-Alpine-system.img,0x0,w',
         'phy:/dev/wedges/data1,0x1,w' ]

On that I run postgresql 11 attempting to load a 1TB database.

Usually this workload keeps a machine continually busy cpu/io-wise.

I was expecting that I/O via the xen backend would be the bottleneck.

Instead DOM0 is only seldom busy for IO. DOMU is crawling along sleeping

at all sorts of places:

load averages:  0.01,  0.11,  0.14;               up 4+00:44:49 16:04:56
68 processes: 66 sleeping, 2 on CPU
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% 
idle
Memory: 314G Act, 26M Inact, 16M Wired, 24M Exec, 312G File, 60G Free
Swap:

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME WCPU    CPU COMMAND
    0 root       0    0     0K  803M CPU/11    12:25 0.00%  0.00% [system]
19870 pgsql    117    0    78M 4056K biowai/4   3:32 0.00%  0.00% postgres
23443 pgsql    117    0    51G 1067M tstile/2   2:05 0.00%  0.00% postgres
17623 kardel    85    0    31M 4584K ttyraw/2   0:50 0.00%  0.00% systat
18581 pgsql     85    0    51G  104M biowai/0   0:36 0.00%  0.00% postgres
19164 pgsql     85    0    51G  106M uvn_fp/1   0:35 0.00%  0.00% postgres
 9695 pgsql     85    0    51G  105M biowai/8   0:31 0.00%  0.00% postgres
23770 pgsql     84    0    51G  103M uvn_fp/4   0:31 0.00%  0.00% postgres
24557 pgsql    117    0    51G 6440K tstile/0   0:30 0.00%  0.00% postgres
  135 pgsql     85    0    51G  113M uvn_fp/4   0:28 0.00%  0.00% postgres
29232 pgsql     85    0    51G  110M uvn_fp/2   0:28 0.00%  0.00% postgres
18124 pgsql     85    0    51G  113M uvn_fp/6   0:27 0.00%  0.00% postgres
27138 pgsql     85    0    51G  106M uvn_fp/0   0:26 0.00%  0.00% postgres
17531 pgsql     85    0    51G  113M uvn_fp/7   0:24 0.00%  0.00% postgres
 3800 pgsql    117    0    51G  104M tstile/0   0:21 0.00%  0.00% postgres
25673 pgsql    117    0    51G  105M tstile/6   0:20 0.00%  0.00% postgres
24550 kardel    85    0    27M 2616K select/0   0:15 0.00%  0.00% screen
22941 pgsql     85    0    51G  104M bioloc/0   0:14 0.00%  0.00% postgres
  345 root      85    0    30M   16M pause/2    0:11 0.00%  0.00% ntpd
24483 pgsql     85    0    51G  405M poll/1     0:06 0.00%  0.00% postgres
19694 pgsql     85    0    51G   20M poll/4     0:05 0.00%  0.00% postgres
24189 kardel    85    0    87M 5260K select/0   0:04 0.00%  0.00% sshd
 1068 root      85    0    62M 3388K poll/10    0:04 0.00%  0.00% 
pg_restore

xl list show the DOMU mostly being blocked most of the time.

Also the buffercache seems small:

      15282 metadata buffers using                233730 kBytes of 
memory ( 0%).
   81970980 pages for cached file data using   327883920 kBytes of 
memory (83%).
       6043 pages for executables using            24172 kBytes of 
memory ( 0%).
     366880 pages for anon (non-file) data       1467520 kBytes of 
memory ( 0%).
   15752203 free pages                          63008812 kBytes of 
memory (16%).

What needs to be changed to avoid these surprising stalls and get the 
DOMU moving.

Do I need some XEN parameter tuning?

Frank