Port-arm archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
High system times on Scaleway Cloud server
I recently setup 8-core cloud server in Scaleway's Amsterdam DC
and I noticed that system time on CPU0 stayed at 50-80% when doing
an IO activity like 'cvs co src' or rsync/scp. I initially thought
that a full disk encryption was a bit of a stress but the problem
appears to be more than a suboptimal code in cgd.
I often observe 80-90% across all cores and I have no idea where
this time is spent. The output below shows a load on the system
running ./build.sh -Uj8 and building a package with 8 jobs. It also
runs several tor instances but they open only about 70 tcp sockets
and generate 100KB/sec traffic.
Any ideas where should I look to understand the problem a bit more?
# uname -a
NetBSD nebula 8.99.28 NetBSD 8.99.28 (GENERIC64) #1: Sat Dec 22 14:49:31 GMT 2018 alnsn@nebeda:/home/alnsn/netbsd-current/src/sys/arch/evbarm/compile/obj/GENERIC64 evbarm
# top
load averages: 11.1, 11.2, 9.70; up 1+11:36:22
114 processes: 4 runnable, 102 sleeping, 1 zombie, 7 on CPU
CPU0 states: 12.9% user, 0.0% nice, 87.1% system, 0.0% interrupt, 0.0% idle
CPU1 states: 4.8% user, 0.0% nice, 93.5% system, 0.0% interrupt, 1.6% idle
CPU2 states: 3.2% user, 0.0% nice, 96.8% system, 0.0% interrupt, 0.0% idle
CPU3 states: 6.5% user, 0.0% nice, 93.5% system, 0.0% interrupt, 0.0% idle
CPU4 states: 6.5% user, 0.0% nice, 93.5% system, 0.0% interrupt, 0.0% idle
CPU5 states: 9.7% user, 0.0% nice, 90.3% system, 0.0% interrupt, 0.0% idle
CPU6 states: 12.9% user, 0.0% nice, 87.1% system, 0.0% interrupt, 0.0% idle
CPU7 states: 8.1% user, 0.0% nice, 91.9% system, 0.0% interrupt, 0.0% idle
Memory: 3494M Act, 1975M Inact, 7572K Wired, 35M Exec, 5078M File, 700M Free
Swap: 9361M Total, 9361M Free
Seconds to delay:
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
0 root 125 0 0K 358M vdrain/0 66:39 3.66% 3.66% [system]
4089 alnsn 77 0 6240K 2072K wait/2 0:00 3.14% 2.44% sh
3351 alnsn 76 0 6552K 1960K pipe_r/7 0:00 2.53% 1.81% sh
14726 alnsn 76 0 12M 1956K pipe_r/5 0:00 2.55% 1.66% sh
26512 alnsn 77 0 7764K 1960K pipe_r/6 0:00 2.27% 1.51% sh
23337 alnsn 42 0 12M 9360K RUN/0 9:34 0.49% 0.49% tmux
29770 alnsn 77 0 5760K 1964K pipe_r/2 0:00 0.68% 0.49% sh
1521 alnsn 77 0 5632K 1960K pipe_r/7 0:00 0.32% 0.20% sh
19782 torrelay 26 0 307M 200M parked/7 16:16 0.00% 0.00% tor
9484 torpubli 85 0 65M 53M kqueue/2 3:25 0.00% 0.00% tor
10604 tor 85 0 55M 45M kqueue/7 2:13 0.00% 0.00% tor
22583 tordev 85 0 49M 39M kqueue/0 1:18 0.00% 0.00% tor-dev
20956 alnsn 85 0 7508K 2296K select/1 0:19 0.00% 0.00% nbmake
29125 root 85 0 8148K 2128K kqueue/4 0:12 0.00% 0.00% syslogd
28197 alnsn 85 0 9224K 2288K select/0 0:11 0.00% 0.00% nbmake
17967 alnsn 85 0 8996K 2300K select/7 0:10 0.00% 0.00% nbmake
1 root 85 0 7844K 1440K wait/2 0:09 0.00% 0.00% init
11126 ntpd 85 0 8976K 7544K pause/1 0:07 0.00% 0.00% ntpd
561 root 85 0 12M 2488K kqueue/4 0:02 0.00% 0.00% master
435 root 83 0 20M 2940K select/1 0:02 0.00% 0.00% sshd
20866 alnsn 80 0 4476K 1756K wait/0 0:02 0.00% 0.00% sh
22586 alnsn 79 0 13M 6052K wait/0 0:02 0.00% 0.00% make
16042 root 84 0 4128K 1480K pause/0 0:01 0.00% 0.00% ksh
17823 alnsn 80 0 8504K 6136K wait/0 0:01 0.00% 0.00% make
3937 alnsn 79 0 12M 6276K wait/0 0:01 0.00% 0.00% make
11041 alnsn 78 0 14M 6712K wait/7 0:01 0.00% 0.00% make
20391 alnsn 77 0 15M 5440K wait/0 0:01 0.00% 0.00% make
24281 root 39 0 11M 1960K CPU/0 0:00 0.00% 0.00% top
13005 alnsn 26 0 5760K 816K CPU/1 0:00 0.00% 0.00% sh
14523 alnsn 25 0 27M 12M CPU/6 0:00 0.00% 0.00% cc1plus
15456 alnsn 25 0 9096K 1076K CPU/3 0:00 0.00% 0.00% sed
3015 alnsn 25 0 10M 1028K CPU/5 0:00 0.00% 0.00% sh
21444 alnsn 25 0 7480K 636K CPU/4 0:00 0.00% 0.00% sed
2953 alnsn 25 0 5268K 336K CPU/7 0:00 0.00% 0.00% sed
970 alnsn 76 0 8308K 884K RUN/0 0:00 0.00% 0.00% sh
21907 alnsn 43 0 23M 4876K RUN/0 0:00 0.00% 0.00% sshd
22377 alnsn 25 0 9252K 376K RUN/4 0:00 0.00% 0.00% sed
7690 root 86 -2 11M 3460K wait/1 0:00 0.00% 0.00% su
602 postfix 85 0 12M 4316K kqueue/4 0:00 0.00% 0.00% qmgr
19194 alnsn 85 0 13M 4292K select/6 0:00 0.00% 0.00% nbmake
19472 postfix 85 0 14M 4164K kqueue/4 0:00 0.00% 0.00% pickup
18761 alnsn 85 0 7496K 2588K select/0 0:00 0.00% 0.00% ssh-agent
799 alnsn 85 0 7496K 2416K select/4 0:00 0.00% 0.00% ssh-agent
27640 alnsn 85 0 7456K 2300K select/0 0:00 0.00% 0.00% nbmake
185 root 85 0 4316K 1728K kqueue/5 0:00 0.00% 0.00% dhcpcd
5554 alnsn 85 0 4128K 1452K pause/2 0:00 0.00% 0.00% ksh
6187 alnsn 85 0 4128K 1448K pause/0 0:00 0.00% 0.00% ksh
472 root 85 0 4200K 1368K nanosl/1 0:00 0.00% 0.00% cron
580 root 85 0 4204K 1324K ttyraw/7 0:00 0.00% 0.00% getty
624 root 85 0 4308K 1152K kqueue/0 0:00 0.00% 0.00% inetd
17903 alnsn 82 0 11M 5824K wait/0 0:00 0.00% 0.00% make
7093 alnsn 82 0 11M 1500K wait/0 0:00 0.00% 0.00% sh
18846 root 81 0 20M 6708K select/0 0:00 0.00% 0.00% sshd
22949 alnsn 81 0 15M 5580K wait/1 0:00 0.00% 0.00% make
29364 alnsn 81 0 11M 1500K wait/1 0:00 0.00% 0.00% sh
26513 alnsn 80 0 9560K 5560K wait/5 0:00 0.00% 0.00% make
4249 alnsn 80 0 6800K 1504K wait/0 0:00 0.00% 0.00% sh
924 alnsn 80 0 11M 1104K wait/7 0:00 0.00% 0.00% sh
18338 alnsn 79 0 4908K 1884K wait/5 0:00 0.00% 0.00% sh
14873 alnsn 79 0 6696K 1756K wait/0 0:00 0.00% 0.00% gmake
22045 alnsn 79 0 12M 1744K wait/0 0:00 0.00% 0.00% gmake
19910 alnsn 79 0 6712K 1504K wait/0 0:00 0.00% 0.00% sh
(taken ~1 hour later)
# pstat -T
624/98304 files
366840 vnodes
0/9585162 blocks swap space
# df -h|grep cgd
/dev/cgd0a 1.5G 75M 1.4G 5% /
/dev/cgd0d 2.0G 562M 1.3G 28% /usr
/dev/cgd0f 1.0G 255M 715M 26% /var
/dev/cgd0g 4.0G 1.0M 3.8G 0% /var/log
/dev/cgd0e 4.0G 1.2G 2.6G 31% /usr/pkg
/dev/cgd0h 4.0G 10M 3.8G 0% /var/tmp
/dev/cgd0i 15G 1.2M 14G 0% /var/crash
/dev/cgd0j 503M 17M 461M 3% /var/db/pkg
/dev/cgd0k 4.0G 2.0K 3.8G 0% /var/db/pkgin/cache
/dev/cgd1e 139G 24G 108G 17% /home
--
Alex
Home |
Main Index |
Thread Index |
Old Index