Re: Bind ending up in Parked state.

[Søren P. Skou <sps%t-rex.dk@localhost>]

Thus spake Christos Zoulas, on Tue, Mar 10, 2015 at 04:08:34PM +0000:
> >I hadn't thought of that at all.
> >
> >I tried -current, that broke the machine rather violently :) ended up in a
> >db{6}: prompt without any USB support. After a bit of fiddling around, I
> >realized that starting over would probably be the fastest way of going. It
> Would be nice to know the trace of that and the steps to reproduce :-)

The steps to reproduce the db{6} - or what I did was to fetch amd64 binaries 
from http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201503060730Z/amd64/binary/sets/ - then run

mkdir /I
cd /I
echo â??mget *gzâ?? | ftp -Va ftp://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201503060730Z/amd64/binary/sets/
cd  /
progress -z -f /I/modules.tgz pax -rpe
progress -z -f /I/kern-GENERIC.tgz pax -rpe
reboot

After a while I ventured into the datacenter, even though it is a HP-G5 it 
shouldn't take this long to get back to life. Here I noticed it was hanging 
with a db{6} - The log leading up to this was:

panic: cnopen: no console device
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip ffffffff8028994d cs 8 rflags 2467 cr2 7f7ff6c08be8 ilevel 0 rsp fffffe811db66ad8
curlwp 0xfffffe843b1538a0 pid 2.1 lowest kstack 0xfffffe811db632c0
Stopped in pid 2.1 (init) at     netbsd:breakpoint +0x5: leave
db{6}> 


The server had an USB keyboard attached, but it didn't seem to respond. 
Fortunately I had an old PS2 keyboard that I connected up and entered "c", 
this resulted in the server doing a dump from 632 to 0, and a reboot. Here it
ended up in the same spot, with the same error.

Tired and a slight feeling of frustration, boot on my NetBSD install USB stick 
to get a 6.1.5 kernel and modules back. Alas, when the server was rebooted into
the 6.1.5 kernel, it just kept rebooting right after the boot menu. I could 
see the first 2 seconds of swirly lines after I pressed enter on the menu, then
it hung up on me and rebooted.

I ended up reinstalling the machine, and getting it back to a 6.1.5. This is 
where the machine is currently. 

> 
> >is now happily chomping away, with a lot less load on the system overall.
> >Without --disable-threads it would ever so often spike all 8 Cores, and it
> >wasn't uncommon for it to have a load >2.0 - now it is sitting comfortably
> >around 0.32. And it doesn't seem to have any impact on the system.
> 
> Still I would like to know what is taking all this time... ktrace it
> and then kdump -R to display relative timestamps.
> 
> >I still see
> >
> > 3417 named     43    0   713M  503M parked/5   3:34 19.82% 19.82% named
> 
> Hit 't' to switch to the thread view to get more details about what's going
> on for each individual thread (for the threaded named).
Never get too excited about things too soon it seems ;-) The Named just broke 
down again. I've started it up and is currently running ktrace against it.

Currently a top (with t) shows the following regarding named:


 127 threads: 35 idle, 85 sleeping, 7 on CPU
CPU0 states:  4.6% user,  0.0% nice, 14.6% system,  5.6% interrupt, 75.2% idle
CPU1 states:  5.6% user,  0.0% nice, 19.0% system,  0.0% interrupt, 75.4% idle
CPU2 states:  5.6% user,  0.0% nice,  5.4% system,  0.0% interrupt, 89.0% idle
CPU3 states:  4.2% user,  0.0% nice, 10.4% system,  0.0% interrupt, 85.4% idle
CPU4 states:  7.6% user,  0.0% nice, 12.6% system,  0.0% interrupt, 79.8% idle
CPU5 states:  3.4% user,  0.0% nice, 10.0% system,  0.2% interrupt, 86.4% idle
CPU6 states:  4.0% user,  0.0% nice,  5.2% system,  0.2% interrupt, 90.6% idle
CPU7 states:  5.0% user,  0.0% nice,  8.4% system,  0.0% interrupt, 86.6% idle
Memory: 4494M Act, 6052K Wired, 24M Exec, 3905M File, 11G Free
Swap: 16G Total, 16G Free

  PID   LID USERNAME PRI STATE      TIME   WCPU    CPU NAME      COMMAND
23874    11 named     34 CPU/6      0:47 21.00% 21.00% -         named
23874     6 named     39 parked/1   0:28 17.09% 17.09% -         named
23874     5 named     41 parked/2   0:27 14.55% 14.55% -         named
23874     4 named     42 parked/3   0:27 12.94% 12.94% -         named
23874     8 named     40 parked/4   0:25 12.89% 12.89% -         named
23874     9 named     39 parked/6   0:25 12.79% 12.79% -         named
23874     2 named     40 parked/7   0:26 12.70% 12.70% -         named
23874     7 named     39 parked/5   0:26 11.43% 11.43% -         named
23874     3 named     39 parked/5   0:27  9.67%  9.67% -         named
23874     1 named     85 sigwai/7   0:00  0.00%  0.00% -         named
23874    10 named     43 parked/1   0:00  0.00%  0.00% -         named


anycast-dns-01# ldd /usr/pkg/sbin/named
/usr/pkg/sbin/named:
        -llwres.141 => /usr/pkg/lib/liblwres.so.141
        -ljson-c.2 => /usr/pkg/lib/libjson-c.so.2
        -lc.12 => /usr/lib/libc.so.12
        -lpthread.1 => /usr/lib/libpthread.so.1
        -lgcc_s.1 => /usr/lib/libgcc_s.so.1
        -lxml2.2 => /usr/pkg/lib/libxml2.so.2
        -lz.1 => /usr/lib/libz.so.1
        -llzma.1 => /usr/lib/liblzma.so.1
        -lm.0 => /usr/lib/libm.so.0
        -lbind9.140 => /usr/pkg/lib/libbind9.so.140
        -lisccfg.140 => /usr/pkg/lib/libisccfg.so.140
        -ldns.146 => /usr/pkg/lib/libdns.so.146
        -lisc.142 => /usr/pkg/lib/libisc.so.142
        -lgssapi.10 => /usr/lib/libgssapi.so.10
        -lkrb5.26 => /usr/lib/libkrb5.so.26
        -lhx509.5 => /usr/lib/libhx509.so.5
        -lasn1.9 => /usr/lib/libasn1.so.9
        -lcom_err.7 => /usr/lib/libcom_err.so.7
        -lcrypt.1 => /usr/lib/libcrypt.so.1
        -lcrypto.8 => /usr/lib/libcrypto.so.8
        -lroken.19 => /usr/lib/libroken.so.19
        -lutil.7 => /usr/lib/libutil.so.7
        -lwind.0 => /usr/lib/libwind.so.0
        -lheimbase.1 => /usr/lib/libheimbase.so.1
        -lheimntlm.4 => /usr/lib/libheimntlm.so.4
        -lcrypto.1.0.0 => /usr/pkg/lib/libcrypto.so.1.0.0
        -lisccc.140 => /usr/pkg/lib/libisccc.so.140

even with --disable-threads it still links into phtreads it seems?

Best regards
Søren P. Skou


-- 
GPG Key: 8E58ACB3