Re: Performance degradation over time on VAX...

[Ken Wellsch <kcwellsch%outlook.com@localhost>]

Thank you Josh for the particulars on kernel profiling.  I installed
11.0RC6 on a PC and tried the instructions and they worked like a charm.
I condensed it down to this, for example:

cd /tmp && kgmon -r && kgmon -b \
        && dd if=/dev/zero of=/dev/null bs=4096 count=500224
        && kgmon -h && kgmon -p
        && gprof /netbsd gmon.out > gprof.zero && rm gmon.out

I then installed 5.2.3 then 6.0.1 on a SUN v120 (sparc64) to try
the process there.  Also with a bit of wishful thinking I might
see a performance hit there too, but not the case.

============================================
---- dd if=/dev/zero  of=/dev/null bs=4096 count=2000000 ----
============================================

5.2.3   8192.00 MB in  66.695 secs (122.83 MB/s)  offset 0.072365 sec
6.0.1   8192.00 MB in  55.738 secs (146.97 MB/s)  offset 0.027679 sec

============================================
---- dd if=/dev/rsd0c of=/dev/null bs=4096 count=500224 ----
============================================

5.2.3   2048.92 MB in 135.932 secs (15.07 MB/s) offset 0.072443 sec
6.0.1   2048.92 MB in 133.915 secs (15.30 MB/s) offset 0.045489 sec

Anyway, the profiling worked fine on both releases on the v120.

I then installed 11.0RC6 on simH to try profiling there.  After over 3
hours depending and building a profile kernel, I found that booting that
kernel hung just before going multiuser. Maybe I should have just
waited longer, but I think I gave it at least 5 minutes with no further
change.

So I decided to try diffing the sys source between 5.2.3 and 6.0.1
(which I'd pulled from the v120 testing).  Plenty of changes as one
might expect.  I did see some assembler versions of the mem functions
and the addition of memset by Matt.  That got me thinking about the
high cost of copying data around.

I decided to look and see, how many memcpy() calls appear in the diff
output between 5.2.3 and 6.0.1.  Just out of curiosity. I am not sure
if this means what I think it means, but on the 5.2.3 side I count 139
memcpy() calls.  On the 6.0.1 side I count 1044 memcpy() calls.

Trying another way, as a sort of sanity check, let me just do a naive
attempt to count memcpy() calls.

5.2.3-src $ find sys -type f -print | xargs grep -w memcpy | fgrep 'memcpy(' | wc -l
    3493


6.0.1-src $ find sys -type f -print | xargs grep -w memcpy | fgrep 'memcpy(' | wc -l
    4562

So the number of new memcpy calls has gone up significantly (over 20%).

I will have to try and see if I can see any new memcpy calls in some critical
code path that might be affecting the VAX port.