NetBSD-Bugs archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: kern/39242: NetBSD 4.0 will start busy-loop an hang on machines with more than 4 GB memory
The following reply was made to PR kern/39242; it has been noted by GNATS.
From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock%nagler-company.com@localhost>
To: gnats-bugs%NetBSD.org@localhost
Cc: Simon Burge <simonb%NetBSD.org@localhost>,
kern-bug-people%NetBSD.org@localhost
Subject: Re: kern/39242: NetBSD 4.0 will start busy-loop an hang on machines
with more than 4 GB memory
Date: Wed, 30 Jul 2008 20:13:32 +0200
Hi again,
I've seen responce to my last mail directly to Simon, but I continue
testing my system.
here the patch I've added to /usr7src/sys/amd64/amd64/cpu.c:
s012# rcsdiff -c -r1.1 c*
===================================================================
RCS file: RCS/cpu.c,v
retrieving revision 1.1
diff -c -r1.1 cpu.c
*** cpu.c 2008/07/29 07:28:03 1.1
--- cpu.c 2008/07/29 08:05:21
***************
*** 217,222 ****
--- 217,239 ----
tcolors /= cai->cai_associativity;
}
ncolors = max(ncolors, tcolors);
+ /*
+ * If the desired number of colors is not a power of
+ * two, it won't be good. Find the greatest power of
+ * two which is an even divisor of the number of colors,
+ * to preserve even coloring of pages.
+ */
+ if (ncolors & (ncolors - 1) ) {
+ int try, picked = 1;
+ for (try = 1; try < ncolors; try *= 2) {
+ if (ncolors % try == 0) picked = try;
+ }
+ if (picked == 1) {
+ panic("desired number of cache colors %d
is "
+ " > 1, but not even!", ncolors);
+ }
+ ncolors = picked;
+ }
}
/*
Just some minutes ago, I've got two new kernel crashes.
1. the kernel process [scsibus0] starts looping and sleeps sometimes in
pglalloc. After reinstalling the system with the patch above, there was
no DDB in the kernel, so I could not get any other information.
2. now I've DDB in the kernel and tried to reproduce the problem. But it
crashes prior reaching this state in pagedaemon ...
Some output from the console below:
uvm_fault(0xffffffff80628800, 0x0, 1) -> e
kernel: page fault trap, code=0
Stopped in pid 26.1 (pagedaemon) at netbsd:uvm_rb_insert+0x37:
movq 0
x40(%rax),%rax
db{0}> trace
uvm_rb_insert() at netbsd:uvm_rb_insert+0x37
uvm_map_enter() at netbsd:uvm_map_enter+0x290
uvm_map() at netbsd:uvm_map+0xfe
uvm_pagermapin() at netbsd:uvm_pagermapin+0x92
uvm_swap_io() at netbsd:uvm_swap_io+0x3c
swapcluster_flush() at netbsd:swapcluster_flush+0x55
uvm_pageout() at netbsd:uvm_pageout+0x42b
db{0}> show uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
2039030 VM pages: 1333964 active, 651446 inactive, 2248 wired, 5 free
pages 1674503 anon, 311641 file, 1580 exec
freemin=64, free-target=85, wired-max=679676
faults=3420439, traps=3975116, intrs=8916183, ctxswitch=14697719
softint=710475, syscalls=12629494, swapins=177, swapouts=210
fault counts:
noram=5, noanon=0, pgwait=0, pgrele=0
ok relocks(total)=1068(1069), anget(retrys)=73451(286), amapcopy=31017
neighbor anon/obj pg=65282/474220, gets(lock/unlock)=116248/783
cases: anon=51370, anoncow=21184, obj=96126, prcopy=20121, przero=63798
daemon and swap counts:
woke=32081, revs=4228, scans=1204885, obscans=1161315, anscans=2987
busy=0, freed=1164012, reactivate=128, deactivate=1857899
pageouts=75209, pending=1088875, nswget=78493
nswapdev=1, swpgavail=6291455
swpages=6291455, swpginuse=1164021, swpgonly=1085489, paging=66
db{0}>
This patch seems to enable the system to work with the 6MB cache of the
E3110 CPU, but the kernel is not realy stable at all.
Any idea? What should I try next?
By the way "vmstat -s | grep colo" reports 32 colors now.
The system was busy again when the crash happens. raidframe sync on one
SATA-raid and one SCSI-raid, transfered something arund 12 GB int /tmp
(tmpfs) so nearly 5 GB of the 24 GB swap was used.
The kernel crashes at the moment where I've tried to copy on oth the
archives from /tmp to a filesystem on the raid just syncing.
OK that may be a lot of work for the system, and it may get slow, but it
may not crash!
I've failed to get a core image this time - sorry.
continue does not work and the system freezes in sync from DDB ...
Bx the way: I've saved the 8 GB core-image from the crash below, but
that one has still 96 page color active - the patch was missing. I think
it makes no sence to look at it at all.
If nobody came to me in the next few day with a request for it, I will
remove it.
W. Stukenbrock
Wolfgang Stukenbrock wrote:
> Hi,
>
> I've took the diff's from x86/x86/cpu.c rev 1.32 and merged them into
> amd64/amd64/cpu.c - there is no x86/x86/cpu.c in 4.0 ...
>
> It looks like it will solve the problem.
> Thanks
>
> The system will no longer freeze after using something around 4 GB
> memory (of the 8 GB installed ...).
>
> But I've recognized, that under (very) heavy load the system will panic
> with "out of memory" in the pagedaemon.
> I have a 8GB core file here - anybody interested in analysing ???. I
> think it will not pass through any mailing system ... (the bzip2
> compresseed version is still larger 2 GB (compression still running ...)
>
> In DDB there was exactly on page stated to be free in "show uvmexp" -
> the output follows:
>
> db{1}> show uvmexp
> Current UVM status:
> pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
> 2039036 VM pages: 1119946 active, 547095 inactive, 4031 wired, 1 free
> pages 1328086 anon, 341555 file, 1764 exec
> freemin=64, free-target=85, wired-max=679678
> faults=13246446, traps=18516116, intrs=17316633, ctxswitch=64031864
> softint=6934242, syscalls=65761334, swapins=431, swapouts=1372
> fault counts:
> noram=2008, noanon=0, pgwait=0, pgrele=0
> ok relocks(total)=2409(2418), anget(retrys)=12931894(1119),
> amapcopy=365963
> neighbor anon/obj pg=702333/4863720, gets(lock/unlock)=1135392/1299
> cases: anon=6302100, anoncow=256103, obj=947341, prcopy=188007,
> przero=24372
> 47
> daemon and swap counts:
> woke=9819, revs=6729, scans=2176414, obscans=1733276, anscans=143716
> busy=0, freed=1876669, reactivate=14780, deactivate=3760043
> pageouts=9330, pending=134669, nswget=1111
> nswapdev=1, swpgavail=6291455
> swpages=6291455, swpginuse=143978, swpgonly=142546, paging=323
> db{1}> trace
> cpu_Debugger() at netbsd:cpu_Debugger+0x5
> panic() at netbsd:panic+0x1f5
> pmap_growkernel() at netbsd:pmap_growkernel+0x446
> uvm_map_prepare() at netbsd:uvm_map_prepare+0x371
> uvm_map() at netbsd:uvm_map+0xae
> uvm_km_alloc() at netbsd:uvm_km_alloc+0x73
> vmem_xalloc() at netbsd:vmem_xalloc+0x130
> vmem_alloc() at netbsd:vmem_alloc+0x86
> amap_alloc() at netbsd:amap_alloc+0xdb
> uvm_map_enter() at netbsd:uvm_map_enter+0x24b
> uvm_map() at netbsd:uvm_map+0xfe
> sys_obreak() at netbsd:sys_obreak+0x106
> syscall_plain() at netbsd:syscall_plain+0x1fc
> uvm_fault(0xffff8000a666f440, 0x6907000, 1) -> e
> kernel: page fault trap, code=0
> Faulted in DDB; continuing...
> db{1}> continue
> syncing disks... 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 giving up
>
> dumping to dev 18,1 offset 33560487
> dump 8189 8188 8187 8186 8185 8184 8183 8182 8181 8180 8179 8178 8177 .....
>
> I've started more than 1000 processes to see what will happen if "some"
> memory is needed for processes. (cat ... | dd obs=... | dd obs=... | ...
> >/dev/null)
> The systems reduces the amount of memory used by the file-cache to
> something about 1,2 GB of the 8 GB main memory - as expected.
> At the time of the crash there was something around 800 MB of 24 GB swap
> space used.
>
> I know that this is not related to the previous problem. Does it make
> sence to create another bug report for that? I'm not shure about it.
>
> W. Stukenbrock
>
> Simon Burge wrote:
>
>> Wolfgang Stukenbrock wrote:
>>
>>
>>> The following reply was made to PR kern/39242; it has been noted by
>>> GNATS.
>>>
>>> From: Wolfgang Stukenbrock
>>> <Wolfgang.Stukenbrock%nagler-company.com@localhost>
>>> To: gnats-bugs%NetBSD.org@localhost
>>> Cc: Subject: Re: kern/39242: NetBSD 4.0 will start busy-loop an hang
>>> on machines with more than 4 GB memory
>>> Date: Tue, 29 Jul 2008 09:07:09 +0200
>>>
>>> Hi,
>>>
>>> yes it is a E3110 CPU with 6MB cache.
>>> What files I need to catch from the current and integrate the changes
>>> into my 4.0-version of netbsd?
>>> You talk about the "cache detection stuff" AND the file named below.
>>>
>>> I think it would be a real great idea to bring this fix into the
>>> releases as soon as possible.
>>> This CPU is "very" cheep compared to the other one's (at least in
>>> germany) and therefore is very attractive for new systems.
>>>
>>
>> Does rev 1.32 of sys/arch/x86/x86/cpu.c apply cleanly to netbsd-4 ? I
>> don't recall which bits of which files moved around with x86 recently.
>> If so, that at least guarantees that a bogus number doesn't get passed
>> deeper into UVM, and that'll be enough for netbsd-4.
>>
>> Note also the problem was to do with "half memory used", not "4G of
>> memory used".
>>
>> Cheers,
>> Simon.
>>
>
>
Home |
Main Index |
Thread Index |
Old Index