Re: continued zfs-related lockups

[Greg Troxel <gdt%lexort.com@localhost>]

The good news is that the problem is not subtle and I have been able to
reproduce the lockup.  And several times, if just barely provoked, the system
came back. At least once, it didn't come back.

I created a netbsd-current domU (pvhvm) with

  6G RAM 
  xbd0: 32G ffs2 root
  xbd1: 8G swap
  xbd2: 32G gpt with one big zfs partiition

  tank11: pool with just dk0 from xbd2

Not sure it matters, but the backing disks for the xbdN are zvol in zfs on
dom0, on a not particularly new Sandisk 1T SATA SSD.

I wrote a script:

  create 100 dirs with 100 files each
  sync
  sleep 10
  remove the files
  sync

Long ago I wrote a program "touchmem" to allocate a specific amount of memory,
writing into each page to force allocation.

I found that the removal process was slow, and if I ran touchmem 6000 (to
allocate 6000K) I would get on the console (this is an example where it
came back).


[ 2247.3254720] arc_reclaim_thread: negative free_memory -15888384
[ 2247.5554953] arc_reclaim_thread: negative free_memory -15908864
[ 2247.7955123] arc_reclaim_thread: negative free_memory -15814656
[ 2248.0355384] arc_reclaim_thread: negative free_memory -15839232
[ 2248.5255792] arc_reclaim_thread: negative free_memory -15847424
[ 2249.2455995] arc_reclaim_thread: negative free_memory -15814656
[ 2250.2756439] arc_reclaim_thread: negative free_memory -15835136
[ 2251.2756950] arc_reclaim_thread: negative free_memory -15847424
[ 2252.2757175] arc_reclaim_thread: negative free_memory -15847424
[ 2252.3257269] arc_reclaim_thread: negative free_memory -15867904
[ 2252.3357294] arc_reclaim_thread: negative free_memory -15876096
[ 2252.3457327] arc_reclaim_thread: negative free_memory -15958016
[ 2252.3657417] arc_reclaim_thread: negative free_memory -15867904
[ 2252.9757657] arc_reclaim_thread: negative free_memory -15835136
[ 2253.9858155] arc_reclaim_thread: negative free_memory -15851520
[ 2254.9958551] arc_reclaim_thread: negative free_memory -15847424
[ 2255.9958915] arc_reclaim_thread: negative free_memory -15839232
[ 2256.9959338] arc_reclaim_thread: negative free_memory -15831040
[ 2257.3259677] arc_reclaim_thread: negative free_memory -15859712
[ 2257.3459560] arc_reclaim_thread: negative free_memory -15831040
[ 2257.3759883] arc_reclaim_thread: negative free_memory -15855616
[ 2257.3859963] arc_reclaim_thread: negative free_memory -15831040
[ 2257.4260042] arc_reclaim_thread: negative free_memory -15839232
[ 2258.4360174] arc_reclaim_thread: negative free_memory -15831040
[ 2259.4360530] arc_reclaim_thread: negative free_memory -15851520
[ 2260.4360959] arc_reclaim_thread: negative free_memory -15831040
[ 2261.4361236] arc_reclaim_thread: negative free_memory -15835136
[ 2262.4361659] arc_reclaim_thread: negative free_memory -15843328
[ 2263.4362053] arc_reclaim_thread: negative free_memory -15831040
[ 2264.4362512] arc_reclaim_thread: negative free_memory -15831040
[ 2265.4362893] arc_reclaim_thread: negative free_memory -15839232
[ 2266.4363080] arc_reclaim_thread: negative free_memory -15855616
[ 2267.4363426] arc_reclaim_thread: negative free_memory -15831040
[ 2268.4463773] arc_reclaim_thread: negative free_memory -15843328
[ 2269.4464351] arc_reclaim_thread: negative free_memory -15847424
[ 2270.4464547] arc_reclaim_thread: negative free_memory -15831040
[ 2271.4465084] arc_reclaim_thread: negative free_memory -15843328
[ 2272.4465497] arc_reclaim_thread: negative free_memory -15839232
[ 2273.4565874] arc_reclaim_thread: negative free_memory -15831040
[ 2274.4566118] arc_reclaim_thread: negative free_memory -15839232
[ 2275.4566479] arc_reclaim_thread: negative free_memory -15843328
[ 2276.4566869] arc_reclaim_thread: negative free_memory -15835136
[ 2277.4567207] arc_reclaim_thread: negative free_memory -15839232
[ 2278.4567630] arc_reclaim_thread: negative free_memory -15831040
[ 2279.4568163] arc_reclaim_thread: negative free_memory -15839232
[ 2280.4568310] arc_reclaim_thread: negative free_memory -15835136
[ 2281.4668732] arc_reclaim_thread: negative free_memory -15843328
[ 2282.4769259] arc_reclaim_thread: negative free_memory -15851520
[ 2283.5369614] arc_reclaim_thread: negative free_memory -15831040
[ 2284.5470001] arc_reclaim_thread: negative free_memory -15835136
[ 2285.5570314] arc_reclaim_thread: negative free_memory -15835136
[ 2286.5671034] arc_reclaim_thread: negative free_memory -7159808
[ 2287.5671318] arc_reclaim_thread: negative free_memory -851968


So the needed combination seems to be
  - lots of removals (may be just metadata changes)
  - memory pressure


I'll put my test programs up someplace and look at arc_reclaim_thread.

I wonder if others who have problems also see this kernel message.