Subject: About softdep and pools.
To: None <tech-kern@netbsd.org>
From: Frank van der Linden <frank@wins.uva.nl>
List: tech-kern
Date: 11/19/1999 16:50:44
Some people who were using the FFS softdep code, hit its memory limits.
The problem observed is, that under bursts of heavy filesystem usage,
quite a number of unresolved dependencies exist, taking up a lot of
memory.

This problem was noticed before, and Kirk added some code that attempts
to deal with this. It checks if the number of allocated dependency
structures of a few types hits some maximum. If it does, some flags
are set that will cause the FFS sync code (called from the trickle
syncer) to clean up soft dependencies sooner. Furthermore, the
syncer is scheduled to run immediately, and will not sleep as long.

This method was chosen, because attempting to synchronously flush
the dependencies, while trying to allocate others, can result
in locking problems.

Now, I would like to poolify the code, it makes sense to do this
for all the fixed-size structures that are used. However, I need
the following:

	1) A mechanism to signal that some limit is reached.
	   Ok, fine, I can use pool_sethardlimit.
	2) I may not just sleep when reaching the limit, because
	   I need to do the trick that speeds up the syncer first.
	   Ok, so I use LIMITFAIL|WAITOK, will speed up the syncer,
	   and then try again with just WAITOK.
	3) When called from the syncer itself, the softdep sync code may
	   may in fact allocate additional dependencies, albeit temporarily.
	   When its done, the total number of allocated dependencies will
	   have dropped, but in the meantime I may need to allocate
	   a few more elements, and can not fail on the hard limit.
	   It looks like I could use pool_prime to extend the pool when
	   the hard limit is reached. However, there seems to be no
	   call to shrink it again later.

What to do (short of rewriting the softdep code)?

- Frank