Hello,
while working on vioscsi(4) improvements, I had quite tedious problems with driver failing to allocate the dma map in bus_dmamap_create().
Symptoms were that the bus_dmamap_create()/bus_dmamem_alloc() call was very often, but not always, failing when driver was loaded via module. For example, when the module was loaded immediatelly after VM reboot, the dmamap/dmamem calls would succeed very often, but also not always. If it failed, it would consistently fail with more module unload/load until another reboot.
When the driver was compiled into kernel and hence attached during boot, the bus dmamem/dmamap never failed for me. So I mostly ended up rebooting kernels.
The driver allocates sizeable, but not excessive amounts of memory - mostly working with values like MAXPHYS / PAGE_SIZE * (queue size) for dmamaps/dma memory, where queue size is 128 or so.
The memory allocation failing was not dependant on amount of available physical memory - it failed regardless of how much memory I allocated to the virtual host, and regardless if I used i386 or x86_64 virtual machien. It also failed regardless of using ALLOCNOW or not, and I think also regardless of using WAITOK/NOWAIT.
Is there something which differs in bus dmamem/dmamap allocation during the boot and after, which would cause the behaviour?
Could I possibly tweak some parameters to make bus_dmamem_alloc()/bus_dmamap_create() fail less for dynamically loaded modules? Failing that, is there maybe some other physical memory allocation limit I should observe in driver code and try to limit the allocation with that?
Jaromir