Now that's actually a good point.

At least one way around that comes to mind right offhand.  Rather than
storing the index in the heap structure, you can store a pointer to
where the driver records the index -- updating the driver's knowledge
whenever the structure moves.  However, this screws your TLB footprint
and starts to eat into any memory savings.