In addition to what Jason says, I will point out that UVM used to
handle the referenced bit -- if in a crack-induced haze.

Originally, the Mach VM system did what Jason recommends.  It relied
on the fact that the order of items on the active and inactive lists
makes very little practical difference, so statistically it was close
to the current method but much more efficient in CPU usage.

When UVM was integrated, this all changed.  UVM revoked mappings when
a page was put on the inactive list -- but the active->inactive scan
never checked the referenced bit!  The referenced bit was only checked
during the inactive->free scan.  So, in addition to the transitions
being more expensive to begin with, more transitions happened because
the referenced bit was only checked after a page was deactivated.

When UBC was integrated, the only remaining check for the referenced
bit was quietly removed.  This cause[ds] pages to be strictly
round-robined, which is almost certainly a large part of the UBC
performance problems some people have noted (including myself).