By the way. This obviously does not at all solve the problem that
the OP had. He was writing code with the expectation that malloc()
should fail. [...] A killed process won't make the OP happy, even
if it was his own program/process.
I'm not sure that last sentence is true. As I read it, the reaction to
malloc failing would have been that we've put as much pressure on as we
can, so it's time to exit. If so, killing that process is a reasonable
reaction - and that's what my proposed approaches were based on.
Perhaps my understanding is wrong, in which case not much will help
except no overcommit.