I'm more interested in correctness first, and then performance, but the good news is that it's definitely not a moving target, and there are only a limited number of optimizations worth pursuing. GCC has builtin code to do basically the right thing if only we give it the correct values for the relative costs of different kinds of operations. In a way, the good news is that even the most complex VAX isn't superscalar, so there's no need or benefit to writing an instruction scheduler to try to keep the pipelines full by reordering instructions. Most VAX CPUs are pipelined, but I suspect the most benefits would come from trying to prefer certain forms of instructions over others, which is where the relative cost tables come in.
So I'll try to come up with a suitable benchmark program that runs a representative set of microbenchmarks to calculate the basic ratios that could be inserted into GCC in place of the hardcoded values that are there now, and also try to figure out the rest of the bugs and issues with the ELF symbol loading that all seems both confusing and suboptimal right now. But I think I'm asking the right questions and focused on the right things to speed up. There really isn't anything horribly wrong with the current VAX backend except for the fact that various bugs have crept in over the years as different parts of it have been hacked on by different people who didn't quite understand the implications on other parts of it.