Daniel, when I first read you post I was going to send you a pm saying you're crazy or have been imbibing too much. My reasoning was a display is just a display so modes wont make any difference to what is put up on them, so attract mode or game play - who cares from the display point of view. Then I started wondering if it's maybe a timing and persistence issue - in attract mode the cpu would (presumably) spend a whole lot more time updating the display pia as it doesn't have to do game scoring and state calculations, maybe this is insignificant in the scheme of things, but I suspect it's not. Imagine all the game code just bypassed because game is not in play, (yes, attract mode code has to run, but this, I think, would be much shorter and therefore faster). I wasn't really convinced by this, but if I were desperate I'd look at it - maybe put the scope on the chip select pins when in display compared to game mode?
Another though along these lines is to look at the cpu data buffer chips IC8 and IC9 - notice that when in attract mode the data comes from the memory chips IC19 - it's just high score and last score and this will be supplied direct from the cmos chip. The cpu logic drawings show this. In game mode data comes via the cpu through the buffer chips. The buffer chips have a read/write select input driven by a gate on IC7 - I'd look at this as well. I think Cliff's guides says these buffers can be dispensed with - are yours still present? I have a few boards, on some they are present on others they have been replaced by bridges. If yours are present, I'd read Cliff and delete them if he agrees. I might be clutching at straws here, but if I had the problem this is the path I'd be sniffing around.
I don't reckon it will be the BCD / 7 seg decoders - they work in attract mode and are fairly straight forward devices.
I also don't totally agree with your statement "It a master display issue for sure" - what happens if you put your Gorgar master display in Time Warp? If my theorising above is correct I would expect the Gorgar master display to play up in Time Warp in the same way you describe.