Didn't realise the supply input voltage was so low, perhaps a mod or addition to provide at least 6-7 volts INTO the driver board will yield a decent 5v to the display and make it work?
Isn't there a 12volt rail on the matrix board that could feed a (added) 7805 regulator to give a solid 5v to the display?

I know you said you have no time to experiment, but it's inevitable, as there's no alternative fix besides installing Cherry displays.

my 2cents.