Aahhh the old Low-Power Schottky fan-out limitations strike again 

Well found!, that woulda stumped many a great tech, i reckon you would have completely bypassed this had all 4 chips been LS04's...an head scratched for another year over it 
As a matter of interest, hopefully the manual shows 7404 on schematic and NOT LS04's ???

In desperate times, you could piggyback solder another LS04 on top of the existing one to 'buffer it up'.

Glad you found it the easy way though 

MM.