Ok here's your problem, the difference between these 2 inputs must be between +- 10mv and 50mV. Since you only have 20mv it switches on the battery input, which means it shorts the capacitor C3 of the reset circuit to ground, which causes the µP to go into reset mode and it makes CE2 of the RAM low which is active high. That's why if you insert the battery it shuts off.
It seems your 5V supply is a bit on the low side. Most 7805 5V regulators have 4% tolerance, so it's below that (should be at least 4.8V), what I would do is replace the 7805 with a 7805ACT which is 2% and replace cap C85 (470µF/25V) and C86 (10µF) while you at it. Or use a battery of 3.6V.