[rsf]

> In addition, the agent learns to exploit a flaw in the emulator to make a key re-appear at minute 4:25 of the video

After a bit of debugging, this appears to be a very intentional feature in the game rather than a flaw. That key appears after a while if you're not in the room (and don't have one).

Based on this disassembly: http://www.bjars.com/source/Montezuma.asm

Here's the relevant code with some annotations added:

https://goo.gl/VUDr9F

I'm not sure if this is a previously known feature in the game (a quick google search does not reveal much). It would be quite interesting if the RL agent was the first to find it!

PS: If you launch MAME with the "-debug" option and press CTRL+M you can see the whole memory (atari 2600 only has 128 bytes!!) while playing the game. If you keep an eye on the byte at 0xEA you will know when the key is about to pop up. Alternatively you can speed things along by changing it yourself to a value just below 0x3F.