[mcilai]

Are humans permanently inscrutable? In the long run, AI may be similar. Given how massively complex the world and the thing AIs learn are, it is not self evident that interpretable systems are possible. However, it's really worth trying.

More likely, we'd have to use AI systems to understand how other AI systems work. Or maybe, once AI systems get very good at language, develop a "truth serum" that will cause it to reveal its true opinions.