[jmwilson]

There's different levels of "understanding how things work" and the author makes it clear what kind of understanding he's going for. If you look at the source code to program, you should be able to point to any line of code and answer "What does this particular line of code do? Why is it important and how does it relate to the rest of the design?" Same applies to a part on a electronic schematic or a mechanical drawing. There is likely no similar meaningful answer to those questions if you look at a particular weight in a model.

[therealdrag0]

I’ve seen mention of researchers being able to point to specific neural pathways in AI for both simple things and even more advanced abstract things like “lying” or “truthfulness”. So it’s not a total lost cause maybe.