[WiSaGaN]

Great article. To me, this highlights a key question in the era of rapidly advancing machine intelligence: if we know machine intelligence is progressing, what is more valuable to build for? As humans, we still find many tools useful even when doing knowledge work. For instance, a calculator. Sure, a smart person can perform calculations in their head, but it’s much easier to teach everyone how to use a calculator, which is 100% reliable in its intended domain.

In this era, we should build these kinds of tools for problems we know are straightforward ones you can’t get smarter than, even as intelligence continues to advance. Using tools like "bash" or command-line interfaces originally designed for humans is a good initial approach, since we can essentially reuse much of what was built for human use. Later, we can optimize specifically for machines, either accounting for their different cognitive structures (e.g., the ability to memorize extremely long contexts compared to humans) or adapting to the stream-based input/output patterns of current autoregressive token generators.

Eventually, I believe machine intelligence will build their own tools based on these foundations, likely a similar kind of milestone to when humans first began using tools.