[Dr_Birdbrain]

There are illegal moves and there are illegal moves. There is trying to move your king five squares forward (which no amateur would ever do) and there is trying to move your King to a square controlled by an unseen piece, which can happen to somebody who is distracted or otherwise off their game.

Trying to castle through check is one that occasionally happens to me (I am rated 1800 on lichess).

[CooCooCaCha]

This is an important distinction. Anyone with chess experience would never try to move their king 5 spaces, but LLMs will do crazy things like that.

[dgfitz]

Moving your king controlled by an unrealized opponent square is simply responded to with “check” no?

[james_marks]

No, that would break the rule that one cannot move into check