[snehesht]

Isn’t this easy for LLMs to avoid by passing an instruction to ignore any hidden links ?

[krackers]

Companies mass crawling don't use LLMs for crawling itself, that would be too expensive.

[snehesht]

Make sense, but doesn't necessarily have to be an llm, just a regular dom parser will be able to tell whether an element is visible or hidden.