top | item 46353128

(no title)

Simplita | 2 months ago

We ran into similar issues with aggressive crawling. What helped was rate limiting combined with making intent explicit at the entry point, instead of letting requests fan out blindly. It reduced both load and unexpected edge cases.

discuss

order

cheshire_cat|2 months ago

What do you mean by "making intent explicit at the entry point"?