top | item 46353128 (no title) Simplita | 2 months ago We ran into similar issues with aggressive crawling. What helped was rate limiting combined with making intent explicit at the entry point, instead of letting requests fan out blindly. It reduced both load and unexpected edge cases. discuss order hn newest cheshire_cat|2 months ago What do you mean by "making intent explicit at the entry point"? unknown|2 months ago [deleted]
cheshire_cat|2 months ago What do you mean by "making intent explicit at the entry point"? unknown|2 months ago [deleted]
cheshire_cat|2 months ago
unknown|2 months ago
[deleted]