[chptung]

RE: pooling - yup, pooling for the win! As a self taught programmer, this was one block of code I was particularly proud of :)

[zasdffaa]

Crawling too fast from the same IP addy can get that IP blocked. Per 10 mins prob ok, per 1 minute may be too much depending on the volume being downloaded and the sensitivity (you mention saleable stuff in another comment so this might well make a difference). FYI. I did a reasonable amount of scraping myself.

(err, crawling = scraping? or maybe they different things)

Edit: tidy + clarify