[pr33tish]

Could you please point out the exact URL you are crawling? Love to match that with robots.txt file. I'm all for improving publicly available data.