This looks very possible. It would only require a two modules for analysis and frontier management. It would be great to collaborate on something like this!
Please do think about it for the next couple of days and I will drop an email to you in the next 48 hours (I live in India). We can then decide how to take it forward.
Just curious: How long does an end-to-end crawl take and how much resources does the process consume (in terms of hardware/mem etc.,)?
vivegi|4 years ago
Please do think about it for the next couple of days and I will drop an email to you in the next 48 hours (I live in India). We can then decide how to take it forward.
Just curious: How long does an end-to-end crawl take and how much resources does the process consume (in terms of hardware/mem etc.,)?