[scottrogowski]

Ah. I think I understand a little better. Knowing nothing else about what you're doing, the way I would structure that would be to run multiple fastmap tasks with crawler + processor code and then pull down just the processed data. So basically, have fastmap as the most upstream node.

Whether or not that's possible with the realities of your application is another question of course :).

If you're curious, I'd be happy to provide some compute time (say 100 vCPU-hours) in exchange for feedback. Email me if that's at all interesting.