[Throwadev]

Can you give any detail on what the end result is? What kind of data do you end up with in redshift, and what kind of queries? What kind of data do you extract from mixpanel?

Is all this running continuously, or different schedules for each worker? Any of these event based rather than schedule based?

[harlow]

> Can you give any detail on what the end result is? What kind of data do you end up with in redshift, and what kind of queries?

I'll have to get our BI team to create a follow-up post.

In a lot of ways its the most interesting part of this project. I'm not entirely sure what data we can share; I'll push to get something out.

> What kind of data do you extract from mixpanel?

The mobile devices push user interactions with the app as events to Mixpanel. We pull that data daily into Redshift, and this allows us to run historical reports and discover patterns of user behavior within the mobile app.

> Is all this running continuously, or different schedules for each worker? Any of these event based rather than schedule based?

The Extractors are Schedule based. So with Mixpanel for example we do a daily dump around 4am (once all the Mixpanel data is available for export).

With our Rails events we push them to IronMQ and the scheduler kicks off workers every 15 mins to pull them off.

The `Transformers` and `Loaders` are event based. So the Extractors would kick them off when they have completed their work.