Ask HN: A SaaS that pulls from various sources with multiple concurrent tenants.

[tajddin]

For a personal project, I’m designing a system that pulls information from various sources, including IMAP, twitter and facebook (and others). Ideally, I’d like the user to be notified when the system receives data from any of these sources (thus timeliness is important), but I’d like some input into how best to accomplish this when there are potentially hundreds of concurrent users.

To solve part of the problem, I will be creating a separate service on a different AWS instance that connects to each data source and interacts with the DB independently from IIS.

Should this be done on an interval (say, every five minutes), looping through each of the users’ accounts and pulling data, or alternatively, should I use a multithreading approach, where each account has an associated polling thread?

Keep in mind, the technologies used are: ASP.NET MVC 4, SQL Server.

Your insight is much appreciated.

[shyn3]

I would start out with a high interval then slowly decrease for each individual user based on the number of times they post on each service.

You wouldn't want to go to my twitter page 20 times an hour when I only post once in a month. It might make more sense to check every 30mins to 1hr.

[tajddin]

Thanks for your response. The issue isn't so much for how often they post, it's more for how often they receive communiques. Being a small-timer, I'm trying to figure out the most resourceful way of accomplishing this.