The interesting thing about all these services is that they're trying to abstract away the same types of problems that all developers come across: Identity, connectivity, sync/backup (and to some extent, deployment).
A different (and more disruptive?) approach to this would be to put more control of these things into the hands of end-users such that they provide the 'backend' into which you (the developer) load your application. i.e something like the app-store model but it connects to the user's 'personal cloud' (or the desktops of old - if you prefer). Such a system needs to be FOSS at its core but with a way for developers to get paid for providing value to end-users (who themselves get control of their data/networks).
Of course, this won't happen overnight, but the alternative is that everyone ends up using proprietary silos, with huge lock-in and innovation suffers as the tech giants get distracted with lawsuits.
I'm working with others on the distributed systems infrastructure we need to make this possible [1], so that we can get to a place where everyone can have their own piece of the cloud. In fact, one of the major components of this is Mirage, which has been discussed quite a bit on HN recently [2].
Do users really care about this? My guess is that 99% don't and don't want to deal with these details. The project sounds cool and like something the HN crowd might want, but my intuition is that most users won't understand the implications.
It also adds an extra step in the data storage process within the app, which increases friction and seems like it would be hard to get developers to adopt. All that said, I'd love to be proven wrong and see an implementation that is killer enough to solve these problems.
There was an MIT startup, Permabit, that tried to make a cloud based storage system which backed your own local disk. One idea was that your local storage is faster so you should use it for local things, and offline computation, but that if a shadow copy was being backed to the cloud in encrypted fashion. If your local storage fails, you just replace it, and the data dribbles back in from the cloud to your new local device.
They also had some hack whereby large common files (like app executables) where compressed using their hash code or something, but that is not the important idea in my opinion.
I think having that transparent local/cloud duality is one piece of the puzzle for a service which is both fast and local and also cloud based. You shouldn't have to care where your data lives. The other pieces of the puzzle are how to grant fine grained shared access without confusing users and introducing security holes.
I'm really intrigued by this model of internet infrastructure. Would you mind helping me understand the relevant difference between what you, Sandstorm[1], and OpenPDS[2] are doing?
Isn't that what App.net was supposed to be? By most standards that project was a huge failure. It's hard to proposition a user to start managing their own silo.
> A different approach to this would be to put more control of these things into the hands of end-users such that they provide the 'backend' into which you (the developer) load your application.
The big thing is each individual user would then be responsible for maintaining their own individual cloud. Otherwise, they'll just outsource that out and we're no better than before.
This sounds quite similar to what the (now more-or-less defunct) Singly was trying to accomplish. Having talked to a former employee about it at some length, the TL;DR is that 1) the technology and ideas are cool and 2) uptake was disappointing. That said, execution matters more than ideas when it comes to success.
I was actually thinking something like this for dropbox would be a no brainer.. like a cloudmail client for all your accounts, or an igoogle/ireader service that's hosted and uses your dropbox for its' data.
Why do so many people (and companies) pretend that app backends are just (logic-less) data stores?
I can definitely see the argument that some mobile games only need to store/sync some basic state information on behalf of the user, but I just don't see it as being that simple for most other apps.
I suppose this datastore-only backend model works for apps with users who are completely isolated from each other, but any time you want to allow users to interact with each other (or provide rules for how their resources interact with the world outside their little "user bubble"), you need business logic that resides somewhere other than the app; otherwise, your business logic (which may affect more than just the authenticated user) is in an untrusted environment. Not only that, but there are huge scalability problems associated with using a logic-less backend storage model if you want to do anything that involves more than just a single user's data.
Even for games, something as simple as providing a leaderboard is impossible (or extremely impractical) with this model. If you have no server-side logic to perform aggregate operations over multiple users, the only other option you'd have is to basically grab all the data and perform those operation in your app, which is absurd.
Backend development, server management, and deployment can be hard (though many PaaS solutions are making it easier than ever), but personally I'd never recommend building an app with a logic-less backend data store like this. Even if you don't think you'll need it today, in most cases you're going to need logic on the backend eventually. If you've started with this, you're just creating a lot more work for yourself when you finally do need to implement a feature that crosses the line from isolated-users-only to users-that-can-interact (or even just obtaining aggregate user data for business intelligence purposes).
Completely agree! At post.fm we're trying to design an email client which will work offline, and synchronize data when online (IMAP isn't sufficient for our needs) - Exchange style. We've thought long and hard about various solutions, and have tried syncing logic-less document-stores such as this. As you say - this doesn't really work, and there's definitely no way to handle anything remotely complex like transactions. In our case "app data" is effectively all your mail and more, and we can't expect each client to have a local copy of everything, meaning partial syncing of a collection is important for us (not something any solution out here seems to handle).
The solution we've come up with, which we'll hopefully get to open-source one day, is to model state over time as a persistent "event log", which means instead of syncing documents we sync events that clients perform, replaying events instead of document-update statements. This approach is infinitely flexible I believe, but requires more work if your clients are written in different languages (we use JS throughout so that isn't an issue).
I don't think anyone suggests that it's that simple. This entire space is young and evolving quickly. Identity and data sync is a good place to start, though.
For your example of a leaderboard, Parse has Cloud Code. It's designed to do exactly what you describe -- define a function that sits on Parse's server, handles client queries against your data, and returns the relevant data. Their example is for computing the average rating for a movie from your users' reviews: https://parse.com/docs/cloud_code_guide#functions
Amazon, Google, and Microsoft aren't there yet, but I can only imagine it's on their roadmap. And Parse's implementation only works for fairly easy stuff. But I think things will look very different in 5 years.
Parse/Cognito/etc allow you to quickly validate an idea that may or may not work out without having to invest in a backend engineer and a bunch of backend infrastructure. If it doesn't work, you've saved a bunch of time and money. If it does work, then you can start worrying about how to scale it and add more features. No piece of software has ever launched without technical debt, and there's much worse technical debt to take on than an outsourced backend.
instead of having to worry about building and managing a backend solution to handle identity management, network state, storage, and sync.
From my reading of this, this wouldn't in any way obviate the need for a backend system of your own. It just does the identity management, network state, storage and sync parts for you. Your own business logic on your own backend can use this to jump start you.
With so many new AWS features rolling out, I am worried that Amazon is spreading themselves too thin and will soon decide to stop supporting some of their less-popular services that I happen to be using.
As far as I can tell, there is no guarantee anywhere that they'll maintain any service for any length of time.
If that happens, it could waste weeks or months of engineering time trying to migrate to something else, or I'll just have to shut down my own services if I decide it's not worth migrating.
I am not too concerned about this – though I do think it's a valid concern considering the actions of other players in the space.
Sitting here in Seattle I see Amazon hoovering up all the local talent they can hire. And I see them putting up new tower after new tower to house them. I don't worry about them stretching themselves too thin as far as manpower goes.
And they don't have a history of closing down AWS products – for instance the first, (and not a runaway hit) Mechanical Turk is still with us.
It's a valid concern. AWS has been around for many years now with numerous services. Has there been a single offering which they retired or stopped supporting? I'm not aware of one.
Amazon doesn't just create services, throws them over the wall and see if they stick. Rather, they almost always RESPOND to customer requests. Dr. Werner Vogels, AWS's CTO makes a point about that approach at about any keynote he gives. Similarly, they try to also avoid feature creep by only providing features which customers really ask for.
Sounds like it saved you a lot of time getting your app up and running and I'm assuming they would give you a lot of time to migrate before just shutting it down. I wouldn't worry about this, however, this is something to consider any time you use a 3rd party to manage a service for you. I've had issues with fully managed RackSpace servers where the OS hit its EOL and it was on me to migrate everything to a new server. I've had this issue with fully managed database providers when they migrated to a new infrastructure and I had to move my data into a new database. There are other examples but most of them are pretty minor and none of them were too big of a deal.
While this (and other similar) Backend-aaS look very appealing at first glance, it seems to suffer from the same problem as most AWS offerings - terrible platform lock-in. Show me that I could move all of this to another provider if I need to, and then we can talk seriously.
I also wonder how they "manage[s] the complexity of conflict resolution" without manual intervention.
I'm a bit confused by your objection to "platform lock-in." Presumably a person or company who has chosen to use this service understands the technical limitations and features it entails. That includes understanding that AWS may in fact implement this service differently than other providers do and that a future move might entail some non-trivial effort. That isn't platform lock-in in any meaningful sense.
Besides, what makes you think you would be prevented from moving all this to another provider? From the FAQ (https://aws.amazon.com/cognito/faqs/), it seems that Cognito data is stored in a straight-forward Key-Value scheme. That's about as easy as it gets when it comes to data structure.
Moreover, the actual service itself would almost certainly be implemented differently with another provider. Why should Amazon, or anybody else who offers similar "menu of services" cloud computing, make it easy for you to do so (they make money off of you not moving)?
Is anyone delivering the equivalent with less lock-in? I can understand both sides of the argument but it seems that over time stuff always migrates towards more control (ie, people migrate OFF of Parse, Heroku, AWS, etc). I'd envision a much better set of images, Fabric/Chef recipes, to the point where you get your AWS but with vanilla OSS on generic hosting providers.
The trick with your suggestion of making data more portable is that if Amazon or any other platform provider would do so, then it would create a race to the bottom in terms of price. This would make the business quickly unappealing for the likes of Amazon. This is why no one will make their platform formats portable. At least not while things are in flux and the battle field is still disputed. Once the market matures (see the mobile telephony of 2000s) then you'll be able to port your phone number.
The similarities between Cognito and EnduroSync (Orando Labs, https://orandolabs.com - announced in May) are striking. Including the latin sounding names. Except that EnduroSync is an object store and has no data size limitations. Even the pricing is similar.
As far as Hacker New goes, we announced EnduroSync back in May, and got no promotion. Not one up vote?
I don't know how Amazon comes up with names for its products, but recently they all strike me as weird and stupid names. Do they confer with marketing first? Zocalo, Cognito? Really? These are the kinds of names you invent when you're looking for a cheap domain name for your startup. For Amazon, they don't make a lot of sense. They just seem like some executive out of their marketing depth and out of touch trying to be clever and anyone with sense afraid to speak up during the meeting when they come up with these names.
Is there anything like this in the enterprise space? Many of our customers want authentication against their internal Active Directory or other single sign-on solution. It would be nice if there was a company that exported an api, and did all the work of connecting up the various types of auth, so that we can focus on our own product rather than redo integration work that has probably been done by many other companies already, probably even for the customer's other services.
for $900/month you could do FAR better just rolling your own
these numbers are all reasonable for a mobile game... so it seems to me that the main use pattern would be for apps that don't need to synch very often. and if an app doesn't synch very often, why even bother throwing an auth wall at your users?
Or lets say thats about $12,000 a year. Very conservatively thats 1/10th of a developers cost. Could you really build, and operate, a truly equivalent service with 1/10th of a developer? Im dubious.
There are many cost/efficiency arguments against outsourcing infrastructure. I dont think this is one of them.
interesting. Higher than their example pricing which was based on 500k sessions/month with a sync at begin and end (https://aws.amazon.com/cognito/pricing/).
I find the aws pricing (and "cloud pricing" in general) fairly confusing, and the things is that you have to be so careful that the wrong checkbox isn't checked or something that opens the spigot to the credit card# you had to enter to try out the service on the free tier in the first place. Note that I saw that you can get an alert when it first charges anything via billing alerts (http://docs.aws.amazon.com/gettingstarted/latest/awsgsg-intr...), but still...
For those wary of vendor lock-in, or for those wanting a self-hosted similar service: Just yesterday I spun up my own oauth.io oauthd server. It's working out fairly well. It's only third-party authentication, it doesn't handle guest users or local authentication.
Am I missing something or could this be useful for non-mobile applications as well like HTML/JS apps if they provided an API outside of the iOS/Android SDKs?
A buzzword used in Cognito and Zocalo is "secure". I'm assuming that means that it's server-side encrypted at rest.
What would be really impressive is if there was an option to client-side encrypt the data before sending it to AWS. Of course, that would mean you'd have to move your syncing logic to the client-side, too, but having a good client-side encryption option would be a real differentiator.
I'm definitely finding it interesting to see Amazon releasing more and more offerings to make life as a developer easier. Will be interesting to see whether they can leverage some of these services to make developing for the Fire phone extra appealing.
This may sound stupid, so I apologise in advance for my ignorance....
I watched the video and the focus of this service seems to be on storing user data in a way that can be accessed by any device. I'm an API developer, isn't this just what every mobile app that uses an API to store data does, right? I personally don't write a different backend datastore for every device type that could connect to it, nor do I 'permanently' store data on the device! user data is always synced 'for me' over an API. I don't get this, can someone explain why we need this service? Do people develop apps different from me? I'm not being sarcastic, I'm genuinely interested.
When I see products like this, my first question is: could you write an free, open-source app with this without having to shell out your personal money to make it happen?
For Amazon Cognito, the answer appears to be "yes, until all of your users together store more than 10GB". ie. until your app gets popular.
So for free open-source apps, I think I'd prefer to stick with Dropbox or Google Drive, where the billing is associated with an account the user already has (which most users can operate entirely within the free tier). That way you don't run into a situation where you are a victim of your own success, by exceeding a free tier that applies to the sum of all user data in your app.
In response to this announcement, Orando Labs (https://orandolabs.com) has decided to offer EnduroSync and Identio with open source licenses and as paid AMI instances. We believe we have a unique solution to some difficult problems (identity and syncing), and want to see our solutions widely adopted. Read more at http://orandolabs.wordpress.com/2014/07/12/amazon-cognito
[+] [-] amirmc|11 years ago|reply
A different (and more disruptive?) approach to this would be to put more control of these things into the hands of end-users such that they provide the 'backend' into which you (the developer) load your application. i.e something like the app-store model but it connects to the user's 'personal cloud' (or the desktops of old - if you prefer). Such a system needs to be FOSS at its core but with a way for developers to get paid for providing value to end-users (who themselves get control of their data/networks).
Of course, this won't happen overnight, but the alternative is that everyone ends up using proprietary silos, with huge lock-in and innovation suffers as the tech giants get distracted with lawsuits.
I'm working with others on the distributed systems infrastructure we need to make this possible [1], so that we can get to a place where everyone can have their own piece of the cloud. In fact, one of the major components of this is Mirage, which has been discussed quite a bit on HN recently [2].
[1] http://nymote.org/blog/2013/introducing-nymote/
[2] https://news.ycombinator.com/item?id=8020125 and https://news.ycombinator.com/item?id=8005130 and https://news.ycombinator.com/item?id=7726748
[+] [-] michaelmior|11 years ago|reply
It also adds an extra step in the data storage process within the app, which increases friction and seems like it would be hard to get developers to adopt. All that said, I'd love to be proven wrong and see an implementation that is killer enough to solve these problems.
[+] [-] hqm|11 years ago|reply
They also had some hack whereby large common files (like app executables) where compressed using their hash code or something, but that is not the important idea in my opinion.
I think having that transparent local/cloud duality is one piece of the puzzle for a service which is both fast and local and also cloud based. You shouldn't have to care where your data lives. The other pieces of the puzzle are how to grant fine grained shared access without confusing users and introducing security holes.
[+] [-] tdaltonc|11 years ago|reply
[1] https://sandstorm.io/ [2] http://openpds.media.mit.edu/
[+] [-] CaveTech|11 years ago|reply
[+] [-] rschmitty|11 years ago|reply
User Dependency Injection!
[+] [-] ebiester|11 years ago|reply
How do you get grandma on her own personal cloud?
[+] [-] rpdillon|11 years ago|reply
[+] [-] tracker1|11 years ago|reply
[+] [-] tn13|11 years ago|reply
[+] [-] ForHackernews|11 years ago|reply
[+] [-] johnchristopher|11 years ago|reply
[+] [-] JPKab|11 years ago|reply
[+] [-] pjkundert|11 years ago|reply
[+] [-] dperfect|11 years ago|reply
I can definitely see the argument that some mobile games only need to store/sync some basic state information on behalf of the user, but I just don't see it as being that simple for most other apps.
I suppose this datastore-only backend model works for apps with users who are completely isolated from each other, but any time you want to allow users to interact with each other (or provide rules for how their resources interact with the world outside their little "user bubble"), you need business logic that resides somewhere other than the app; otherwise, your business logic (which may affect more than just the authenticated user) is in an untrusted environment. Not only that, but there are huge scalability problems associated with using a logic-less backend storage model if you want to do anything that involves more than just a single user's data.
Even for games, something as simple as providing a leaderboard is impossible (or extremely impractical) with this model. If you have no server-side logic to perform aggregate operations over multiple users, the only other option you'd have is to basically grab all the data and perform those operation in your app, which is absurd.
Backend development, server management, and deployment can be hard (though many PaaS solutions are making it easier than ever), but personally I'd never recommend building an app with a logic-less backend data store like this. Even if you don't think you'll need it today, in most cases you're going to need logic on the backend eventually. If you've started with this, you're just creating a lot more work for yourself when you finally do need to implement a feature that crosses the line from isolated-users-only to users-that-can-interact (or even just obtaining aggregate user data for business intelligence purposes).
[+] [-] akrymski|11 years ago|reply
The solution we've come up with, which we'll hopefully get to open-source one day, is to model state over time as a persistent "event log", which means instead of syncing documents we sync events that clients perform, replaying events instead of document-update statements. This approach is infinitely flexible I believe, but requires more work if your clients are written in different languages (we use JS throughout so that isn't an issue).
[+] [-] chaz|11 years ago|reply
For your example of a leaderboard, Parse has Cloud Code. It's designed to do exactly what you describe -- define a function that sits on Parse's server, handles client queries against your data, and returns the relevant data. Their example is for computing the average rating for a movie from your users' reviews: https://parse.com/docs/cloud_code_guide#functions
Amazon, Google, and Microsoft aren't there yet, but I can only imagine it's on their roadmap. And Parse's implementation only works for fairly easy stuff. But I think things will look very different in 5 years.
[+] [-] GregorStocks|11 years ago|reply
[+] [-] 300bps|11 years ago|reply
instead of having to worry about building and managing a backend solution to handle identity management, network state, storage, and sync.
From my reading of this, this wouldn't in any way obviate the need for a backend system of your own. It just does the identity management, network state, storage and sync parts for you. Your own business logic on your own backend can use this to jump start you.
[+] [-] fragsworth|11 years ago|reply
As far as I can tell, there is no guarantee anywhere that they'll maintain any service for any length of time.
If that happens, it could waste weeks or months of engineering time trying to migrate to something else, or I'll just have to shut down my own services if I decide it's not worth migrating.
[+] [-] nmcfarl|11 years ago|reply
Sitting here in Seattle I see Amazon hoovering up all the local talent they can hire. And I see them putting up new tower after new tower to house them. I don't worry about them stretching themselves too thin as far as manpower goes.
And they don't have a history of closing down AWS products – for instance the first, (and not a runaway hit) Mechanical Turk is still with us.
[+] [-] moonlighter|11 years ago|reply
Amazon doesn't just create services, throws them over the wall and see if they stick. Rather, they almost always RESPOND to customer requests. Dr. Werner Vogels, AWS's CTO makes a point about that approach at about any keynote he gives. Similarly, they try to also avoid feature creep by only providing features which customers really ask for.
[+] [-] todd3834|11 years ago|reply
[+] [-] yourad_io|11 years ago|reply
I also wonder how they "manage[s] the complexity of conflict resolution" without manual intervention.
[+] [-] Iftheshoefits|11 years ago|reply
Besides, what makes you think you would be prevented from moving all this to another provider? From the FAQ (https://aws.amazon.com/cognito/faqs/), it seems that Cognito data is stored in a straight-forward Key-Value scheme. That's about as easy as it gets when it comes to data structure.
Moreover, the actual service itself would almost certainly be implemented differently with another provider. Why should Amazon, or anybody else who offers similar "menu of services" cloud computing, make it easy for you to do so (they make money off of you not moving)?
[+] [-] pbreit|11 years ago|reply
[+] [-] rabc|11 years ago|reply
That's why I always advocate for building your own backend, if you have a long-run plan and/or expect to generate a lot of data for the app.
[+] [-] mnglkhn2|11 years ago|reply
[+] [-] orandolabs|11 years ago|reply
As far as Hacker New goes, we announced EnduroSync back in May, and got no promotion. Not one up vote?
[+] [-] aendruk|11 years ago|reply
[+] [-] bellerocky|11 years ago|reply
[+] [-] sophacles|11 years ago|reply
[+] [-] yalogin|11 years ago|reply
[+] [-] SimianLogic2|11 years ago|reply
imagine a game with 100k DAU (big, but not huge)
2 sessions per user per day
say 10 "saves" per session
100,000 * 2 * 10 = 2 million synchs / 10k * $0.15 = $30/day
for $900/month you could do FAR better just rolling your own
these numbers are all reasonable for a mobile game... so it seems to me that the main use pattern would be for apps that don't need to synch very often. and if an app doesn't synch very often, why even bother throwing an auth wall at your users?
[+] [-] donavanm|11 years ago|reply
There are many cost/efficiency arguments against outsourcing infrastructure. I dont think this is one of them.
[+] [-] nni|11 years ago|reply
I find the aws pricing (and "cloud pricing" in general) fairly confusing, and the things is that you have to be so careful that the wrong checkbox isn't checked or something that opens the spigot to the credit card# you had to enter to try out the service on the free tier in the first place. Note that I saw that you can get an alert when it first charges anything via billing alerts (http://docs.aws.amazon.com/gettingstarted/latest/awsgsg-intr...), but still...
[+] [-] tim333|11 years ago|reply
[+] [-] joeframbach|11 years ago|reply
[+] [-] skram|11 years ago|reply
[+] [-] diafygi|11 years ago|reply
What would be really impressive is if there was an option to client-side encrypt the data before sending it to AWS. Of course, that would mean you'd have to move your syncing logic to the client-side, too, but having a good client-side encryption option would be a real differentiator.
[+] [-] mikecb|11 years ago|reply
[+] [-] matthewarkin|11 years ago|reply
[+] [-] jflowers45|11 years ago|reply
[+] [-] glynjackson|11 years ago|reply
I watched the video and the focus of this service seems to be on storing user data in a way that can be accessed by any device. I'm an API developer, isn't this just what every mobile app that uses an API to store data does, right? I personally don't write a different backend datastore for every device type that could connect to it, nor do I 'permanently' store data on the device! user data is always synced 'for me' over an API. I don't get this, can someone explain why we need this service? Do people develop apps different from me? I'm not being sarcastic, I'm genuinely interested.
[+] [-] nostromo|11 years ago|reply
Instead being the vanguard of innovation, they're letting other companies validate the need for a service before swooping in.
Who's next? Probably Stripe.
[+] [-] eclipxe|11 years ago|reply
[+] [-] newobj|11 years ago|reply
[+] [-] haberman|11 years ago|reply
For Amazon Cognito, the answer appears to be "yes, until all of your users together store more than 10GB". ie. until your app gets popular.
So for free open-source apps, I think I'd prefer to stick with Dropbox or Google Drive, where the billing is associated with an account the user already has (which most users can operate entirely within the free tier). That way you don't run into a situation where you are a victim of your own success, by exceeding a free tier that applies to the sum of all user data in your app.
[+] [-] moonlighter|11 years ago|reply
[+] [-] orandolabs|11 years ago|reply