"We’ve effectively hired them to deal with managing the hard parts of our scaleability infrastructure and they are working for a fraction of what it would cost to do this ourselves."
This, in my mind, is the most insightful line in this post. When cloud hosts like Heroku and GAE are discussed on HN often there is cost comparison between using them and doing the sysadmin yourself on Amazon, or on your own hardware. But what doesn't figure in is that with Heroku and GAE (I suppose more with GAE) is that you aren't just getting out of doing your own sysadmin work, you are also getting out of doing a lot of scaling work yourself. This is expensive, difficult work. Of course it is the type of work that many of us here salivate over, but that's another issue... ;)
It may be expensive, but you don't have to do "scaling" right now. A startup doesn't even know the business requirements or bottlenecks yet, how can it even begin to talk about scaling?
Scaling on GAE really works because of all the restrictions, and nothing stops you from restricting yourself in a similar way, e.g. in a similar vein the FriendFeed people have done: http://bret.appspot.com/entry/how-friendfeed-uses-mysql
The sysadmin part is also easy when you've got only 1 or 2 servers to worry about, the hard part of doing sysadmin is also when "scaling". Personally I can get an initial EC2 instance up and running in 2 hours tops, and then I can and have been automating that.
I've also worked on GAE apps, and from my experience the sysadmin stuff is replaced by at least 3/2 as much developer time, and this for trivial stuff.
Sorry for the analogy / bad language, but do you know what else scales? Fucking in the ass, i.e. no unexpected pregnancies, but that would be a stupid suggestion to make, wouldn't it?
At the moment, it's a tradeoff. Google App Engine frees you from operations but requires more development time. A VPS like EC2 requires less development time but a lot more operations time.
It's good to have the options but there's no silver bullet yet.
"On top of the great steady-state service, we were mentioned on ABC’s The View and had a massive surge in traffic which was handled flawlessly by AppEngine. It transparently scaled us up to 20+ instances that handled all of the traffic without a sweat. In the end, this surge cost us less than $1.00"
I'm the original author of the post. If anyone has any questions about AppEngine or what it's like to develop on that platform, feel free to post here.
I'd love to hear about your view on which frameworks to use for python (tipfy, webapp, tornado, django ports etc) and their advantages over each other. I've developed a webapp using django-nonrel, it was pretty quick to have it going but I have this uneasy feeling in my gut that something may be wrong.
My first impression of GAE (admittedly, many months ago) was that it was very painful to deal with loading or otherwise manipulating data in BigTable. Do you have any shortcuts or advice here? Has it improved?
Annoying Example: I realized that I needed to delete a lot of records from the table. I created a URL that I could ping periodically to delete a bunch, then get killed because it was taking too long, repeat.
I am really glad someone wrote a response from another POV. I was thinking about doing the same if I get the time on the weekend. We have been very happy appengine campers and the 1.4.0 SDK release looks very promising.
There are and have been some rough edges but overall it is a great platform and it was well worth it for us to invest time to evaluate and embrace the constraints.
Do you know of any good links for GAE development patterns?
I've read tips in various comments on various websites, such as use filters instead of GQL, using task queues and handling storage exceptions, but very little in one place :)
One particularly good use case for App Engine is a simple back end for thick client applications that do most of their heavy lifting on the client side.
I'm working on a moderately sized app that uses a fairly complex range of data, but I only have a few tables with a few columns each in the datastore. All the relational complexity of the schema is tucked away in blob fields and handled on the client. Besides greatly streamlining the back end code, it also cuts down on cpu cycles drastically by dumping most work to the user's machine, thereby saving lots of money.
I'm using this approach with a Flex UI, but it applies just as well to javascript.
As a programming noob, I've used both RailsTutorial's method (RoR/Git/Heroku), DjangoBook's Method (Py-Dj/no mention of version control/Apache+mod_python), and GAE's method.
I found RailsTutorial to be the easiest for my noobishness - though all were admittedly pretty good. GAE didn't seem easily portable at all compared to the other two - which was concerning, while DjangoBook's system seemed incomplete without teaching version control - like R-T does. For noobs, I like the easy way RoR/Git/Heroku are together, but I like Python better.
I'm anxious for http://www.Djangy.com to come out as the Heroku for Django and encouraged to see them use Git and not Hg.
It is an interesting conundrum right now. I am considering investing in using Ruby/Rails DUE TO Heroku. If djangy was ready and of comparable quality it would be an easy choice for me (I am comfortable in python, completely new to Ruby/Rails). But having to compare Heroku vs GAE vs Python-Stack is much more complex. I am not a fan of the limits in GAE, but I don't want to deal with system administration at this point. But I want to use python. Arg.
Just wanted to say that we're happy with GAE too. We use GAE for small, fast projects, like our main "brochure-ware" website. For big, burly projects, we use Amazon.
This really isn't about good and bad... it's about right tool for the job. The guy yesterday whining about GAE picked wrong.
I understand your article and benefits you seem to have received through GAE. But apart from sticking it through, I don't see useful message. I would like to know exact scalability issues you faced and how you leveraged GAE infrastructure to tackle them.
We face the same scalability issues as most other startups: a fairly steady stream of normal traffic and the occasional big spike from big mentions on blogs/TV/etc.
We could have done it the same way that we did with DotSpots: EC2 + MongoDB/MySQL. Instead we chose AppEngine to host our system and have spent very little time managing infrastructure.
An added bonus: we've also spent somewhere around $2.00 total to keep the lights on.
The main thing keeping me off AppEngine for my current project is that there doesn't seem to be an easy way to populate the blobStore. If there was an easy way to upload 10000 binary blobs I'd switch tomorrow.
I have tried quite a few different languages/frameworks on App Engine and Gaelyk is the one I have stuck with. I love it now. If I don't know how to do something in Groovy there is no hiccups because I can just default to writing Java code. Pairing it with Objectify makes it a very productive, fairly lightweight setup.
Yes, it's great. There are some limitations in the types of queries you can run, but other than that it's a pretty seamless transition. You also still have full access to the GAE framework's api if you need to do something specialized that web2py doesn't cover, though I haven't needed to do this very often (it also breaks portability of course). Overall an excellent combination.
The main point of this article is how GEA can scale very very fast... Basically, it sounds like: "you can win a lottery too, and GAE is there to scale in case you win a lottery".
Of course, installing everything on more expensive EC2/rackspace machine / mysql service would take half of a day, ....
Using heroku would be way more easier, but heroku is not so cool and hippie as Google.
I don’t think it has anything to do with hype.
Heroku is a great choice to run apps written in ruby. It is the only language they support — with support for node.js coming soon.
GAE, through Java, can run a multitude of languages, basically everything that runs on the JVM. Not every language is a good choice for GAE, but if you want, you can.
You’re right about installing many things on EC2/Rackspace is becoming easier every day. But tuning the system to get the most out of the infrastructure requires advanced knowledge in all layers of the stack. On appengine, you just have to write your app, you don't worry about the infrastructure. It’s a fairly good thing for many people.
With the generous free quotas, you don't have to pay a cent until your project gets some traction, which is not the case with EC2 for instance, because they charge you by the hour and not by the amount of resources used.
GAE is not perfect, but it enables small teams to rapidly launch prototype and iterate until the project takes off. Whether you have 10 users per day or 10M, you’re running on the same stack, built by some of the brightest engineers at Google.
Heroku would be easier for some, but not myself. I'm not a Ruby developer. I do most of my work in Java and dabble in Python for the occasional project. AppEngine fits this perfectly.
We spent three years working on DotSpots on the EC2 platform. While you can certainly boot up an instance of your application and get it running, there's still a lot more infrastructure work required to keep it going, scale beyond a single webserver, etc. On top of that, you've got to keep something like Pagerduty running to make sure that your instances aren't wedged or your MySQL box isn't running close to its storage limit.
The big benefit for me was being able to chose a language I'm strong in and just deploy something that automatically scales. This is particularly important during the early phase of the project where you are a lone coder doing all of the work.
The main point of the article was to pick the right tool for the job. The poster even mentions they are using EC2 for videos instead of wedging it into GAE.
Of course Heroku can be easier if you're using Ruby/RoR, but again using Ruby means Heroku is the right tool. Heroku is useless for someone that wants to use Java or Python to build their site.
At the end of the day a real entrepreneur doesn't care what's cool or hip and instead one focuses on the end product and how he can deliver it.
[+] [-] Goosey|15 years ago|reply
This, in my mind, is the most insightful line in this post. When cloud hosts like Heroku and GAE are discussed on HN often there is cost comparison between using them and doing the sysadmin yourself on Amazon, or on your own hardware. But what doesn't figure in is that with Heroku and GAE (I suppose more with GAE) is that you aren't just getting out of doing your own sysadmin work, you are also getting out of doing a lot of scaling work yourself. This is expensive, difficult work. Of course it is the type of work that many of us here salivate over, but that's another issue... ;)
[+] [-] bad_user|15 years ago|reply
Scaling on GAE really works because of all the restrictions, and nothing stops you from restricting yourself in a similar way, e.g. in a similar vein the FriendFeed people have done: http://bret.appspot.com/entry/how-friendfeed-uses-mysql
The sysadmin part is also easy when you've got only 1 or 2 servers to worry about, the hard part of doing sysadmin is also when "scaling". Personally I can get an initial EC2 instance up and running in 2 hours tops, and then I can and have been automating that.
I've also worked on GAE apps, and from my experience the sysadmin stuff is replaced by at least 3/2 as much developer time, and this for trivial stuff.
Sorry for the analogy / bad language, but do you know what else scales? Fucking in the ass, i.e. no unexpected pregnancies, but that would be a stupid suggestion to make, wouldn't it?
[+] [-] alecco|15 years ago|reply
It's good to have the options but there's no silver bullet yet.
[+] [-] Kilimanjaro|15 years ago|reply
That's the peace of mind I am talking about...
[+] [-] mmastrac|15 years ago|reply
[+] [-] znt|15 years ago|reply
[+] [-] kevindication|15 years ago|reply
Annoying Example: I realized that I needed to delete a lot of records from the table. I created a URL that I could ping periodically to delete a bunch, then get killed because it was taking too long, repeat.
[+] [-] tosh|15 years ago|reply
There are and have been some rough edges but overall it is a great platform and it was well worth it for us to invest time to evaluate and embrace the constraints.
[+] [-] krosaen|15 years ago|reply
[+] [-] nl|15 years ago|reply
One thing I'm curious about the is continuous deployment. AppEngine has a 1000 deployment limit (http://code.google.com/appengine/docs/quotas.html#Deployment...)
Is that a worry for you?
[+] [-] redstripe|15 years ago|reply
How do you do full text search?
The other two GAE topics posted this week had quite a few reports about datastore time outs. Has this been a problem for you?
[+] [-] mootothemax|15 years ago|reply
I've read tips in various comments on various websites, such as use filters instead of GQL, using task queues and handling storage exceptions, but very little in one place :)
[+] [-] danenania|15 years ago|reply
I'm working on a moderately sized app that uses a fairly complex range of data, but I only have a few tables with a few columns each in the datastore. All the relational complexity of the schema is tucked away in blob fields and handled on the client. Besides greatly streamlining the back end code, it also cuts down on cpu cycles drastically by dumping most work to the user's machine, thereby saving lots of money.
I'm using this approach with a Flex UI, but it applies just as well to javascript.
[+] [-] nickpinkston|15 years ago|reply
I found RailsTutorial to be the easiest for my noobishness - though all were admittedly pretty good. GAE didn't seem easily portable at all compared to the other two - which was concerning, while DjangoBook's system seemed incomplete without teaching version control - like R-T does. For noobs, I like the easy way RoR/Git/Heroku are together, but I like Python better.
I'm anxious for http://www.Djangy.com to come out as the Heroku for Django and encouraged to see them use Git and not Hg.
[+] [-] Goosey|15 years ago|reply
[+] [-] arthur_kalm|15 years ago|reply
[+] [-] mmastrac|15 years ago|reply
[+] [-] andrewljohnson|15 years ago|reply
This really isn't about good and bad... it's about right tool for the job. The guy yesterday whining about GAE picked wrong.
[+] [-] sushrutbidwai|15 years ago|reply
[+] [-] mmastrac|15 years ago|reply
We could have done it the same way that we did with DotSpots: EC2 + MongoDB/MySQL. Instead we chose AppEngine to host our system and have spent very little time managing infrastructure.
An added bonus: we've also spent somewhere around $2.00 total to keep the lights on.
[+] [-] dagw|15 years ago|reply
[+] [-] therockhead|15 years ago|reply
[+] [-] RoyceFullerton|15 years ago|reply
[+] [-] luisns|15 years ago|reply
[+] [-] danenania|15 years ago|reply
[+] [-] sdizdar|15 years ago|reply
Of course, installing everything on more expensive EC2/rackspace machine / mysql service would take half of a day, ....
Using heroku would be way more easier, but heroku is not so cool and hippie as Google.
[+] [-] iampims|15 years ago|reply
GAE, through Java, can run a multitude of languages, basically everything that runs on the JVM. Not every language is a good choice for GAE, but if you want, you can.
You’re right about installing many things on EC2/Rackspace is becoming easier every day. But tuning the system to get the most out of the infrastructure requires advanced knowledge in all layers of the stack. On appengine, you just have to write your app, you don't worry about the infrastructure. It’s a fairly good thing for many people.
With the generous free quotas, you don't have to pay a cent until your project gets some traction, which is not the case with EC2 for instance, because they charge you by the hour and not by the amount of resources used.
GAE is not perfect, but it enables small teams to rapidly launch prototype and iterate until the project takes off. Whether you have 10 users per day or 10M, you’re running on the same stack, built by some of the brightest engineers at Google.
[+] [-] mmastrac|15 years ago|reply
We spent three years working on DotSpots on the EC2 platform. While you can certainly boot up an instance of your application and get it running, there's still a lot more infrastructure work required to keep it going, scale beyond a single webserver, etc. On top of that, you've got to keep something like Pagerduty running to make sure that your instances aren't wedged or your MySQL box isn't running close to its storage limit.
The big benefit for me was being able to chose a language I'm strong in and just deploy something that automatically scales. This is particularly important during the early phase of the project where you are a lone coder doing all of the work.
[+] [-] matwood|15 years ago|reply
Of course Heroku can be easier if you're using Ruby/RoR, but again using Ruby means Heroku is the right tool. Heroku is useless for someone that wants to use Java or Python to build their site.
At the end of the day a real entrepreneur doesn't care what's cool or hip and instead one focuses on the end product and how he can deliver it.