The wrong lesson from this would be to immediately try and get your sites' performance to similar numbers. But what isn't explicitly stated in this post is that at the root of it, these guys first figured out which metrics matter for their goals. Only then did the optimization pay back. In their case, a few seconds can make decent impact because of volume. Someone with 100 uniques/day may have a better metric than load time to optimize.
There are always a thousand things that you can prematurely optimize (page load time, HTML, conversions, etc). The trick is figuring out what the highest ROI thing is for you to optimize right now given your specific point in time. If you get 100 visits/day set metrics around sign up, retention, or whatever is going to build you up to 1K/day. Then figure out how to get to 10K/day, and so on.
I see a lot of concern for integration with API, hosting, etc. but what I don't see is disgust. Disgust because to become president of the US, you must participate in this monumental financial circus of who spends what. Is this what people want? To know the best candidate will need to raise over a billion dollars to compete? Campaign costs should be in line with needs. Remember when Color got $40 mil for doing basically what other startups manage to do with a couple thousand dollars? Why would HN readers champion this kind of spending? The reasons are probably in line with the SuperPAC reasoning, which is "in order to change the game, you must play within the game's rules". Sure. But when do you say enough is enough? In Canada, our candidates spend on their campaign, but dollar-for-dollar, they are getting more done with less money. Somehow.
Or you could view this as a fascinating case study regardless of all that. I'm personally horrified that Obama got re-elected but this post is cool as hell and I'm absolutely fascinated by this work and appreciate that Kyle took a few hours to compose all of this and share it.
Not to defend the current campaign status quo, but when comparing the US to Canada it is worth pointing out the population of the US is ten times that of Canada.
A solution is to try to find ways to change the game.
An independent-reasoning educated populace is one way to fortify against manipulation.
Reply written in the NYC Public Library reading room. This inscription is out front:
"The City of New York has erected this building to be maintained forever as a free library for the use of the people. On the diffusion of education among the people rest the preservation and perpetuation of our free institutions."
Although Canada has been blessed with a twenty-year streak of good federal governance, I wouldn't say Canadian politicians get more done with their campaign spending.
I would say that their spending is strictly capped at the national and local level by Elections Canada, which means the major political parties simply aren't allowed to get into an advertising arms race. Without the caps - who knows?
Practical, but that sort of tight restriction on free speech is a very, very hard sell down here.
This misses a key point, which is that the Obama campaign relied heavily on small donations from _individuals_ in contrast with the heavily superpac funded Romney campaign which pulled from a small set of extremely wealthy donors (admittedly conjecture based on being an avid political news junkie). So, we're talking about crowd-sourcing a president; this is an encouraging direction, even if one finds the overall amount spent to be distressing.
It seems to me that disappointment in the current state of campaign finance reform and interest in how a campaign dealt with the current reality are not mutually exclusive.
At the risk of sounding like a troll, honest question here. Why wasn't the CVV part of this page? Seems like a lot of headaches and bad press could have been avoiding by just actually including it (and you know, less fraudulent donations).
Great work by the way with the optimizations. As patio11 noted, add a 0 or two to your rate and point people to this blog post.
Checking CVV isn't required, but may result in a higher transaction fee for the merchant. The campaign likely made a choice of "fewer fields = better conversion".
When running numbers like the A/B conversion improvements, how do you model or isolate external factors like "conversion rates increase as election draws closer". Is it correct to assume that no one is willing to continue to run original, unoptimized pages for some percent of visitors as a baseline?
Within any one particular A/B test, the changes in the true propensity to convert over time don't matter, since you're randomly apportioning folks into the alternatives, such that the only difference in the samples is whether they're exposed to A or B. Over the course of an A/B testing regimen, this doesn't apply, and you should be skeptical of results reported like "CR went from 10% to 15%", but a string of winning A/B tests is virtually statistically certain to be driving true wins and clients love hearing the change-over-time thing, so even if you're getting a tailwind from the environment one tends to report the numbers anyhow.
It is a poor allocation of resources for most clients to continue to run original, unoptimized pages just to be able to satisfy one's curiosity as to what portion of an improvement observed over, say, 6 ~ 24 months is due to one's own effort and what portion is extrinsic to the business. There's both a direct financial cost to maintaining old code branches and a regret cost of not giving them the best you're capable of serving. Round numbers: to add one sentence of clarification to this blog post, Team Obama would have had to give up ~$5 million in donations.
Can you describe how you set Akamai to host the site? I'm familiar with using S3 to host a static site, and using a CDN to host assets, but it's not clear to me how a CDN can be set to host a full static site. Did contribute.barackobama.com just mirror something like contribute-s3.barackobama.com?
Our Devops team configured Akamai so they would have specifics for you (@scottvdp).
With most CDNs you have what's called a pull zone that points to your origin server on a different domain. So when the user requests something like cdndomain.com/whateverpage the CDN looks for the file at pullzonedomain.com/whateverpage and then it serves what it gets back from the pull zone.
Akamai has more advanced configuration options than Amazon Cloudfront. One example of this is that our payment processors did not actually live on contribute.barackobama.com, only the static files did. So we used Akamai to setup a reverse proxy to POST the donations to a different domain. Without this reverse proxy it wouldn't have been possible because the same origin policy forbids cross domain, client-side POST requests.
In regards to Akamai and not Cloudfront - Akamai had a different offering than cloudfront. JdotP mentioned in this thread that we used Akamai GTM, it allowed us to do a lot of fun failover management which cloudfront didn't offer. It also allowed for SSL.
Would love to know more about the reasoning and discussion behind going from a 1 page process to 4 steps. (ie: The typical purchasing mindset vs. donation mind set)
Thanks for posting this fantastic write-up! I'm looking forward to reading more.
Does anyone know if other team members from the campaign have publicly discussed how other aspects of the online campaign worked? (Such as analytics, ads, CRM, social media, etc)
First of all, congratulations and thank you for doing the work you did to re-elect the president. Your write-up there is really interesting. I have a couple questions:
1. According to FEC reports, Blue State Digital made millions of dollars working with OFA... With that in mind, why is it that the campaign tech team had to build the redundant donation API? You would think for millions of dollars BSD probably should handle that, no? I only ask because in all the post-election stories a big theme is the "we decided to do it in house"...which makes sense...but begs the question - If you were "doing it all in house" then why was this third party vendor making so much bank?
2. Since we have you here on HN, can you tell us (very generally) what you and the other OFA Chicago Digital/Tech/Analytics people are going to be doing now? Going back into politics? Forming political/non-political startups?
I do not agree to it. Has any one seen the Obama campaign spending on software and infrastructure. They spend half a million dollar on Microsoft software (the highest spending), and then all the highlights of the campaign are coming from Open source software.
What was the benefit of spending on Microsoft software. Why it is always that money is spent on Microsoft and benefits are highlighted of using open source software.
Kyle, great post. You said the final donation was made via an XHR POST, did you guys have a plan B for users without JavaScript or were the numbers small enough to ignore?
Yes. It was very simple. We just put the URL to the vendor hosted platform's POST endpoint in the donation form's action attribute so that if you didn't have JavaScript the form would POST like a normal form.
Reading this write-up after having read Nate Silver's post[1] today on how the Republican party has a harder time getting Silicon Valley talent leads to an interesting thought. Going forward the party that can successfully target the younger, smarter base will have a SUBSTANTIAL advantage in developing better systems. Assuming there is not some ceiling on how much better technology can impact efficiency, these systems will translate to exponentially more effective platforms for the party with the brightest youth.
You could also argue that Republicans will slowly peel off disaffected or disenfranchised talent as technology in politics becomes less of a cottage industry for seasonal fruit pickers and they have the lawyers to push against the true ceiling you're talking about... the legal hurdles and the stagnation of innovation that comes with winning and scaling.
Seems a little dubious to claim that the increase in donations was a direct result of increased speed. The nature of the election cycle seems like more donations would take place as campaigning ramped up and the election date drew nearer. Were those kind of external factors accounted for in your analysis?
The increase in performance that I mentioned is directly due to the performance increase. We built identical pages on both platforms and tested them against each other at the same moment in time.
The only difference between the two pages was the page load time. Because we removed all other variables (time, appearance, functionality, etc.) we can say that the 60% performance increase lifted conversions by 14%.
I have a question about the donation API though, I feel like I am missing something:
We settled on a very simple solution of turning the hosted platform into a REST API. The only big change was adding JSON as an output option instead of HTML... We consumed the newly created donation API using JavaScript on static HTML pages which were served by our CDN (Akamai).
What operations did this API handle? Were donation submissions being sent from the user's browser to this API?
By the way, I think there might be a bug with your PJAX code on kylerush.net - when you click links to other parts of the site or to read other articles, the page title displayed in the browser stays as the initial title from the initial HTTP request.
The API was quite simple. Essentially it just processed a donation. It had some other functions to it that involved saving a users payment information to their account as well.
Yes, the donation submission was an POST request executed with JavaScript (AJAX) from the user's browser.
Thanks for noticing the bug! I'll have to check that out.
I've some questions to further understand the optimization process.
- In general, how did you measure improvements in conversion? What metric were you using? Something like LTV or the % of conversions each variation got?
- How exactly did you verify that the new version had a 14% larger conversion rate? Did you run an A/B test with both?
Kyle, amazing post! You mentioned that the 60% speed increase led to a 14% donation increase. You used webpagetest.org to measure the 60% speed increase. How did you measure the donation increase? Is there some sort of A/B tool for CDNs to help quantify the impact of speed improvements on CR?
We used Optimizely (https://www.optimizely.com/) to conduct most of our a/b testing. Optimizely works on the client level (browser), not the server level (CDN).
Optimizely is the best a/b testing software I've ever used. It's flexible and easy to use. I suggest you check it out.
How did you implement the fail-over to a different amazon region when one fails? I'm interested in common approaches to this SPOF and what have been done in this case.
*seeing the discussion that is going on around this post, it almost feels off-topic to post a technical question :)
[+] [-] patio11|13 years ago|reply
[+] [-] Radim|13 years ago|reply
[+] [-] bhousel|13 years ago|reply
[+] [-] zaidf|13 years ago|reply
[+] [-] kevinconroy|13 years ago|reply
There are always a thousand things that you can prematurely optimize (page load time, HTML, conversions, etc). The trick is figuring out what the highest ROI thing is for you to optimize right now given your specific point in time. If you get 100 visits/day set metrics around sign up, retention, or whatever is going to build you up to 1K/day. Then figure out how to get to 10K/day, and so on.
[+] [-] tomkin|13 years ago|reply
[+] [-] crikli|13 years ago|reply
[+] [-] brown9-2|13 years ago|reply
[+] [-] ceejayoz|13 years ago|reply
[+] [-] ISL|13 years ago|reply
An independent-reasoning educated populace is one way to fortify against manipulation.
Reply written in the NYC Public Library reading room. This inscription is out front:
"The City of New York has erected this building to be maintained forever as a free library for the use of the people. On the diffusion of education among the people rest the preservation and perpetuation of our free institutions."
http://www.gutenberg.org/files/27954/27954-h/27954-h.htm
[+] [-] gyardley|13 years ago|reply
I would say that their spending is strictly capped at the national and local level by Elections Canada, which means the major political parties simply aren't allowed to get into an advertising arms race. Without the caps - who knows?
Practical, but that sort of tight restriction on free speech is a very, very hard sell down here.
[+] [-] gdubs|13 years ago|reply
[+] [-] alevans4|13 years ago|reply
[+] [-] clarky07|13 years ago|reply
Great work by the way with the optimizations. As patio11 noted, add a 0 or two to your rate and point people to this blog post.
[+] [-] ceejayoz|13 years ago|reply
Checking CVV isn't required, but may result in a higher transaction fee for the merchant. The campaign likely made a choice of "fewer fields = better conversion".
[+] [-] trotsky|13 years ago|reply
[+] [-] patio11|13 years ago|reply
It is a poor allocation of resources for most clients to continue to run original, unoptimized pages just to be able to satisfy one's curiosity as to what portion of an improvement observed over, say, 6 ~ 24 months is due to one's own effort and what portion is extrinsic to the business. There's both a direct financial cost to maintaining old code branches and a regret cost of not giving them the best you're capable of serving. Round numbers: to add one sentence of clarification to this blog post, Team Obama would have had to give up ~$5 million in donations.
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] luigi|13 years ago|reply
And why Akamai and not Amazon Cloudfront?
[+] [-] kylerush|13 years ago|reply
With most CDNs you have what's called a pull zone that points to your origin server on a different domain. So when the user requests something like cdndomain.com/whateverpage the CDN looks for the file at pullzonedomain.com/whateverpage and then it serves what it gets back from the pull zone.
Akamai has more advanced configuration options than Amazon Cloudfront. One example of this is that our payment processors did not actually live on contribute.barackobama.com, only the static files did. So we used Akamai to setup a reverse proxy to POST the donations to a different domain. Without this reverse proxy it wouldn't have been possible because the same origin policy forbids cross domain, client-side POST requests.
[+] [-] harper|13 years ago|reply
[+] [-] ccarpenterg|13 years ago|reply
[+] [-] justjimmy|13 years ago|reply
[+] [-] akamaka|13 years ago|reply
Does anyone know if other team members from the campaign have publicly discussed how other aspects of the online campaign worked? (Such as analytics, ads, CRM, social media, etc)
[+] [-] luigi|13 years ago|reply
http://www.quora.com/obamatech
[+] [-] dccoolgai|13 years ago|reply
1. According to FEC reports, Blue State Digital made millions of dollars working with OFA... With that in mind, why is it that the campaign tech team had to build the redundant donation API? You would think for millions of dollars BSD probably should handle that, no? I only ask because in all the post-election stories a big theme is the "we decided to do it in house"...which makes sense...but begs the question - If you were "doing it all in house" then why was this third party vendor making so much bank?
2. Since we have you here on HN, can you tell us (very generally) what you and the other OFA Chicago Digital/Tech/Analytics people are going to be doing now? Going back into politics? Forming political/non-political startups?
[+] [-] jdotp|13 years ago|reply
Many of us are joining startups, some are starting startups, and others are yet to decide.
[+] [-] ceejayoz|13 years ago|reply
[+] [-] faisalchohan|13 years ago|reply
What was the benefit of spending on Microsoft software. Why it is always that money is spent on Microsoft and benefits are highlighted of using open source software.
[+] [-] tterrace|13 years ago|reply
[+] [-] kylerush|13 years ago|reply
[+] [-] stupandaus|13 years ago|reply
[1]: http://fivethirtyeight.blogs.nytimes.com/2012/11/28/in-silic...
[+] [-] buddhaofdoubt|13 years ago|reply
[+] [-] sshconnection|13 years ago|reply
Still, very interesting writeup and congrats!
[+] [-] kylerush|13 years ago|reply
The only difference between the two pages was the page load time. Because we removed all other variables (time, appearance, functionality, etc.) we can say that the 60% performance increase lifted conversions by 14%.
[+] [-] delano|13 years ago|reply
[+] [-] gideon_b|13 years ago|reply
[+] [-] brown9-2|13 years ago|reply
I have a question about the donation API though, I feel like I am missing something:
We settled on a very simple solution of turning the hosted platform into a REST API. The only big change was adding JSON as an output option instead of HTML... We consumed the newly created donation API using JavaScript on static HTML pages which were served by our CDN (Akamai).
What operations did this API handle? Were donation submissions being sent from the user's browser to this API?
By the way, I think there might be a bug with your PJAX code on kylerush.net - when you click links to other parts of the site or to read other articles, the page title displayed in the browser stays as the initial title from the initial HTTP request.
[+] [-] kylerush|13 years ago|reply
Yes, the donation submission was an POST request executed with JavaScript (AJAX) from the user's browser.
Thanks for noticing the bug! I'll have to check that out.
[+] [-] rynop|13 years ago|reply
[+] [-] msinia|13 years ago|reply
I've some questions to further understand the optimization process.
- In general, how did you measure improvements in conversion? What metric were you using? Something like LTV or the % of conversions each variation got?
- How exactly did you verify that the new version had a 14% larger conversion rate? Did you run an A/B test with both?
[+] [-] naterosenberg|13 years ago|reply
[+] [-] kylerush|13 years ago|reply
Optimizely is the best a/b testing software I've ever used. It's flexible and easy to use. I suggest you check it out.
[+] [-] maximveksler|13 years ago|reply
How did you implement the fail-over to a different amazon region when one fails? I'm interested in common approaches to this SPOF and what have been done in this case.
*seeing the discussion that is going on around this post, it almost feels off-topic to post a technical question :)