RankScience automates split-testing for SEO to grow organic search traffic for businesses. 80% of clicks from Google go to organic results and yet most companies don't know how to improve their SEO, or can't effectively measure their efforts to do so. Because of the scale of SEO and the constant change of both Google's ranking algorithm and your competitors' SEO campaigns, the only way to succeed in the long-run is with software and continuous testing.
We've built a CDN that enables our software to provide tactical SEO execution and run A/B testing experiments for SEO across millions of pages. Experiments typically take 14-21 days for Google to index and react to changes, and we use Bayesian Structural Time Series and Negative Binomial Regression models to determine the statistical significance of our experiments.
Our software is 100% technical SEO, and doesn't do anything black-hat, spammy, or anything related to link-building. One of our goals is to bring transparency and shed light on what is largely considered a shady industry, but is so important to so many companies' revenue and growth. In fact, If SEO didn't have such a bad reputation, we think someone else would have built this a long time ago.
SEO as an industry earned itself a stigma for being spammy: between buying links, creating low-quality pages stuffed with keywords and text intended for Google rather than humans, and the used car salesmen attitude that many SEOs have, many people have been conditioned to dismiss SEO as an invalid or illegitimate growth channel.
We're software engineers-turned-SEO's, who have previously consulted for dozens of companies on SEO, from YC startups to Fortune 500 companies like Pfizer. We previously shared our case study with HN, where we increased search traffic to Coderwall with one A/B test: https://www.rankscience.com/coderwall-seo-split-test
Ask us anything! We'd love to answer any questions you have about SEO, A/B testing, and RankScience.
A few questions about your product (I run SEO and digital marketing at LendUp - YCW12). This is very, very cool.
* What types of optimizations does it do/do you test?
* I assume tests take a while to run, waiting for Google to re-index etc. Do you essentially monitor rank changes, assume that Google has re-indexed at that point, and use that as the data to optimize on? Or do you have some smart way to monitor for when Google re-indexes?
* I'm again assuming here - that a user will plug in a few variations of things to test, and let your software test them? Or are there automatic optimizations the software tries to make?
* How many tests can one do in a given time period, without confounding test & control variants? It seems like they would take a while to run?
* Is there a good way to control for non-technical/non-content-based changes (e.g. external, links)? For example we get hundreds of negative links pointed at us per week. Do we just hope/assume that's not the cause of rank changes?
We test everything on-page. This usually includes CTR stuff, like titles and meta descriptions and paragraph text that Google extracts for the meta descript, headers, images, calls to action, even conversion rates.
Correct, we continually iterate based on feedback like rankings and clicks, and we're working on smarter ways to monitor Google's crawler too!
Users are as hands-on or hands-off as they'd like. We usually own the experiments, but some of our customers design their own too.
The number of tests depends on the number of pages and your traffic. Our largest customers have full factorial experiments with thousands of concurrent split tests every day.
Guys, The idea looks great. Congrats on the launch.
2 SEO questions:
1) a CDN means "thousands of websites on one IP address". Google doesn't like it UNTIL it knows, the IP belongs to a well-known CDN like cloundfront/cloudflare etc. Please comment?
2) An A/B test might look like "cloaking" to googles. How exactly do you run it? I assume, not in parallel, but "variation A then variation B" - correct? If yes, does it mean I have to leave the website UNTOUCHED for 21 days so the test results are not distorted by my other activities? (adding new content, internal links etc)
1) We actually work with CloudFlare/CloudFront etc if companies are already using them. In that case, we sit in between CloudFlare and your origin webserver. But if you don't already have a CDN in front of your website, our SEO CDN goes in front of your origin webserver, and we use AWS so as far as Google is concerned it looks like AWS. We also use different IPs for every customer.
2) We don't do cloaking - we're essentially testing somewhat small tweaks across groups of similar pages, and seeing if they have any impact on clicks or rankings. It is preferred if the pages remain largely the same, but they don't have to be totally untouched.
(1) The idea of the product is clearly conveyed, but I'm confused on exactly how it works. The landing page mentions that title tags, headlines, meta-tags, etc get tweaked - exactly how is this done? Do I have to manually enter a bunch of alternative text, or are you using a big fancy thesaurus to switch out some key terms?
(2) How do you evaluate performance of the product? Solely through click rates, or by search rankings? How often do google search results get updated? In short, how do I know the product is working?
(1) We use both humans and software to generate experiments. For customers, it's completely automated.
(2) We look at all primary search metrics (clicks, impressions, CTR, and rankings), with clicks being our main metric. Search results get updated at a pace determined by Google's crawl rate, which varies per site depending on multiple factors including domain authority. We use bayesian structural time series and negative binomial regression models to measure impact and statistical significance to power our data-driven SEO recommendations.
As somebody who wrote an article about A/B testing title tags in 2011 before it was cool [1], this is an awesome idea. I've talked about SEO with many companies and coming up with the proper title tags and meta descriptions alone is often worth so much traffic for such little effort (once you get past the upfront cost of running the tests).
However, I think a critical aspect of SEO is thinking about an entire site holistically. Not only because certain signals are site-wide, but because a key aspect of SEO is deciding what pages of your website are "good" for SEO and which ones aren't, and then focusing on making the "good" pages better and not worrying about the "bad" pages. Good and bad in quotes because it is often quite a bit of an art and not a science.
How does RankScience play into this? You've nailed the on-page stuff but is there any world in which RankScience is able to talk about a site holistically and recommend which types of pages and content seem to be working most effectively (and maybe even suggesting pages that could be removed/de-indexed)? Or do you leave that to SEO consultants and you just nail the hell out of the on-page stuff.
It's a true honor to speak with an SEO A/B testing OG! :)
> How does RankScience play into this?
That's a really great point. Right now, our software either focuses on only pages which are good for SEO, or it runs less frequent experiments on the "bad" pages. We've had a few success stories of turning "bad" pages into good ones and having them become big revenue generators for our customers, but that's not common as I'm sure you can imagine.
Though using A/B testing to improve user experience or conversion rates is fine, I thought using A/B testing to reverse engineer the ranking algorithm was against the guidelines. Has this been updated?
> Best practices for website testing with Google Search
> The amount of time required for a reliable test will vary depending on factors like your conversion rates, and how much traffic your website gets; a good testing tool should tell you when you’ve gathered enough data to draw a reliable conclusion. Once you’ve concluded the test, you should update your site with the desired content variation(s) and remove all elements of the test as soon as possible, such as alternate URLs or testing scripts and markup. If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if you’re serving one content variant to a large percentage of your users.
Next to this, the advice is to use rel="canonical" to avoid duplicate issues with Googlebot crawling your variations. When using rel="canonical" this should not show you how a variation influences ranking.
> If you’re running an A/B test with multiple URLs, you can use the rel=“canonical” link attribute on all of your alternate URLs to indicate that the original URL is the preferred version. We recommend using rel=“canonical” rather than a noindex meta tag because it more closely matches your intent in this situation.
This is link is related to Conversation Rate Optimization testing (like Optimizely). We don't do A/B testing on single pages, or do cloaking or anything of the sort, but we run experiments across groups of URLs, and then we sum up the results and run our analysis.
Also, our goal is not to deceive Google in anyway - a lot of our tests are related to increasing CTR (which is testing humans) and on-page times. (again testing humans) Overall we're trying to make pages better according to Google guidelines -- which leads to a better experience for users.
Just want to chime in here and share that I'm a very happy customer. Ryan, Dillon, and Chad are some super smart guys who deeply understand SEO. I use RankScience for 7 Cups and Edvance360 and I've seen a huge ROI. They always find time to meet with me and my other team members. The advice and feedback they provide is worth the cost of the service alone. Highly, highly, highly recommended!
seems like a hard nut to crack, considering there are constant changes to google algos that could screw with test results mid-test, and many other moving parts. I think what you are doing is great, have built similar tools in the past and always ran into issues due to so many moving parts, but im not a statistician.
Also what do you consider success? what if one title/meta combo gets a better rank but fewer clicks(ie; the bots like it but the users dont)? which one would you keep? I know many people like to brag about how many rankings they have but as an SEO for over a decade, i think traffic count is still king. in that id rather rank 11th for an extremely high volume term, then rank 2nd for a low volume one.
another thought... a blackhat site gets banned halfway through one of your tests and you move up based on no actions you took. thus you would get a false positive, no?
- Is there a minimize traffic to the website required to get statistically significant results? Or does it matter since you're trying to move the site up in the Google ranking?
- Does site traffic affect the length of time it takes to test?
We look at Clicks, Rankings and Impressions. We use Google Analytics, Webmaster Tools, and another third party data source.
More clicks is obviously the ultimate goal, but sometimes we also have significant experiments where rankings or impressions going up are the key metric
I consult with startups and tech companies (recently: SurveyMonkey) on SEO, so this is super-interesting... especially the automated aspect, which is pretty novel in this space.
What are some of the specific types of automated tests that you run?
It seems like you integrate with the likes of Google (Webmaster Tools) and Cloudflare in ways not done before, so my assumptions here are probably a bit outdated.
Let's say you want to test a new title. You collect stats for a week, change the title, collect stats for another week. The two datasets are then compared. Am I close?
Does it mean that you can't AB test in parallel? If so, it's not optimal for time sensitive stuff like breaking news.
We don't run experiments on a single page, but instead run them across groups of similar pages. The easiest way to think about this is running a test across a "product" template on an e-commerce site. A site could have thousands of product pages, but they're all on the same template. We would grab a set of a few thousand product pages, and split them into control and variant groups, execute the change on the variant group, and monitor how Google reacts to the groups in aggregate over a period of time.
Depending on how many pages you have, we can run many tests in parallel.
The only way to deal with Google algorithm updates is through experimentation. A data-driven approach is the only approach. : ) Staying abreast of what both Google is saying publicly and what the SEO community is saying also helps (ie Google announced they're cracking down on intrusive pop-ups for mobile sites)
Site performance is always relative to your competitors. Optimal server response time is around 200ms, and that's what folks should strive for, but I've seen sites have really slow pages and still get lots of traffic.
You guys mention < 25ms as the performance hit of having you guys in front of the client's application. Is there a more concrete SLA you guys provide? Do you have a sense of what that hit might cost in page rank (there's some theories out there that time to first byte is one of the features that search engines reward)?
Hey 0bfus, we're working on a fancy performance metrics dashboard and status page to answer these questions! Also, we've done extensive research on pageload speed's effect on page rank, and we found that <25ms of incremental latency does not have any impact. We're working on a case study for the latter!
I am bit wary of the idea, despite the fair intentions, but Google in particular tends to view anything remotely close to 'improving' search ranking using 3rd party 'software', as a threat to their own Ego-rithms.
I wish I could use a tool like this and know that it won't affect my rankings negatively, but I feel that Google will eventually punish sites that do a lot of A/B testing.
We focus on being automated! Our software never stops working to grow your search traffic, even when you're asleep or if you don't want to think about SEO at all.
We use Google Analytics, Google Search Console, and third party services to supplement keyword rankings data -- we don't crawl Google SERPs. :) Plenty of other companies do, though. Moz has great keyword ranking data.
[+] [-] ryanb|9 years ago|reply
We've built a CDN that enables our software to provide tactical SEO execution and run A/B testing experiments for SEO across millions of pages. Experiments typically take 14-21 days for Google to index and react to changes, and we use Bayesian Structural Time Series and Negative Binomial Regression models to determine the statistical significance of our experiments.
Our software is 100% technical SEO, and doesn't do anything black-hat, spammy, or anything related to link-building. One of our goals is to bring transparency and shed light on what is largely considered a shady industry, but is so important to so many companies' revenue and growth. In fact, If SEO didn't have such a bad reputation, we think someone else would have built this a long time ago.
SEO as an industry earned itself a stigma for being spammy: between buying links, creating low-quality pages stuffed with keywords and text intended for Google rather than humans, and the used car salesmen attitude that many SEOs have, many people have been conditioned to dismiss SEO as an invalid or illegitimate growth channel.
We're software engineers-turned-SEO's, who have previously consulted for dozens of companies on SEO, from YC startups to Fortune 500 companies like Pfizer. We previously shared our case study with HN, where we increased search traffic to Coderwall with one A/B test: https://www.rankscience.com/coderwall-seo-split-test
Ask us anything! We'd love to answer any questions you have about SEO, A/B testing, and RankScience.
[+] [-] etler|9 years ago|reply
Does it integrate with web master tools and google analytics?
Does it help identify page populations that are stable and similar enough to be compared?
How are the different test versions sent to you guys?
Can we use our own CDN on top of your CDN?
What about dynamic pages that change over time?
How big of a sample size is needed to get statistically significant results?
[+] [-] xiaoma|9 years ago|reply
[+] [-] austenallred|9 years ago|reply
* What types of optimizations does it do/do you test?
* I assume tests take a while to run, waiting for Google to re-index etc. Do you essentially monitor rank changes, assume that Google has re-indexed at that point, and use that as the data to optimize on? Or do you have some smart way to monitor for when Google re-indexes?
* I'm again assuming here - that a user will plug in a few variations of things to test, and let your software test them? Or are there automatic optimizations the software tries to make?
* How many tests can one do in a given time period, without confounding test & control variants? It seems like they would take a while to run?
* Is there a good way to control for non-technical/non-content-based changes (e.g. external, links)? For example we get hundreds of negative links pointed at us per week. Do we just hope/assume that's not the cause of rank changes?
[+] [-] dillonforrest|9 years ago|reply
We test everything on-page. This usually includes CTR stuff, like titles and meta descriptions and paragraph text that Google extracts for the meta descript, headers, images, calls to action, even conversion rates.
Correct, we continually iterate based on feedback like rankings and clicks, and we're working on smarter ways to monitor Google's crawler too!
Users are as hands-on or hands-off as they'd like. We usually own the experiments, but some of our customers design their own too.
The number of tests depends on the number of pages and your traffic. Our largest customers have full factorial experiments with thousands of concurrent split tests every day.
[+] [-] jitbit|9 years ago|reply
2 SEO questions:
1) a CDN means "thousands of websites on one IP address". Google doesn't like it UNTIL it knows, the IP belongs to a well-known CDN like cloundfront/cloudflare etc. Please comment?
2) An A/B test might look like "cloaking" to googles. How exactly do you run it? I assume, not in parallel, but "variation A then variation B" - correct? If yes, does it mean I have to leave the website UNTOUCHED for 21 days so the test results are not distorted by my other activities? (adding new content, internal links etc)
[+] [-] ryanb|9 years ago|reply
1) We actually work with CloudFlare/CloudFront etc if companies are already using them. In that case, we sit in between CloudFlare and your origin webserver. But if you don't already have a CDN in front of your website, our SEO CDN goes in front of your origin webserver, and we use AWS so as far as Google is concerned it looks like AWS. We also use different IPs for every customer.
2) We don't do cloaking - we're essentially testing somewhat small tweaks across groups of similar pages, and seeing if they have any impact on clicks or rankings. It is preferred if the pages remain largely the same, but they don't have to be totally untouched.
[+] [-] sgslo|9 years ago|reply
Two points of feedback:
(1) The idea of the product is clearly conveyed, but I'm confused on exactly how it works. The landing page mentions that title tags, headlines, meta-tags, etc get tweaked - exactly how is this done? Do I have to manually enter a bunch of alternative text, or are you using a big fancy thesaurus to switch out some key terms?
(2) How do you evaluate performance of the product? Solely through click rates, or by search rankings? How often do google search results get updated? In short, how do I know the product is working?
[+] [-] dillonforrest|9 years ago|reply
(1) We use both humans and software to generate experiments. For customers, it's completely automated.
(2) We look at all primary search metrics (clicks, impressions, CTR, and rankings), with clicks being our main metric. Search results get updated at a pace determined by Google's crawl rate, which varies per site depending on multiple factors including domain authority. We use bayesian structural time series and negative binomial regression models to measure impact and statistical significance to power our data-driven SEO recommendations.
[+] [-] birken|9 years ago|reply
However, I think a critical aspect of SEO is thinking about an entire site holistically. Not only because certain signals are site-wide, but because a key aspect of SEO is deciding what pages of your website are "good" for SEO and which ones aren't, and then focusing on making the "good" pages better and not worrying about the "bad" pages. Good and bad in quotes because it is often quite a bit of an art and not a science.
How does RankScience play into this? You've nailed the on-page stuff but is there any world in which RankScience is able to talk about a site holistically and recommend which types of pages and content seem to be working most effectively (and maybe even suggesting pages that could be removed/de-indexed)? Or do you leave that to SEO consultants and you just nail the hell out of the on-page stuff.
1: https://www.thumbtack.com/engineering/seo-tip-titles-matter-...
[+] [-] dillonforrest|9 years ago|reply
> How does RankScience play into this?
That's a really great point. Right now, our software either focuses on only pages which are good for SEO, or it runs less frequent experiments on the "bad" pages. We've had a few success stories of turning "bad" pages into good ones and having them become big revenue generators for our customers, but that's not common as I'm sure you can imagine.
[+] [-] BickNowstrom|9 years ago|reply
From https://support.google.com/webmasters/answer/7238431?hl=en
> Best practices for website testing with Google Search
> The amount of time required for a reliable test will vary depending on factors like your conversion rates, and how much traffic your website gets; a good testing tool should tell you when you’ve gathered enough data to draw a reliable conclusion. Once you’ve concluded the test, you should update your site with the desired content variation(s) and remove all elements of the test as soon as possible, such as alternate URLs or testing scripts and markup. If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if you’re serving one content variant to a large percentage of your users.
Next to this, the advice is to use rel="canonical" to avoid duplicate issues with Googlebot crawling your variations. When using rel="canonical" this should not show you how a variation influences ranking.
> If you’re running an A/B test with multiple URLs, you can use the rel=“canonical” link attribute on all of your alternate URLs to indicate that the original URL is the preferred version. We recommend using rel=“canonical” rather than a noindex meta tag because it more closely matches your intent in this situation.
[+] [-] ryanb|9 years ago|reply
Also, our goal is not to deceive Google in anyway - a lot of our tests are related to increasing CTR (which is testing humans) and on-page times. (again testing humans) Overall we're trying to make pages better according to Google guidelines -- which leads to a better experience for users.
Some more explanation of how it works here: https://www.rankscience.com/how-it-works
[+] [-] 7cupsoftea|9 years ago|reply
[+] [-] dillonforrest|9 years ago|reply
[+] [-] killion|9 years ago|reply
[+] [-] dillonforrest|9 years ago|reply
[+] [-] dillonforrest|9 years ago|reply
[+] [-] saycheese|9 years ago|reply
Why A/B testing and not a more complex statistical hypothesis testing method?
[+] [-] mhoad|9 years ago|reply
[+] [-] eonw|9 years ago|reply
Also what do you consider success? what if one title/meta combo gets a better rank but fewer clicks(ie; the bots like it but the users dont)? which one would you keep? I know many people like to brag about how many rankings they have but as an SEO for over a decade, i think traffic count is still king. in that id rather rank 11th for an extremely high volume term, then rank 2nd for a low volume one.
another thought... a blackhat site gets banned halfway through one of your tests and you move up based on no actions you took. thus you would get a false positive, no?
[+] [-] inthewoods|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] opendomain|9 years ago|reply
I have a proposal for you - please respond via my email
Ric AT myUserName
[+] [-] rloc|9 years ago|reply
[+] [-] hartator|9 years ago|reply
How do you check the effectiness of a SEO change? You check for Google rankings or traffic?
[+] [-] ryanb|9 years ago|reply
More clicks is obviously the ultimate goal, but sometimes we also have significant experiments where rankings or impressions going up are the key metric
[+] [-] franciscassel|9 years ago|reply
What are some of the specific types of automated tests that you run?
[+] [-] ryanb|9 years ago|reply
* CTR on titles/meta descriptions * Any HTML changes * Design changes * Time on site
[+] [-] gukov|9 years ago|reply
Let's say you want to test a new title. You collect stats for a week, change the title, collect stats for another week. The two datasets are then compared. Am I close?
Does it mean that you can't AB test in parallel? If so, it's not optimal for time sensitive stuff like breaking news.
[+] [-] ryanb|9 years ago|reply
Depending on how many pages you have, we can run many tests in parallel.
[+] [-] coolswan|9 years ago|reply
[+] [-] ryanb|9 years ago|reply
Site performance is always relative to your competitors. Optimal server response time is around 200ms, and that's what folks should strive for, but I've seen sites have really slow pages and still get lots of traffic.
[+] [-] 0bfus|9 years ago|reply
[+] [-] dillonforrest|9 years ago|reply
[+] [-] ijafri|9 years ago|reply
[+] [-] sixQuarks|9 years ago|reply
[+] [-] Mizza|9 years ago|reply
[+] [-] dillonforrest|9 years ago|reply
[+] [-] gargarplex|9 years ago|reply
[+] [-] ryanb|9 years ago|reply
[+] [-] jjyoung135|9 years ago|reply
[+] [-] dillonforrest|9 years ago|reply
[+] [-] ortekk|9 years ago|reply
[+] [-] ryanb|9 years ago|reply