top | item 5993914

What is A/B Testing?

220 points| paraschopra | 12 years ago |visualwebsiteoptimizer.com | reply

109 comments

order
[+] modernerd|12 years ago|reply
What motivates web designers to tell stories by repurposing the scroll mechanism instead of, say, using video? This type of "scrollmation" feels incredibly clunky to me. I have never got to the end of one.

If you need that much animation on a page to tell your story, isn't it a good sign that video might be more fit for purpose? Video was designed for storytelling, it will work on most devices, and it has well-established UI and conventions, such as a play button and progress bar.

If you're going to provide a play button in your scrollmation to automatically advance the action anyway, you've already recognised how irritating it is to manually page through an animated story by scrolling or swiping. So why then go to the trouble of creating a scroll-based animation that repurposes a 20+ year old convention and attempts to reinvent a tiny proportion of what video has to offer?

Is it just for the novelty value?

Is it because it makes you feel smarter?

How is it better for users trying to understand your story?

I'd love to hear from anyone who's considered both video and scrollmation and chosen the latter.

[+] huhtenberg|12 years ago|reply
There's a good reason. Two of them, actually.

This particular animation in a video format would take 10 seconds of running time. It would also take few seconds to load and initialize whatever video playing widget it'd be using. That would make it for an oddly short video and one of those "What? That's it?" situations. So it simply wouldn't work as the only content on the page. OTOH, if you present it as an interactive animation, it will take users longer to work their way through it and so it would feel more substantial.

Secondary reason is that it makes the page more memorable and helps it stick in more people's heads than some yet another embedded video. The goal of the page is to engage people. This is the reason why you'd want to have things presented rather than just thrown on a page in black and white Times New Roman / 12px. You want your page to engage and clicking on Play Video is not engagement, it's a reflex. So by forcing visitors to do something out of their routine the page, effectively, makes them to pay more attention than usual.

[+] reedlaw|12 years ago|reply
I've closed websites with nothing but a video on the landing page because I didn't want to wait for a video to load. This landing page, on the other hand, is the best product presentation I've ever seen. It's simple and clear and doesn't have boring narration to waste my time.
[+] dgunn|12 years ago|reply
Does anyone else just want a written description of things? Make it look pretty. Create beautiful pictures/characters throughout the page. I don't care. I just want a good explanation of what a service offers.

- It's user-friendly.

- Compatible with everything.

- I can read as much or as little as I want.

- If I want to re-read something, I know exactly how to do that and it doesn't involve clicking a play head trying to find the place it was in.

I feel like people keep trying to bring back reincarnations of flash interactivity and since it uses html5 and other buzzwords, it's ok. Well, it's not.

Every time someone "fixes" the experience of what should just be reading online, they make it much more cumbersome and annoying. One day everyone will look back at what people did with html5 and laugh the way we all look back and laugh at flash intros and playing music on load.

[+] gojomo|12 years ago|reply
Video forces me to go at someone else's pace -- the creator or expected 'average viewer' -- which is often too slow.

It also makes skipping or rewinding a matter of finely-navigated clicks into tiny target areas, whereas the best hardware scrolling controls offer giant, forgiving action-areas yet that still enable nearly pixel-level resolution and fine steps of speed/acceleration.

Depending on your device, scrolling in many directions might be very, very easy. Large trackpads, the Apple 'Magic Mouse', and most touch screens can make scrolling far more easy, adaptive, and intuitive than even 'forward' and 'back' buttons.

I tend to think people most peeved by scrolling as a major navigational technique (in both space and time) are still using tiny trackpads, mice with nubby scroll wheels, or even (shudder) ThinkPad-like joysticks or keyboard arrows. Stone knives and bearskins!

Does this apply to you?

[+] GoldfishCRM|12 years ago|reply
I think scrolling is better in the way that it does not aquire audio and user can controll the speed of information. This part is boring and I get it, lets skip to the next part. Video has no way for that.
[+] karma_fountain|12 years ago|reply
I'd usually agree with the scrolling mechanism comment, but this time it's very well done. The initial page clearly explains that you can scroll down for more content, or click play, or click a button to get to the next page. The story and graphics are fun and keeps me interested.
[+] run4yourlives|12 years ago|reply
What motivates web designers to tell stories by repurposing the scroll mechanism instead of, say, using video?

I can only speak for myself, but I rarely, if ever, watch website videos, especially at work.

I made it through this entire thing by scrolling.

While I understand video is sometimes the best way to convey an idea, I fricken hate it. It would need to be something very, very intriguing for me to watch. 99% of websites selling things don't fall into that category.

[+] nutanc|12 years ago|reply
Maybe there should be an A/B testing version. One with video and one with animation :-)

Personally, I liked the animation.

[+] paraschopra|12 years ago|reply
I'd say it is mostly novelty. Video could do the same job, maybe even slightly better. We have considered adapting it into a video as well, but then everyone does that. Second important factor after novelty was integration of signup form and other interactive elements within the story. With video, that's not possible.
[+] nwh|12 years ago|reply
I've made this sort of stuff before at request, and I can't stand it either. It quickly degrades the one thing that users can expect to do; control the vertical position of the page.
[+] mbesto|12 years ago|reply
With stuff like this I always refer to how my mom uses computers. (Note - This a person who thought when you copied text on her iPhone that the text magically went into her finger, only to have to click paste with another finger so the text didn't get lost in her finger...)

I agree. There is no way she would know to scroll properly in this interface and in most cases would get confused and give up. The objective of the presentation is to make it easy for someone (like my mom) to understand A/B testing. Make it a video, or at least provide a link to a video version.

EDIT: This akin to Amazon's usability - http://www.nngroup.com/articles/amazon-no-e-commerce-role-mo... i.e. once some behavior is establish on your site, keep it that way, but doesn't mean you need to emulate it.

[+] brandoncapecci|12 years ago|reply
Lots of personal preferences in this thread. As an A/B testing company - they should have A/B tested a scrolling site against video and then been transparent about their results.
[+] pelario|12 years ago|reply
I'm not a "UI" guy, but I loved it. For me, it feels like a presentation, but the transition from slide to slide make it feel like a video. My 20 cents.
[+] bjz_|12 years ago|reply
A back/forward button would be enough for me. The scrollimations always work confusingly on platforms with smooth, momentum based scrolling (like OSX).
[+] ulisesrmzroche|12 years ago|reply
These are questions worth A/B testing though. I'm on team video, but if anyone has a link to the research, let me know.
[+] Sujan|12 years ago|reply
> What motivates web designers to tell stories by repurposing the scroll mechanism instead of, say, using video?

Control for the users.

[+] dirkgently|12 years ago|reply
If they had used video, they would have lost me in 5 seconds.
[+] L4mppu|12 years ago|reply
Also this scrollmation pages load very slowly.
[+] petenixey|12 years ago|reply
I love it that you've produced this and I think the simplicity of it is wonderful but I fear the story could leave someone a bit confused.

It's confusing first of all because you've chosen a product which people don't generally sell online - fish. If you'd chosen bikes or clothes it would have made sense.

The other thing is on the "play" version - it takes about half the animation before you even get to the online bit and when you do, the illustration of how A/B testing actually works only takes up a tiny bit of real estate.

For my money I'm not convinced that the offline analogy really adds to it. You know your market better than me (or anyone else here except Pete and Dan) but I feel like you could jump straight in with a story about an online store.

"Mike sells bikes online but he's not sure which ones to put on the front page. Will he get the most sales from showing someone bikes on the frontpage or showing them chainsets? He doesn't know and he finds it hard to read his analytics to figure out which one is best. VisualWebsiteOptimiser does all this for him..."

[+] paraschopra|12 years ago|reply
Thanks for the feedback. Yep, we had debated whether fish is the right product to be sold online. We concluded that probably the choice of what product is sold is not as important as demonstrating the actual confusions during the sales process (both online and offline). We considered selling books, cakes and some other products but with fish, the options were limited and very clearly distinct.
[+] speeder|12 years ago|reply
Beside the stupidity of selling fish online...

You CAN do A/B testing of sorts on the physical store too. This might break the attention of people that know that.

I guess the intention were good, and the drawings are of good quality, but otherwise the execution is just terrible.

[+] vinhboy|12 years ago|reply
Agreed.

First off, I was distracted by that slideshow because I kept thinking, "Who the f sells fishes online?"

Second, I felt like there were two points in the video because of the whole offline analogy. 1) Selling online is better than offline 2) A/B testing makes you more money

[+] mtrimpe|12 years ago|reply
Really love the animation, but I can't help but wonder if A/B testing couldn't be explained more powerfully.

Within the example of the store for example, the real pain A/B testing addresses is cyclical variation leading to incorrect conclusions.

Working from that you could've shown the story of:

* the store owner rearranging his display and seeing sales go up

* but then the rest of the year sales actually going way down

* and when he changes things back to the old layout sales went waaay up again! How can that be?

* Cue image of cyclical charts and the facepalm of realizing it wasn't his redesign at all!

* Now ... on his website he can send visitors to different versions at the same time!

* So he'll never have a year of bad sales again!

[+] ad93611|12 years ago|reply
The animation is very good. Could you explain a bit about how this was done? Did you have to write the code or is there a tool that can generate a page like this?
[+] bhaisaab|12 years ago|reply
Heya, this was done by one of our talented interns Kushagra A. (https://twitter.com/kushsolitary). We did not use any parallax page making tools, it was written using HTML5, CSS3 and some js library.
[+] huhtenberg|12 years ago|reply
Stars of David coming out of the "A/B-Testing" explosion - I bet it's some sort of clever innuendo, but it just seems to escape me :)
[+] goatcurious|12 years ago|reply
Certainly not a clever innuendo. Looks more like an oversight. There isn't much awareness about Star of David in India.

It is the easiest star to sketch -- even my teacher in elementary school used to use the same star in my grading.

[+] avk|12 years ago|reply
Likewise, I found it distracting and off-putting.
[+] felideon|12 years ago|reply
Maybe the shop owner sells... gefilte fish. :)
[+] zachlatta|12 years ago|reply
Very neat. Hate the loading screen though. Having to display a loading screen is almost as bad as using flash. Maybe load the assets in the background?
[+] paraschopra|12 years ago|reply
Actually, we load first 4 screens at first go. Rest is lazily loaded.
[+] rajington|12 years ago|reply
Beyond the comments over the presentation style, the metaphor, and browser compatibility I want to see more about your specific app. I like how you did the VWO interface on top of his website, you could easily change the line graph section to show the "Intuitive Reports" feature. Line graphs are easy enough to understand so by including more of your value add features it will still stay simple enough.

I know it isn't specifically A/B testing related, but throwing another shoutout to your app at the end like "heatmaps too!" might be nice. It seems people that would look at this slideshow might not be familiar with any web analytics services, and it seems you guys offer many services.

[+] paulodeon|12 years ago|reply
Scrolls Down...Meet Bob...Goodbye Bob...Clicks back
[+] IanCal|12 years ago|reply
When say "Bob knows which version brings him more sales", I'm curious about how you extract the relevant information from someone to know if it's a statistically significant change in a user friendly way. Do you find people understand this well?

What about multi-variate testing? How do you explain the increase in the number of visitors needed to make a decision?

Finally, how do you reconcile giving up-to-date information without compromising on the results? (keeping an experiment going until it's statistically significant is something I can see users doing)

[+] nixarn|12 years ago|reply
A bit off topic. But I've had problems with the visualwebsiteoptimizer, not as a customer but as a user on websites using their system. At two different occasions websites have had problems to render, no matter how many times I try to re-load them and wait. I've then checked the webdevtools to check what's wrong and it's been the websiteoptimizer's script that's been blocking the whole site.
[+] philfrasty|12 years ago|reply
I personally find it „too much effort“ to scroll down all the way. Nice story, but make it somehow shorter and choose a storyline that is familiar to people so they only have to concentrate on the A/B-aspect because otherwise the story is distracting. Would be great to get some feedback how the site is doing (compared to previous versions).
[+] unknown|12 years ago|reply

[deleted]

[+] retube|12 years ago|reply
IE8: "There is a problem with this website's security certificate".

Also doesn't render at all well...

[+] alexcroox|12 years ago|reply
Nice work! We did something on a smaller scale at the bottom of our homepage (with live stats on users choices)

http://sidigital.co/

(you'll have to wait for the liquid to reach there to see the effect of your choice)

[+] chinchang|12 years ago|reply
checked this a while ago. Clean work!
[+] TamDenholm|12 years ago|reply
I hate to be "that guy" but you've got a spelling mistake. "Fishes sell volumes". The plural of fish, is fish. "Fishes" would be the possessive.
[+] zekenie|12 years ago|reply
Are there jewish stars that appear when AB testing is introduced? No issue with this, it just seems a little odd. It reminds me of Hannuka at my parents house! :)
[+] dbg31415|12 years ago|reply
Sorry, why does this warrant an animated deck? Sheesh, next up, "How to use gravity to fall over."
[+] alxbrun|12 years ago|reply
Just when you thought Powerpoint was finally dead at last, it comes back disguised in scrolling.. Damned !
[+] fny|12 years ago|reply
While I appreciate the responsiveness, the page is obnoxious to navigate on a mobile device.