AWS Device Farm

[stickydink]

> unlimited testing for a flat monthly fee of $250 per device

Renting remote-controlled Android devices for $250 a month, is this even remotely worth it? There aren't many devices that wouldn't pay for themselves by the end of the 2nd month...

[hexedpackets]

You don't pay for a specific device. You pay for unlimited runs using 1 device, but that device could change each run. So its sort of akin to leasing without an option to buy.

[smackfu]

Generally the value proposition of these kind of services is that you can try to reproduce an issue that only shows up on a particular piece of hardware and OS. For instance, someone with iOS 7 on an iPhone 5, which you can't even buy new right now. Or a particular Samsung phone that wasn't very popular and was only sold for a few months.

[oblio]

But then you have to manage the devices yourself :)

[davnicwil]

Came here to comment this myself - was wondering if I read it right.

I don't understand the value proposition here at all. If you're subscribing for unlimited use then more likely than not you're all-in and are going to be testing for more than 2 or 3 months - probably continuously for years in fact.

At that scale doesn't the up-front cost and commitment of buying/managing the device make much more sense? What have I missed?

[jastanton]

Correct me if I'm wrong but does this feel pretty expensive?

> Pricing is based on device minutes, which are determined by the number of devices you use and the duration of your tests. AWS Device Farm comes with a free tier of 250 device minutes. After that you are charged $0.17 per device minute. As your testing needs grow, you can opt for our unmetered testing plan, which allows unlimited testing for a flat monthly fee of $250 per device.

[aalbertson]

I used to work for a mobile services company. We used technology similar to this. It is NOT cheap, nor easy to build. Some places give you a web interface that connects remotely to physical devices on the other side and some nifty robotics to control the phones/push buttons/etc... This is down right affordable!

[Someone1234]

Strange, I was actually thinking the pricing was very reasonable. 250 minutes free is pretty generous and will likely be all many small developers need.

17c/minute/device is fine after that since you'll only likely need 20 minutes to test your average app against each device (and they only seem to support a handful). So $10~ to test your apps against a bunch of Amazon devices (ignoring the free 250 minutes), is fine by me.

The only reason I might look elsewhere is that other services offer a more broad range of Android devices, not just Amazon Fire stuff. So you can spin up a Galaxy S6, Nexus, Fire Tablet, etc all under one roof.

[lubesGordi]

I'm guessing people will use this right before they push their first beta version, just as a quick sanity check against devices they don't already have. Most compatibility issues will probably arise from supporting pre-ICS devices and those are basically all under $250. Besides that, mostly what developers are going to be concerned with is how the app looks on different devices, having taken care of functional issues by developing on slower/lower memory devices in the first place.

[middleman90]

Yes, but consider that is not a virtualization, is a proper device

[Artemis2]

Amazon is coming for Google's [Cloud Test Lab](https://developers.google.com/cloud-test-lab/) and [Nativetap](https://beta.nativetap.io/).

[ColinDabritz]

On a brief look, it appears that all of these choices (Including Amazon's new one) only support Android devices and not iOS?

I believe Xamarin Test Cloud includes iOS, although I'm not certain if it integrates outside of the Xamarin framework. http://xamarin.com/test-cloud

Seems like a gap they should all be trying to fill.

[yla92]

Pretty interesting. Now, we have more choices. The other day, I've found this project on Github called OpenSTF[1] which allows you to set up your own devices labs. It's pretty interesting and I've got to try yet. Generally, I'd prefer to set up my own device labs rather than testing with 3rd party services. It depends on your target market. For us, the apps we made are target for the local market (people and devices). And the phones sold in our market are different with the phone sold in States and I don't expect to have some China devices to be available on 3rd party services.

[1] : https://openstf.github.io

[shimo5037]

Hey that's ours :) Right now it's heavily focused on remote control rather than test automation, so it would not be a direct replacement yet. We just open sourced last week, though, so perhaps we can start looking into that in the near future after the initial batch of bugs is fixed.

[martin_tipgain]

Check us out at https://www.testmunk.com/ we do have iOS and Android support :) Don't hesitate to reach out if you have any questions.

[Someone1234]

That Flipboard animation/video (showing the report I guess) is super distracting/annoying. I literally deleted it with developer bar just so I could read your copy in peace.

Also calling a $800 package with 25 hours and 5x accounts the "starter" level seems a little odd. It is your lowest paid tier, but there's nothing starter-y about it. It is your normal tier. The free tier is the starter tier.

I have no idea what a "project" is. Seems like an arbitrary limitation. What do you care what code I am running with my hours? It is none of your business. Seems like you created the concept just so you could "sell us more of it" with the pro tier.

Also pro coming with only 5 users is odd. If I am paying for 100 hours/month of usage, that just seems like an annoyance. You'll force businesses into only giving out access to a handful of people rather than every single developer (and if so few people use it, the company might decide the pro tier doesn't make sense and downgrade to "starter").

Why is it so hard to find a list of devices you currently support? I actually googled it and still cannot figure out what devices your service offers. That seems like absolutely vital information for anyone even considering you.

Overall I like the service, but I like Amazon's model more. Just give me a cost/minute and get out of my way.

PS - You absolutely CAN justify charging more than Amazon. iOS support alone is a massive value add. I just like their pricing model, I am not proposing you charge 17c/minute/device.

[techaddict009]

$ 790/mo arent you guys too costlier compared to AWS?

[RyJones]

The most crucial bit for me, which I don't see on the one-pager, is the ability to test devices in-market on carrier networks. Testing phones for Korean or Japanese carriers (or British, or Brazilian) on simulations or using GSM roaming is not good enough.

[Wonnk13]

Didn't Google introduce something similar to this at IO a few weeks ago?

[yla92]

Yes, Google announced Cloud Test Lab[1] in this year I/O

[1] : https://developers.google.com/cloud-test-lab/

[saurik]

I am always really curious to know how these kinds of services deal with security issues: they don't get to control the security of the devices they are running, and often will only even sort of have access to reflashing them; how do they deal with someone testing an app with a kernel exploit, installing a persistent backdoor, and then watching what everyone else later using that device is testing?

[nickpsecurity]

I had an idea, more a need, for something like this many years ago. It came back to my mind when OpenBSD needed funding partly due to all the different pieces of hardware they test on. Wouldn't it be nice for portability-focused projects to be able to pool together resources on hardware to cut down on it?

Anyway, awesome to see AWS doing it in practice. As usual, it will be more interesting to see what happens when competition turns up. Cloud space has more innovation and cut-throat competition than many IT sectors. Can't wait to see what the competition costs. ;)

[jakozaur]

I wonder if they add IOS at some point. Maybe it's the Amazon way of doing. Start with MVP and iterate.

[varelse]

So basically these guys?

https://en.wikipedia.org/wiki/Keynote_DeviceAnywhere

[ex3ndr]

Why Amazon and not solutions from other mobile-focused platforms?

[mwcampbell]

Care to list any of those alternatives? I imagine a lot of companies will use Amazon's device farm simply because Amazon is so well-known.

[mwcampbell]

Curious about what the built-in, no-scripting-required test suite can do. I wonder if this is what the app reviewers for the Amazon Appstore have been using.

[fudged71]

I was hoping this would be a Raspberry Pi farm!

[fbaptista]

ohhh :) take a look at another monkey at www.monkop.com