Why would I want to pay you $40/mo, if I'm paying $3/mo for shared hosting that I know can handle 250 users?
And its the same way through out. Even at the highest level, it'd be cheaper for me to lease a second dedicated server to spread the load, than it is to pay you for your service.
And I don't think this is something that should be a monthly service. I only want to check my load once in a while to see if its doing fine, so I think by asking so much money on a monthly basis, you are driving users away. I think you'll be better off doing it as one off results. i.e. the $499/mo option, should be a $29.99 one time fee for 3 reports
First - the people on $3 /mo shared hosting are not in the target market for this service. If you are on shared hosting, you don't need loadtesting.
Second - I think subscription is fine. As a subscription, I recognize that I -should be- load testing on a regular basis. As a-la-carte, I am much more likely to say "umpteen dollars?I can do my own loadtest for cheaper then that."
But, seriously, this should be strictly an opt-in service. Only if there is a loadimpact.txt on the server, you run your tests. Otherwise - refuse and explain that the server admin has not consented to the stress testing using your service.
I'm sure you have some DoS provisions in place, but in the end it really boils down to the fact that if you screw up, then it's my server that gets DoS'd.
Anyone with Pylot (or any other load testing tool) could do this at any point they wanted to your server from the comfort of their own home. Shall we boycott those next? And this site even seems to have denial-of-service protection of different types that you don't get with those DIY tools.
I love being able to test this out right away, so I'm in favor of keeping it pretty much how it is. I was looking for this exact service two weeks ago and was surprised I couldn't easily find someone who did this with transparent pricing/plans.
Keep the transparency and ease of trying it out-- there are too many companies in this space saying "Call us for a free quote" with "account execs" when this doesn't need to be that complicated of a service.
You passed the first test: I asked myself "wait, how is this not a one-stop-shopping service for aspiring, technically incompetent DOS attackers"? And I found your FAQ and answered that question inside of 45 seconds:
You failed my first test. I tried to load test my start-up - www.trailbehind.com - using your free service. However, it said I had already tried to load test my site and it failed.
I went through your registration process, and got nothing. Very disappointing - the type of thing that will make me never look at your website again and scoff about it to my friends should they bring it up.
And a couple other nits:
1) Your registration page where I choose my plan type is not intuitive. Maybe it's because green is not a good color to call attention to stuff.
2) Your Register and Cancel buttons should be different colors. I clicked the wrong one and had to redo my registration.
3) Don't ask for number of employees and industry. They're not required and they will hurt your conversion rate slightly. And it indicates to me you might sell my data.
We have a ton of security features to limit abuse of the system. A site that cannot handle the impact of a single basic (free) test can be DOS:ed once, yes. But we have a limit on how often we will generate load to (or from, really) individual IP addresses, so the second or third time in a day someone tries to "load test" the same site, they will be denied their test. This means you can at most DOS-attack a small site a few minutes per day. We hope it will be enough to make would-be abusers go somewhere else - i.e. not worth their time.
Nice effort! After a few weeks of mediocre "review my site" submissions here, it's nice to see something come through that I would actually use!
You did a good job of making the reports and graphs pretty while still keeping a clean feel to the whole site. There seems to be a good amount of polish and attention to detail there too. I don't feel scared to click anything while the test is running for fear of breaking something or redirecting me to a FAQ page and canceling the test.
I tried to test my site (NSFW: http://fapseek.com) and got the message "This configuration contains addresses that has been tested too many times the last 24 hours." I assumed someone else had tested the site and that I would be able to test after logging in and verifying ownership of the domain. Even after creating an account, logging in, and creating the loadimpact.txt file, I'm still getting the same error message about my address having already been tested too many times.
That's our somewhat restrictive security features that notice some object being loaded as part of your webpage, an object that other sites also use. For example, many sites load urchin.js from google-analytics.com. This means that the daily load our system allows itself to subject google-analytics.com to will quickly be reached. Anyone trying to start a new test for a webpage that loads urchin.js will then be denied to run their test.
We have a list of URLS/sites that are exempted from these checks, but this list needs to be populated. So far we only have a few entries on it (like google-analytics). It seems your site loads things from googleapis.com so I added an exception for that address now. Try again and see if it works.
We probably need to work on the information to users whenever a test is denied also. If you know what the offending URL is, you can remove it from your load script and run the test without it.
Maybe I'm just retarded, but I couldn't find the answer to my question on your site. Does your service just pound the exact page that's targeted, or does it follow links, submit forms, etc to try to simulate actual users?
The simplest test you can run will just load a singe page, and all its dependencies (images etc) over and over again. However, if you register an account you can create your own load scripts that load different pages, with pauses in between. Paying users also get access to the recorder feature which allows you to record a session that you run in a new browser window - you surf around your site and click on things, POST forms, etc and when you're done you have a load script automatically generated that will let your simulated clients do the same thing.
We don't currently have any "crawler" mode though, where the system tries to follow links etc on the site automatically.
I see a potential legal problem with the way the stress test is "authorized". The thing is that web server access (putting the "loadimpact.txt" file in the server) is not exactly the same as web site ownership.
My paranoid mind can think of an scenario like this (INAL so I maybe be off): a web master asks for the load test, this is done and the site goes down for a while, incurring a monetary loss (loss in sales for example). The owner sues the testing company for a 'DoS' and the testing company has no way to prove that it was authorized (what text file, there's none!).
I personally wouldn't do a stress test on a server without written authorization of the owner. The way I would implement this on an online service is by asking the target web site and the email address, where this address has to be in the public whois information. Then I'd send a special message to that email address and by replying the owner gives authorization (this way I'd have something from the official email address). This is not a 'written authorization' but comes as close as it can and the whois email address is used for domain transactions etc so it has the same level of protection/legal procedure behind.
Of course you should permit to run such tests only if you are sure you are dealing with site owners or their representatives. You can use the system that is used in Google webmasters tools - users can confirm their ownership by adding specific document or meta-tag. Also - registration system's usability is not perfect. But the idea is great - I would use this service.
Yeah, this is something we have thought about also. We already have this security feature that uses a special file (loadimpact.txt) on the target system(s) to verify that it's OK to run large-scale tests. We should probably improve on that and allow people to bypass the other checks if they have this file. It is the safest way we have to verify that a test is legal, so we should give it more priority before other checks.
You guys should offer a daily or limited time use rate. I would think many sites dont need to load test every month. But instead when they are first building an infrastructure or making upgrades.
One of the sites I am working on now is moving to a new codebase. We wanted to load test for a few hours but found everything we looked at to be too expensive.
That is actually on its way also. We're going to offer a "single test" option, where a user gets to use the system for an hour. But maybe we should make it 24 hours instead?
It depends in what way you mean. It generates traffic and measures performance in a similar way as ab/httperf. The biggest difference is that it is an online service, which means you don't need to own any servers yourself to run ab or httperf on. You don't have to install or configure ab/httperf, and you get results nicely formatted as graphs with no extra work on your part and test configurations and results are always available to you, on the web.
If you have the time, knowledge, interest and a server to spare, then ab or httperf (or I would recommend the Grinder - http://grinder.sourceforge.net/ ) is definitely worth a try however.
Thank you, and while I'm at it: thank you very much everybody for all the great feedback. It is very appreciated!
The session recorder is pretty cool, we think. We have managed to make it zero-configuration (i.e. no user configuration necessary) while still being able to handle client-side javascript. There are other HTTP session recorders out there, but to be able to handle javascript they're usually implemented as a true HTTP proxy that you have to configure your browser to use, or they're a browser plugin that you have to download and install.
The recorder outputs a load script that you can edit afterwards, if you want. However, the load script is currently more of a "list" of things to load, with pauses in between. It has no logic, conditional statements etc. and does not support parameterized data. This is something we intend to change soon though. It just needs some thought so we don't open up new possibilities for abuse/overuse of resources when we let users execute their own load script code on our systems.
The graph is pretty flat, which indicates the test is not really putting any stress on your server. The load added does not make much of a difference for improvingtheweb.com. One thing we need to put more effort in, I think, is to communicate these things. What can you learn from a graph, what are the prerequisites for getting good test results (in this case, more load would be a good thing).
I think there is a lot we need to do to explain things better. That is the most difficult part of all.
This is a good start, though not as comprehensive as we're used to. Currently we use WAPT which is significantly more comprehensive while still maintaining the ease of use that most other comprehensive tools lack.
That said, it would be great to incorporate some of their features such as different scenarios, load patterns, particular series of pages, random, etc. It would be great to not have to have a dedicated resource to run the stress tests, it's also tricky to know if the local bandwidth can properly emulate x number of machines.
5,000 users is what one single instance of the non-threaded load generator can simulate today. We have some optimizations we can make that should improve this though, and we are also going to implement multi-source/multi-program load generation that means we can distribute the load generation for a single test over several processes and several load generator hosts. Using only our own infrastructure I think we can scale the system quite a lot, but obviously, if we want to run really large simulations with several hundred thousand, or millions, of simulated users, then we need to buy cloud server capacity.
I'm interested in how many paid registrations posting here on HN resulted in. It seems like a pretty positive response here and a great tool! Great work!
Thank you. We have had a ton of visits and a good number of registrations, although only a handful of paid accounts as of yet. But that's OK, it's natural that people want to try before they buy. We hope they'll convert once they have tested the service a bit.
If you ask me, however, I'd say that the feedback we are getting here is what's really invaluable. It will help us improve the service a lot.
No, it is most likely because your site is loading some object - like urchin.js from google-analytics.com - that a lot of other tests have been loading also. That means that our security checks will think someone is trying to DOS google-analytics.com and so stop your test. We have a list of exclusions that the system should not bother checking (such as google-analytics.com) but the list is not complete so people run into this problem frequently right now.
I am going to fix so that we inform people what the offending URL/object is (rather than just saying "one of the systems involved in the test..."), so they can remove it from their load scripts (if you're a registered user you can edit your load scripts before starting a load test) and also contact us and ask us to put new systems into our exclusion list.
[+] [-] vaksel|17 years ago|reply
Why would I want to pay you $40/mo, if I'm paying $3/mo for shared hosting that I know can handle 250 users?
And its the same way through out. Even at the highest level, it'd be cheaper for me to lease a second dedicated server to spread the load, than it is to pay you for your service.
And I don't think this is something that should be a monthly service. I only want to check my load once in a while to see if its doing fine, so I think by asking so much money on a monthly basis, you are driving users away. I think you'll be better off doing it as one off results. i.e. the $499/mo option, should be a $29.99 one time fee for 3 reports
[+] [-] teej|17 years ago|reply
First - the people on $3 /mo shared hosting are not in the target market for this service. If you are on shared hosting, you don't need loadtesting.
Second - I think subscription is fine. As a subscription, I recognize that I -should be- load testing on a regular basis. As a-la-carte, I am much more likely to say "umpteen dollars?I can do my own loadtest for cheaper then that."
[+] [-] huhtenberg|17 years ago|reply
But, seriously, this should be strictly an opt-in service. Only if there is a loadimpact.txt on the server, you run your tests. Otherwise - refuse and explain that the server admin has not consented to the stress testing using your service.
I'm sure you have some DoS provisions in place, but in the end it really boils down to the fact that if you screw up, then it's my server that gets DoS'd.
[+] [-] thorax|17 years ago|reply
I love being able to test this out right away, so I'm in favor of keeping it pretty much how it is. I was looking for this exact service two weeks ago and was surprised I couldn't easily find someone who did this with transparent pricing/plans.
Keep the transparency and ease of trying it out-- there are too many companies in this space saying "Call us for a free quote" with "account execs" when this doesn't need to be that complicated of a service.
[+] [-] copenja|17 years ago|reply
See here: http://loadimpact.com/forum/viewtopic.php?id=7
[+] [-] a-priori|17 years ago|reply
[+] [-] volida|17 years ago|reply
[+] [-] mechanical_fish|17 years ago|reply
http://loadimpact.com/forum/viewtopic.php?id=7
[+] [-] andrewljohnson|17 years ago|reply
I went through your registration process, and got nothing. Very disappointing - the type of thing that will make me never look at your website again and scoff about it to my friends should they bring it up.
And a couple other nits:
1) Your registration page where I choose my plan type is not intuitive. Maybe it's because green is not a good color to call attention to stuff.
2) Your Register and Cancel buttons should be different colors. I clicked the wrong one and had to redo my registration.
3) Don't ask for number of employees and industry. They're not required and they will hurt your conversion rate slightly. And it indicates to me you might sell my data.
[+] [-] rlonn|17 years ago|reply
[+] [-] davidw|17 years ago|reply
[+] [-] jakewolf|17 years ago|reply
[+] [-] jasonkester|17 years ago|reply
You did a good job of making the reports and graphs pretty while still keeping a clean feel to the whole site. There seems to be a good amount of polish and attention to detail there too. I don't feel scared to click anything while the test is running for fear of breaking something or redirecting me to a FAQ page and canceling the test.
I look forward to seeing how this pans out.
[+] [-] jey|17 years ago|reply
[+] [-] rlonn|17 years ago|reply
We have a list of URLS/sites that are exempted from these checks, but this list needs to be populated. So far we only have a few entries on it (like google-analytics). It seems your site loads things from googleapis.com so I added an exception for that address now. Try again and see if it works.
We probably need to work on the information to users whenever a test is denied also. If you know what the offending URL is, you can remove it from your load script and run the test without it.
[+] [-] chops|17 years ago|reply
[+] [-] rlonn|17 years ago|reply
We don't currently have any "crawler" mode though, where the system tries to follow links etc on the site automatically.
[+] [-] lazyant|17 years ago|reply
My paranoid mind can think of an scenario like this (INAL so I maybe be off): a web master asks for the load test, this is done and the site goes down for a while, incurring a monetary loss (loss in sales for example). The owner sues the testing company for a 'DoS' and the testing company has no way to prove that it was authorized (what text file, there's none!).
I personally wouldn't do a stress test on a server without written authorization of the owner. The way I would implement this on an online service is by asking the target web site and the email address, where this address has to be in the public whois information. Then I'd send a special message to that email address and by replying the owner gives authorization (this way I'd have something from the official email address). This is not a 'written authorization' but comes as close as it can and the whois email address is used for domain transactions etc so it has the same level of protection/legal procedure behind.
[+] [-] vladimir|17 years ago|reply
[+] [-] rlonn|17 years ago|reply
[+] [-] agotterer|17 years ago|reply
One of the sites I am working on now is moving to a new codebase. We wanted to load test for a few hours but found everything we looked at to be too expensive.
[+] [-] rlonn|17 years ago|reply
[+] [-] pierrefar|17 years ago|reply
Refs:
http://httpd.apache.org/docs/2.0/programs/ab.html
http://www.hpl.hp.com/research/linux/httperf/
[+] [-] rlonn|17 years ago|reply
If you have the time, knowledge, interest and a server to spare, then ab or httperf (or I would recommend the Grinder - http://grinder.sourceforge.net/ ) is definitely worth a try however.
[+] [-] modoc|17 years ago|reply
One feature request that would have me signup in an instant is if you could provide web service load testing as well.
I'd also like to know more about the session recorder and if I can feed parameterized data into the scripts.
[+] [-] rlonn|17 years ago|reply
The session recorder is pretty cool, we think. We have managed to make it zero-configuration (i.e. no user configuration necessary) while still being able to handle client-side javascript. There are other HTTP session recorders out there, but to be able to handle javascript they're usually implemented as a true HTTP proxy that you have to configure your browser to use, or they're a browser plugin that you have to download and install.
The recorder outputs a load script that you can edit afterwards, if you want. However, the load script is currently more of a "list" of things to load, with pauses in between. It has no logic, conditional statements etc. and does not support parameterized data. This is something we intend to change soon though. It just needs some thought so we don't open up new possibilities for abuse/overuse of resources when we let users execute their own load script code on our systems.
[+] [-] kleevr|17 years ago|reply
[+] [-] wesley|17 years ago|reply
http://loadimpact.com/result/www.improvingtheweb.com-b75533d...
[+] [-] rlonn|17 years ago|reply
I think there is a lot we need to do to explain things better. That is the most difficult part of all.
[+] [-] d13hard|17 years ago|reply
[deleted]
[+] [-] asnyder|17 years ago|reply
That said, it would be great to incorporate some of their features such as different scenarios, load patterns, particular series of pages, random, etc. It would be great to not have to have a dedicated resource to run the stress tests, it's also tricky to know if the local bandwidth can properly emulate x number of machines.
[+] [-] jrockway|17 years ago|reply
Also, is there a way to simulate more than 5000 users?
[+] [-] rlonn|17 years ago|reply
5,000 users is what one single instance of the non-threaded load generator can simulate today. We have some optimizations we can make that should improve this though, and we are also going to implement multi-source/multi-program load generation that means we can distribute the load generation for a single test over several processes and several load generator hosts. Using only our own infrastructure I think we can scale the system quite a lot, but obviously, if we want to run really large simulations with several hundred thousand, or millions, of simulated users, then we need to buy cloud server capacity.
[+] [-] rubentopo|17 years ago|reply
-I believe that you'd have much more success if you charged on a test basis, your monthly billing might scare people away.
[+] [-] leftnode|17 years ago|reply
[+] [-] rlonn|17 years ago|reply
If you ask me, however, I'd say that the feedback we are getting here is what's really invaluable. It will help us improve the service a lot.
[+] [-] quickpost|17 years ago|reply
[+] [-] epi0Bauqu|17 years ago|reply
[+] [-] jusob|17 years ago|reply
[+] [-] timf|17 years ago|reply
[+] [-] paraschopra|17 years ago|reply
Does this mean somebody is DOSsing my site through your service?
[+] [-] rlonn|17 years ago|reply
I am going to fix so that we inform people what the offending URL/object is (rather than just saying "one of the systems involved in the test..."), so they can remove it from their load scripts (if you're a registered user you can edit your load scripts before starting a load test) and also contact us and ask us to put new systems into our exclusion list.