[onalark]

Awesome, I'm a huge fan of new and innovative tools that help improve the process of refactoring and improving existing code. This looks like a really promising tool for Ruby developers, and I'm always grateful when companies and their employees invest the time and effort to release their tools to the community. I really liked the point about "buggy data" as opposed to just buggy code, I think that's a really important point.

A few reactions from reading through the release:

Scientist appears to be largely limited to situations where the code has no "side effects". I think this is a pretty big caveat, and it would have been helpful in the introduction/summary to see this mentioned. Similarly, I think it would be nice to point out that Scientist is a Ruby-only framework :)

You don't mention "regression test" at any point in the article, which is the language I'm most familiar with for referring to this sort of testing. How does a scientist "experiment" compare to a regression test over that block of code?

Anyway, thanks again for writing this up, I'll be thinking more about the Experiment testing pattern for my own projects.

[gregmac]

> Scientist appears to be largely limited to situations where the code has no "side effects".

That's one of the things I was initially thinking too, but then as I thought about where I could have used it in the past, I can think of only a few cases where it wouldn't be possible to keep it isolated.

For example, have your new code running against a non-live data store. Example: When a user changes permissions, the old code changes the live DB, while the new code changes the temporary DB. Later (or continuously), you could also compare the databases to ensure they're identical (easier if the datastore remains constant, a bit harder if you are changing to a different storage schema or product).

Where it would be in the difficult-to-impossible range is when touching on external services that don't have the ability to copy state and set up a secondary test service, but even in that case, you could record the requests made (or that would have been made) and ensure they both would have done the same thing.

Definitely an interesting concept overall.