We've had a couple of cases of flaky tests failing builds over the last two years at my company. Most often it's browser / end-to-end type tests (e.g. selenium-style tests) that are the most flaky. Many of them only fail in 1-3% of cases, but if you have enough of them the chances of a failing build is significant.
If you have entire builds that are flaky, you end up training developers to just click "rebuild" the first one or two times a build fails, which can drastically increase the time before realizing the build is actually broken.
An important realization is that unit testing is not a good tool for testing flakyness of your main code - it is simply not a reliable indicator of failing code. Most of the time it's the test itself that is flaky, and it's not worth your time making every single test 100% reliable.
Some things we've implemented that helps a lot:
1. Have a system to reproduce the random failures. It took about a day to build tooling that can run say 100 instances of any test suite in parallel in CircleCI, and record the failure rate of individual tests.
2. If a test has a failure rate of > 10%, it indicates an issue in that test that should be fixed. By fixing these tests, we've found a couple of techniques to increase overall robustness of our tests.
3. If a test has a failure rate of < 3%, it is likely not worth your time fixing it. For these, we retry each failing test up to three times. Not all test frameworks support retying out of the box, but you can usually find a workaround. The retries can be restricted to specific tests or classes of tests if needed (e.g. only retry browser-based tests).
> If a test has a failure rate of < 3%, it is likely not worth your time fixing it.
How do you know? What you say is plausible, but it's also plausible that these rarely-failing tests also rarely-fail in production, and occasionally break things badly and cause outages or make customers think of your software as flaky.
Since you say this, I presume you've spent the time to actually track down the root causes of several tests that fail < 3% of the time? If so, what did you find? Some sort of issues with the test framework, or issues with your own code that you're confident would only ever be exposed by testing, or something else? I'm very curious.
> Most of the time it's the test itself that is flaky
I have always understood that unit tests must inherently be deterministic for the reason you explain.
A small test that is not deterministic is testing something other than "the unit" since there is another independent variable unaccounted for, often the state of the database or the configuration of a test environment.
Not that unit tests are perfect. Unit testing a concurrent data structure without threads (which are inherently nondeterministic) is not especially useful.
> 3. If a test has a failure rate of < 3%, it is likely not worth your time fixing it. For these, we retry each failing test up to three times. Not all test frameworks support retying out of the box, but you can usually find a workaround. The retries can be restricted to specific tests or classes of tests if needed (e.g. only retry browser-based tests).
to be pretty terrifying. I know that folks are under different amounts of pressure but we'd reject that code from merging here (or revert it out when we discovered the flakiness) as it's basically just a half-finished test that requires constant baby sitting.
> Most of the time it's the test itself that is flaky
I recently went through a heavy de-flaking on a suite of Selenium tests. I found this comment to be true in my case; it was reasonable-seeming assumptions in the tests that caused flakiness more often than anything else. The second most common cause was timing or networking issues with the Selenium farm.
Spending the time actually de-flaking the tests was quite enlightening and lead to some new best practices for both writing tests, and for spinning up Selenium instances.
Because of that experience, I'm not sure I would agree with giving up on tests that fail less than 3% of the time, because fixing one of those cases can sometimes fix all of them. Learning the root causes of test failures lead me to implement some fixes that increased the stability of the entire test suite. Sometimes there's only one problem but it causes all the tests in a group to fail one at a time, infrequently and seemingly randomly.
I fix flaky tests, and a 3% failure rate would drive me crazy. But I don't automatically rerun tests. I was skeptical of the idea that rerunning tests would help, so I did a bit of math:
A test suite with 1 test that fails 3% of the time will succeed 97% of the time. (1-.03)
A test suite with 10 tests that fail 3% of the time will succeed 74% of the time. (.97^10)
100 flaky tests? Now half your test runs fail. (.97^100)
You're retrying three times? Now your test suite is slow, but you can have up to 2,000 flaky tests before it starts becoming a real problem. Or 60,000 if you retry four times. (1-.03^3)^2000 = 94.7%; (1-.03^4)^60000 = 95.3%.
My conclusion: rerunning flaky tests is a legit way of solving the problem, as long as your tests aren't too slow. Still makes my skin itch, though. Fixing flaky tests forces me to face design flaws in my code.
(The math, in case I did it wrong: .03^3 = f = chance of a 3% failure test failing three runs in a row. 1-f = s = chance of test succeeding. s^1000 = chance of test run with 1000 flaky tests succeeding.)
Every company I've founded or worked for has struggled with flaky tests.
Twitter had a comprehensive browser and system test suite that took about an hour to run (and they had a large CI worker cluster). Flaky tests could and did scuttle deploys. It was a never-ending struggle to keep CI green, but most engineers saw de-flaking (not just deleting the test) as a critical task.
PhotoStructure has an 8-job GitLab CI pipeline that runs on macOS, Windows, and Linux. Keeping the ~3,000 (and growing) tests passing reliably has proven to be a non-trivial task, and researching why a given task is flaky on one OS versus another has almost invariably led to discovery and hardening of edge and corner conditions.
It seems that TFA only touched on set ordering, incomplete db resets and time issues. There are many other spectres to fight as soon as you deal with multi-process systems on multiple OSes, including file system case sensitivity, incomplete file system resets, fork behavior and child process management, and network and stream management.
There are several aspects I added to stabilize CI, including robust shutdown and child process management systems. I can't say I would have prioritized those things if I didn't have tests, but now that I have it, I'm glad they're there.
In my experience complex end-to-end tests cast a wide net that often results in finding a lot of issues and they provide enormous value. Their main negative is maintenance around robustness as the article discusses and hardening tests can take a lot of investment. That said, the alternative is worse (not having them) so I find your approach, and the author’s is what I’ve often done. I think there needs to be understanding (across the team and management) that automated tests are software and it will require a similar dev effort to maintaining any other software, especially so because there usually aren’t tests to test the tests!
I’m founder at Tesults (https://www.tesults.com) where we have a flaky test indicator that makes identifying these tests easier. It’s free to try and if you can’t get budget for a proper plan send me an email and I’ll do what I can.
In general the only way to never have flaky tests is to have simpler tests but I find those often don’t provide as much value - that’s just my personal belief after having spent years focused on automated tests, e2e tests do have robustness issues but the bugs they find make them totally worth it. Out of the issues mentioned in the article that affected my tests the most, it’s timing. They can be overcome though, I’ve run test suites with a couple of thousand e2e tests (browser) that have been highly robust and reliable after time was devoted to hardening them. You do have to focus on that and refuse to add new test cases until the existing ones are sorted out in some cases.
In an old job, we had a frustrating test that passed well over 99 times in 100. It was shrugged off for a very long time until a developer eventually tracked it down to code that was generating a random SSL key pair. If the first byte of the key was 0, faulty code elsewhere would mishandle the key and the test failed.
Keeping the randomness in the test was the key factor in tracking down this obscure bug. If the test had been made completely deterministic, the test harness would never have discovered the problem. So although repeatable tests are in most cases a good thing, non-determinism can unearth problems. The trick is how to do this without sucking up huge amounts of bug-tracking time...
(Much effort was spent in making the test repeatable during debugging, but of course the crypto code elsewhere was deliberately trying to get as much randomness as it could source...)
It doesn't seem to be a strong argument to have non-deterministic tests.
There was the logic that generates the SSL key pair, and there is the faulty logic that consumes it. Based on the description, it seems it's an indication of missing test coverage around the faulty code. If, when the faulty code was written, more time were spent on understanding the assumptions the code has made, then maybe the test wouldn't appear in the first place.
This anecdote, however, does bring up a good point: Don't shrug off intermittently failed tests - Dig in and understand the root cause of it.
I'm a bit confused, that seems like the test did it's job, it found the faulty code elsewhere. (unless by elsewhere you mean in the test code) It just appeared to be flaky and was treated as such until someone looked into it.
I would say that your non-deterministic components here weren't a good thing - instead the test was poorly written and didn't cover the assumptions of the code under test well. The fact that this bug was revealed by a test is useful, but I should hope that now the tests have cemented that case in a regression suite in a deterministic manner.
What I found to be the major reasons for flaky tests:
* Non-determinism in the code - e.g. select without an order by, random number generators, hashmaps turned into lists, etc. - Fixed by turning non-deterministic code into deterministic code, testing for properties rather than outcomes or isolating and mocking the non-deterministic code.
* Lack of control over the environment - e.g. calling a third party service that goes down occasionally, use of a locally run database that gets periodically upgraded by the package manager - fixed by gradually bringing everything required to run your software under control (e.g. installing specific versions without package manager, mocking 3rd party services, intercepting syscalls that get time and replacing them with consistent values).
* Race conditions - in this case the test should really repeat the same actions so that it consistently catches the flakiness.
There was one weird bug reported to me in an microcontroller based project I was recently working on which shut off half the LCD screen. I wrote a test which blasted the LCD screen with random characters and commands and did not see the same error for awhile... but it finally happened during a test! I was able to then see that when I was checking the LCD state between commands I only would toggle the chip select for the first half of the LCD (there were 2 driver chips built into the screen and you had to read each chip individually). There would be no way I could have recreated the bug without automated tests.
I have had to deal with non-deterministic tests with my embedded systems and robotic test suites and have found a few solutions to deal with them:
- Do a full power reset between tests if possible, or do it between test suites when you can combine tests together in suites that don't require a complete clean slate
- Reset all settings and parameters between tests. A lot of embedded systems have settings saved in Flash or EEPROM which can affect all sorts of behaviors, so make sure it always starts at the default setting.
- Have test commands for all system inputs and initialize all inputs to known values.
- Have test modes for all system outputs such as motors. If there is a motor which has a speed encoder you can make the test mode for the speed encoder input to match the commanded motor value, or also be able to trigger error inputs such as a stalled motor.
- Use a user input/dialog option to have user feedback as part of the test (for things like the LCD bug).
Robot Framework is a great tool which can do all these things with a custom Python library! I think testing embedded systems is generally much harder so people rarely do it, but I think it is a great tool which can oftentimes uncover these flaky errors.
Related stories: "unit tests fail when run in Australia" [1] and "the case of the 500-mile email" [2]. There is a whole GitHub repository dedicated to some very interesting debugging stories [3].
We do a lot of integration testing, more so than unit testing, and those tests, which randomly fail, are a real headache.
One thing I learned is that setting up tests correctly, independent of each other, is hard. It is even harder if databases, local and remote services are involved or if your software communicates with other software. You need to start those dependencies and take care of resetting their state, but there's always something: Services sometimes take longer to start, file handles not closing on time, code or applications which keeps running when another test fails... etc, etc...
There are obvious solutions: Mocking everything, removing global state, writing more robust test setup code... But who has time for this? Fixing things correctly can even take more time and usually does not guarantee that some new change in the future disregards your correct code...
>There are obvious solutions: Mocking everything, removing global state, writing more robust test setup code... But who has time for this?
I find that doing all of this tends to actually save time overall it's just that the up front investment is high and the payoff is realized over a long time.
Most software teams seem to prefer higher ongoing costs if it comes with quick wins to up front investment.
>There are obvious solutions: Mocking everything, removing global state, writing more robust test setup code... But who has time for this?
If you do it from the beginning and structure your code in a testable way it doesn't take much time. It saved me a few time in my current company; make a small change -> turns out it breaks a feature from 3-4 years ago that no one even remember -> look at the tests -> understand the feature as well as why what you did broke it.
If you try to do it after X years of coding without thinking about tests you're doomed though.
The world is non-deterministic. A test suite that can represent non-determinism is much more powerful than one that cannot. To paraphrase Dijkstra, "Determinism is just a special case of non-determinism, and not a very interesting one at that."
If a test is non-deterministic then a test framework needs to characterize the distribution of results for that test. For example "Branch A fails 11% (+/- 2%) of the time and Branch B fails 64% (+/- 2%) of the time." Once you are able to measure non-determinism then you can also effectively optimize it away, and you start looking for ways to introduce more of it into your test suites e.g. to run each test on a random CPU/distro/kernel.
But you pay the cost of retrying the failing tests and lack of clear signal. And if the application code is flaky, users get to experience the breakage too.
Call it a pet peeve, but if we call it "chaos engineering" it costs a ton and gets people conference talks when a sporadic system integration issue is found. But if you have the same thing happen in a plain old CI half the time it will be ignored or flagged flaky.
IIUC, Chaos Engineering is about moving things out of "eh, it won't happen in production, I'll ignore it" into "it will happen in production, I'd better handle it", and making sure mitigation and recovery code is actually exercised in a realistic setting. "Periodic errors in CI that go unmitigated and produce test failures" seems very meaningfully distinct from chaos engineering.
As soon as I saw that whole section on database-related flakiness my mind went from "flaky unit tests" to "tests called unit tests that are actually integration tests". I worked on a team where we labored under that misconception for a long, long time. By the time we finally realized that many of the tests in our suite were integration tests and not unit tests it was too late to change (due to budget and timeline pressure).
I really like the different approaches to dealing with these flaky tests, that is a good list.
I think it's important that engineers can distinguish between testing code in isolation versus "integration" or "system" testing, but I've seen a sophomoric stigma around integration tests that lead to mocking hell, and a hatred towards testing in general.
Unit tests are great. You want them. Craft your interfaces to enable them.
Integration and system tests are important too. Again, crafting higher level interfaces that allow for testing will, in general, lead to a more ergonomic API.
Analogously: unit tests ensure each of your LEGO blocks are individually well-formed. Integration tests ensure that the build instructions actually result in something reasonable.
I think the definition has evolved. Since usually the DB is reset between tests (and such reset is snappy), it's transparent enough to appear as unit testing. That we do this in a buggy manner or do not understand how to properly reset the DB does not negate that fact in my opinion. You could mock it or inject it, therefore doing unit testing in the traditional sense, but again you could introduce even nastier bugs and a whole lot of indirection overhead.
Hi - I'm Jon, creator of "Flaptastic" (https://www.flaptastic.com/) and passionate advocate for unit test health.
Having coded at both Lyft and at DoorDash, I noticed both companies had the exact same unit test health problems and I was forced to manually come up with ways to make the CI/CD reliable in both settings.
In my experience, most people want a turnkey solution to get them to a healthier place with their unit testing. "Flaptastic" is a flaky unit tests recognition engine written in a way that anybody can use it to clean up their flaky unit tests no matter what CI/CD or test suite you're already using.
Flaptastic is a test suite plugin that works with a SAAS backend that is able to differentiate between a unit test that failed due to broken application code versus tests that are failing with no merit and only because the tests are not written well. Our killer feature is that you get a "kill switch" to instantly disable any unit test that you know is unhealthy with an option to unkill it later when you've fixed the problem. The reason is this is so powerful is that when you kill an unhealthy test, you are able to immediately unblock the whole team.
We're now working on a way to accept the junit.xml file from your test suite. We can run it through the flap recognition engine allowing you to make decisions on what you will do next if you know all of the tests that failed did fail due to known flaky test patterns.
If Flaptastic seems interesting, contact us on our chat widget we'll let you use it for free indefinitely (for trial purposes) to decide if this makes your life easier.
At Undo we develop a "software flight recorder technology" - basically think of `rr` reversible debugger, it is our open source competitor.
One particular usecase for Undo (besides obviously recording software bugs per se) is recording execution of tests. Huge time saver. We do this ourselves - when a test fails in CI, engineers can download a recording file of a failing test and investigate it with our reversible debugger.
Yeah, this is huge. rr also has "chaos mode" to randomize things to make test failures easier to reproduce. (I understand Undo has something similar.)
I think that's one message that is completely lost in the article and in the rest of the comments here: it is possible to improve technology so that flaky tests are more debuggable.
With enough investment (hardware and OS support for low-impact always-on recording) we could make every flaky test debuggable.
At our place, we call them "peuteterli" (losely translated: "could-be-ish" constructed from the French "peut être" and slapped on the local German diminutive -li.
For the ID issue I have a monkey patch for Activerecord:
if ["test", "cucumber"].include? Rails.env
class ActiveRecord::Base
before_create :set_id
def set_id
self.id ||= SecureRandom.random_number(999_999_999)
end
end
end
Unique IDs are also helpful when scanning for specific objects during test development. When all objects of different classes start with 1, it is hard to following the connections.
I deal with this issue a lot in my current job, and did in my last job too. IMX timing issues are by far the most common culprit. Usually it's because a test has to guess how long a background repair or garbage-collection activity will take, when in fact that duration can be highly variable. Shorter timeouts mean tests are unreliable. Longer timeouts mean greater reliability but tests that sometimes take forever. Speeding up the background processes can create CPU contention if tests are being run in parallel, making other tests seem flaky. Various kinds of race conditions in tests are also a problem, but not one I personally encounter that often. Probably has to do with the type of software I work on (storage) and the type of developers I consequently work with.
No matter what, developers complain and try to avoid running the tests at all. I'd love to force their hand by making a successful test run an absolute requirement for committing code, but the very fact that tests have been slow and flaky since long before I got here means that would bring development to a standstill for weeks and I lack the authority (real or moral) for something that drastic. Failing that, I lean toward re-running tests a few times for those that are merely flaky (especially because of timing issues), and quarantine for those that are fully broken. Then there's still a challenge getting people to fix their broken tests, but life is full of tradeoffs like that.
We're usually calling them "blinker tests" in our integration test suite. Reasons for blinker tests vary, but most are in line with what others here have already stated: concurrency, especially correct synchronization of test execution with stuff happening in asynchronous parts of the (distributed) system under test, is by far the biggest cause for problematic tests. This one is often exagerrated by the difference in concurrent execution on developer machines with maybe 4-6 cores and the CI server with 50-80, which often leads to "blinking" behavior that never happens locally, but every few builds on the CI server.
Second biggest is database transaction management and incorrect assumptions over when database changes become visible to other processes (which are in some way also concurrency problems, so it basically comes down to that). Third biggest is unintentional nondeterminism in the software, like people assuming that a certain collection implementation has deterministic order, but actually it doesn't, someone was just lucky to get the same order all the time while testing on the dev machine.
"Making bad assumptions about DB ordering"
That's caught me out before. Postgres is just weird, I had to run the same test in a loop for an hour before it'd randomly change the order.
There's several reasons for potential ordering changes:
- the order of items on the page is different, due to the way tuples have been inserted (different external scheduling, different postgres internal scheduling)
- concurrent sequential scans can coordinate relation scans, which is quite helpful for relations that are larger than the cache
- different query plans, e.g. sequential vs index scans
Unless you specify the ORDER BY, there really isn't any guarantee by postgres. We could make it consistent, but that'd add overhead for everyone.
If anyone is looking for ideas for how to build tooling that fights flaky tests, I consolidated a number of lessons into a tool I open sourced a while ago.
It will do things like separate out different kinds of test failures (by error message and stacktrace) and then measure their individual rates of incidence.
You can also ask it to reproduce a specific failure in a tight loop and once it succeeds it will drop you into a debugger session so you can explore what's going on.
The two big problems seem to be concurrency (always a problem) and state, which immediately suggest that making things as functional as possible would help a lot.
Ideally all state that's used in a test would be reset to a known value at or before the start of the test, but this is quite hard for external non-mocked databases, clocks and so on.
For integration tests, do you run in a controllable "safe" environment and risk false-passes, or an environment as close as possible to production and risk intermittent failure?
A variant I've seen is "compiled languages may re-order floating point calculations between builds resulting in different answers", which is extremely annoying to deal with especially when you can't just epsilon it away.
Both this article and this comment thread include a number of different ideas regarding controlling (or randomizing) environmental factors: test ordering, system time, etc.
But why do all of this piecemeal? Our philosophy is to create a controlled test sandbox environment that makes all these aspects (including concurrency) reproducible:
The idea is to guarantee that any flake is easy to reproduce. If people have objections to that approach, we'd love to hear them. Conversely, if you would be willing to test out our early prototype, get in touch.
I used to work at a company with over 10,000 tests where we weren't able to get more than an 80% pass rate due to flaky tests. This article is great and covers a lot of the options for handling flaky tests. I founded Appsurify to make it easy for companies to handle flaky tests, with minimal effort.
First, don't delete them, flaky tests are still valuable and can still find bugs. We also had the challenge where a lot of the 'flakiness' was not the test or the application's fault but was caused by 3rd party providers. Even at Google "Almost 16% of our tests have some level of flakiness associated with them!" - John Micco, so just writing tests that aren't flaky isn't always possible.
Appsurify automatically raises defects when tests fail, and if the failure reason looks to be 'flakiness' (based on failure type, when the failure occurred, the change being made, previous known flaky failures) then we raise the defect as a "flaky" defect. Teams can then have the build fail based only on new defects and prevent it from failing when there are flaky test results.
We also prioritize the tests, which causes fewer tests to be run which are more likely to fail due to a real defect, which also reduces the number of flaky test results.
[+] [-] matharmin|6 years ago|reply
If you have entire builds that are flaky, you end up training developers to just click "rebuild" the first one or two times a build fails, which can drastically increase the time before realizing the build is actually broken.
An important realization is that unit testing is not a good tool for testing flakyness of your main code - it is simply not a reliable indicator of failing code. Most of the time it's the test itself that is flaky, and it's not worth your time making every single test 100% reliable.
Some things we've implemented that helps a lot:
1. Have a system to reproduce the random failures. It took about a day to build tooling that can run say 100 instances of any test suite in parallel in CircleCI, and record the failure rate of individual tests.
2. If a test has a failure rate of > 10%, it indicates an issue in that test that should be fixed. By fixing these tests, we've found a couple of techniques to increase overall robustness of our tests.
3. If a test has a failure rate of < 3%, it is likely not worth your time fixing it. For these, we retry each failing test up to three times. Not all test frameworks support retying out of the box, but you can usually find a workaround. The retries can be restricted to specific tests or classes of tests if needed (e.g. only retry browser-based tests).
[+] [-] justinpombrio|6 years ago|reply
How do you know? What you say is plausible, but it's also plausible that these rarely-failing tests also rarely-fail in production, and occasionally break things badly and cause outages or make customers think of your software as flaky.
Since you say this, I presume you've spent the time to actually track down the root causes of several tests that fail < 3% of the time? If so, what did you find? Some sort of issues with the test framework, or issues with your own code that you're confident would only ever be exposed by testing, or something else? I'm very curious.
[+] [-] humanrebar|6 years ago|reply
I have always understood that unit tests must inherently be deterministic for the reason you explain.
A small test that is not deterministic is testing something other than "the unit" since there is another independent variable unaccounted for, often the state of the database or the configuration of a test environment.
Not that unit tests are perfect. Unit testing a concurrent data structure without threads (which are inherently nondeterministic) is not especially useful.
[+] [-] munk-a|6 years ago|reply
> 3. If a test has a failure rate of < 3%, it is likely not worth your time fixing it. For these, we retry each failing test up to three times. Not all test frameworks support retying out of the box, but you can usually find a workaround. The retries can be restricted to specific tests or classes of tests if needed (e.g. only retry browser-based tests).
to be pretty terrifying. I know that folks are under different amounts of pressure but we'd reject that code from merging here (or revert it out when we discovered the flakiness) as it's basically just a half-finished test that requires constant baby sitting.
[+] [-] dahart|6 years ago|reply
I recently went through a heavy de-flaking on a suite of Selenium tests. I found this comment to be true in my case; it was reasonable-seeming assumptions in the tests that caused flakiness more often than anything else. The second most common cause was timing or networking issues with the Selenium farm.
Spending the time actually de-flaking the tests was quite enlightening and lead to some new best practices for both writing tests, and for spinning up Selenium instances.
Because of that experience, I'm not sure I would agree with giving up on tests that fail less than 3% of the time, because fixing one of those cases can sometimes fix all of them. Learning the root causes of test failures lead me to implement some fixes that increased the stability of the entire test suite. Sometimes there's only one problem but it causes all the tests in a group to fail one at a time, infrequently and seemingly randomly.
[+] [-] jdlshore|6 years ago|reply
A test suite with 1 test that fails 3% of the time will succeed 97% of the time. (1-.03)
A test suite with 10 tests that fail 3% of the time will succeed 74% of the time. (.97^10)
100 flaky tests? Now half your test runs fail. (.97^100)
You're retrying three times? Now your test suite is slow, but you can have up to 2,000 flaky tests before it starts becoming a real problem. Or 60,000 if you retry four times. (1-.03^3)^2000 = 94.7%; (1-.03^4)^60000 = 95.3%.
My conclusion: rerunning flaky tests is a legit way of solving the problem, as long as your tests aren't too slow. Still makes my skin itch, though. Fixing flaky tests forces me to face design flaws in my code.
(The math, in case I did it wrong: .03^3 = f = chance of a 3% failure test failing three runs in a row. 1-f = s = chance of test succeeding. s^1000 = chance of test run with 1000 flaky tests succeeding.)
[+] [-] mceachen|6 years ago|reply
Twitter had a comprehensive browser and system test suite that took about an hour to run (and they had a large CI worker cluster). Flaky tests could and did scuttle deploys. It was a never-ending struggle to keep CI green, but most engineers saw de-flaking (not just deleting the test) as a critical task.
PhotoStructure has an 8-job GitLab CI pipeline that runs on macOS, Windows, and Linux. Keeping the ~3,000 (and growing) tests passing reliably has proven to be a non-trivial task, and researching why a given task is flaky on one OS versus another has almost invariably led to discovery and hardening of edge and corner conditions.
It seems that TFA only touched on set ordering, incomplete db resets and time issues. There are many other spectres to fight as soon as you deal with multi-process systems on multiple OSes, including file system case sensitivity, incomplete file system resets, fork behavior and child process management, and network and stream management.
There are several aspects I added to stabilize CI, including robust shutdown and child process management systems. I can't say I would have prioritized those things if I didn't have tests, but now that I have it, I'm glad they're there.
[+] [-] ajeet_dhaliwal|6 years ago|reply
I’m founder at Tesults (https://www.tesults.com) where we have a flaky test indicator that makes identifying these tests easier. It’s free to try and if you can’t get budget for a proper plan send me an email and I’ll do what I can.
In general the only way to never have flaky tests is to have simpler tests but I find those often don’t provide as much value - that’s just my personal belief after having spent years focused on automated tests, e2e tests do have robustness issues but the bugs they find make them totally worth it. Out of the issues mentioned in the article that affected my tests the most, it’s timing. They can be overcome though, I’ve run test suites with a couple of thousand e2e tests (browser) that have been highly robust and reliable after time was devoted to hardening them. You do have to focus on that and refuse to add new test cases until the existing ones are sorted out in some cases.
[+] [-] hexfran|6 years ago|reply
[+] [-] joosters|6 years ago|reply
Keeping the randomness in the test was the key factor in tracking down this obscure bug. If the test had been made completely deterministic, the test harness would never have discovered the problem. So although repeatable tests are in most cases a good thing, non-determinism can unearth problems. The trick is how to do this without sucking up huge amounts of bug-tracking time...
(Much effort was spent in making the test repeatable during debugging, but of course the crypto code elsewhere was deliberately trying to get as much randomness as it could source...)
[+] [-] kenha|6 years ago|reply
There was the logic that generates the SSL key pair, and there is the faulty logic that consumes it. Based on the description, it seems it's an indication of missing test coverage around the faulty code. If, when the faulty code was written, more time were spent on understanding the assumptions the code has made, then maybe the test wouldn't appear in the first place.
This anecdote, however, does bring up a good point: Don't shrug off intermittently failed tests - Dig in and understand the root cause of it.
[+] [-] aidenn0|6 years ago|reply
[+] [-] jon889|6 years ago|reply
[+] [-] munk-a|6 years ago|reply
[+] [-] tinus_hn|6 years ago|reply
[+] [-] pytester|6 years ago|reply
* Non-determinism in the code - e.g. select without an order by, random number generators, hashmaps turned into lists, etc. - Fixed by turning non-deterministic code into deterministic code, testing for properties rather than outcomes or isolating and mocking the non-deterministic code.
* Lack of control over the environment - e.g. calling a third party service that goes down occasionally, use of a locally run database that gets periodically upgraded by the package manager - fixed by gradually bringing everything required to run your software under control (e.g. installing specific versions without package manager, mocking 3rd party services, intercepting syscalls that get time and replacing them with consistent values).
* Race conditions - in this case the test should really repeat the same actions so that it consistently catches the flakiness.
[+] [-] roland35|6 years ago|reply
- The temperature is much hotter/colder than normal
- Someone is inadvertently holding down a button or key on the machine under test
- The wrong version of software is loaded onto the machine
[+] [-] taneq|6 years ago|reply
I thought tests weren't meant to have external dependencies (or at least, ones outside the control of the test harness)?
[+] [-] roland35|6 years ago|reply
I have had to deal with non-deterministic tests with my embedded systems and robotic test suites and have found a few solutions to deal with them:
- Do a full power reset between tests if possible, or do it between test suites when you can combine tests together in suites that don't require a complete clean slate
- Reset all settings and parameters between tests. A lot of embedded systems have settings saved in Flash or EEPROM which can affect all sorts of behaviors, so make sure it always starts at the default setting.
- Have test commands for all system inputs and initialize all inputs to known values.
- Have test modes for all system outputs such as motors. If there is a motor which has a speed encoder you can make the test mode for the speed encoder input to match the commanded motor value, or also be able to trigger error inputs such as a stalled motor.
- Use a user input/dialog option to have user feedback as part of the test (for things like the LCD bug).
Robot Framework is a great tool which can do all these things with a custom Python library! I think testing embedded systems is generally much harder so people rarely do it, but I think it is a great tool which can oftentimes uncover these flaky errors.
[+] [-] darekkay|6 years ago|reply
[1] https://github.com/angular/angular.js/issues/5017
[2] http://www.ibiblio.org/harris/500milemail.html
[3] https://github.com/danluu/debugging-stories
[+] [-] zubspace|6 years ago|reply
We do a lot of integration testing, more so than unit testing, and those tests, which randomly fail, are a real headache.
One thing I learned is that setting up tests correctly, independent of each other, is hard. It is even harder if databases, local and remote services are involved or if your software communicates with other software. You need to start those dependencies and take care of resetting their state, but there's always something: Services sometimes take longer to start, file handles not closing on time, code or applications which keeps running when another test fails... etc, etc...
There are obvious solutions: Mocking everything, removing global state, writing more robust test setup code... But who has time for this? Fixing things correctly can even take more time and usually does not guarantee that some new change in the future disregards your correct code...
[+] [-] pytester|6 years ago|reply
I find that doing all of this tends to actually save time overall it's just that the up front investment is high and the payoff is realized over a long time.
Most software teams seem to prefer higher ongoing costs if it comes with quick wins to up front investment.
[+] [-] lm28469|6 years ago|reply
If you do it from the beginning and structure your code in a testable way it doesn't take much time. It saved me a few time in my current company; make a small change -> turns out it breaks a feature from 3-4 years ago that no one even remember -> look at the tests -> understand the feature as well as why what you did broke it.
If you try to do it after X years of coding without thinking about tests you're doomed though.
[+] [-] lukego|6 years ago|reply
The world is non-deterministic. A test suite that can represent non-determinism is much more powerful than one that cannot. To paraphrase Dijkstra, "Determinism is just a special case of non-determinism, and not a very interesting one at that."
If a test is non-deterministic then a test framework needs to characterize the distribution of results for that test. For example "Branch A fails 11% (+/- 2%) of the time and Branch B fails 64% (+/- 2%) of the time." Once you are able to measure non-determinism then you can also effectively optimize it away, and you start looking for ways to introduce more of it into your test suites e.g. to run each test on a random CPU/distro/kernel.
[+] [-] muro|6 years ago|reply
[+] [-] mrkeen|6 years ago|reply
[+] [-] throwaway5752|6 years ago|reply
[+] [-] dllthomas|6 years ago|reply
[+] [-] mekane8|6 years ago|reply
I really like the different approaches to dealing with these flaky tests, that is a good list.
[+] [-] mceachen|6 years ago|reply
Unit tests are great. You want them. Craft your interfaces to enable them.
Integration and system tests are important too. Again, crafting higher level interfaces that allow for testing will, in general, lead to a more ergonomic API.
Analogously: unit tests ensure each of your LEGO blocks are individually well-formed. Integration tests ensure that the build instructions actually result in something reasonable.
[+] [-] why-el|6 years ago|reply
[+] [-] jonthepirate|6 years ago|reply
Having coded at both Lyft and at DoorDash, I noticed both companies had the exact same unit test health problems and I was forced to manually come up with ways to make the CI/CD reliable in both settings.
In my experience, most people want a turnkey solution to get them to a healthier place with their unit testing. "Flaptastic" is a flaky unit tests recognition engine written in a way that anybody can use it to clean up their flaky unit tests no matter what CI/CD or test suite you're already using.
Flaptastic is a test suite plugin that works with a SAAS backend that is able to differentiate between a unit test that failed due to broken application code versus tests that are failing with no merit and only because the tests are not written well. Our killer feature is that you get a "kill switch" to instantly disable any unit test that you know is unhealthy with an option to unkill it later when you've fixed the problem. The reason is this is so powerful is that when you kill an unhealthy test, you are able to immediately unblock the whole team.
We're now working on a way to accept the junit.xml file from your test suite. We can run it through the flap recognition engine allowing you to make decisions on what you will do next if you know all of the tests that failed did fail due to known flaky test patterns.
If Flaptastic seems interesting, contact us on our chat widget we'll let you use it for free indefinitely (for trial purposes) to decide if this makes your life easier.
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] andrey_utkin|6 years ago|reply
One particular usecase for Undo (besides obviously recording software bugs per se) is recording execution of tests. Huge time saver. We do this ourselves - when a test fails in CI, engineers can download a recording file of a failing test and investigate it with our reversible debugger.
[+] [-] roca|6 years ago|reply
I think that's one message that is completely lost in the article and in the rest of the comments here: it is possible to improve technology so that flaky tests are more debuggable.
With enough investment (hardware and OS support for low-impact always-on recording) we could make every flaky test debuggable.
[+] [-] bhaak|6 years ago|reply
For the ID issue I have a monkey patch for Activerecord:
Unique IDs are also helpful when scanning for specific objects during test development. When all objects of different classes start with 1, it is hard to following the connections.[+] [-] notacoward|6 years ago|reply
No matter what, developers complain and try to avoid running the tests at all. I'd love to force their hand by making a successful test run an absolute requirement for committing code, but the very fact that tests have been slow and flaky since long before I got here means that would bring development to a standstill for weeks and I lack the authority (real or moral) for something that drastic. Failing that, I lean toward re-running tests a few times for those that are merely flaky (especially because of timing issues), and quarantine for those that are fully broken. Then there's still a challenge getting people to fix their broken tests, but life is full of tradeoffs like that.
[+] [-] Slartie|6 years ago|reply
Second biggest is database transaction management and incorrect assumptions over when database changes become visible to other processes (which are in some way also concurrency problems, so it basically comes down to that). Third biggest is unintentional nondeterminism in the software, like people assuming that a certain collection implementation has deterministic order, but actually it doesn't, someone was just lucky to get the same order all the time while testing on the dev machine.
[+] [-] jonatron|6 years ago|reply
[+] [-] anarazel|6 years ago|reply
- the order of items on the page is different, due to the way tuples have been inserted (different external scheduling, different postgres internal scheduling) - concurrent sequential scans can coordinate relation scans, which is quite helpful for relations that are larger than the cache - different query plans, e.g. sequential vs index scans
Unless you specify the ORDER BY, there really isn't any guarantee by postgres. We could make it consistent, but that'd add overhead for everyone.
[+] [-] adamb|6 years ago|reply
https://github.com/ajbouh/qa
It will do things like separate out different kinds of test failures (by error message and stacktrace) and then measure their individual rates of incidence.
You can also ask it to reproduce a specific failure in a tight loop and once it succeeds it will drop you into a debugger session so you can explore what's going on.
There are demo videos in the project highlighting these techniques. Here's one: https://asciinema.org/a/dhdetw07drgyz78yr66bm57va
[+] [-] pjc50|6 years ago|reply
Ideally all state that's used in a test would be reset to a known value at or before the start of the test, but this is quite hard for external non-mocked databases, clocks and so on.
For integration tests, do you run in a controllable "safe" environment and risk false-passes, or an environment as close as possible to production and risk intermittent failure?
A variant I've seen is "compiled languages may re-order floating point calculations between builds resulting in different answers", which is extremely annoying to deal with especially when you can't just epsilon it away.
[+] [-] AstralStorm|6 years ago|reply
[+] [-] rrnewton|6 years ago|reply
But why do all of this piecemeal? Our philosophy is to create a controlled test sandbox environment that makes all these aspects (including concurrency) reproducible:
https://www.cloudseal.io/blog/2018-04-06-intro-to-fixing-fla...
The idea is to guarantee that any flake is easy to reproduce. If people have objections to that approach, we'd love to hear them. Conversely, if you would be willing to test out our early prototype, get in touch.
[+] [-] invertednz|6 years ago|reply
First, don't delete them, flaky tests are still valuable and can still find bugs. We also had the challenge where a lot of the 'flakiness' was not the test or the application's fault but was caused by 3rd party providers. Even at Google "Almost 16% of our tests have some level of flakiness associated with them!" - John Micco, so just writing tests that aren't flaky isn't always possible.
Appsurify automatically raises defects when tests fail, and if the failure reason looks to be 'flakiness' (based on failure type, when the failure occurred, the change being made, previous known flaky failures) then we raise the defect as a "flaky" defect. Teams can then have the build fail based only on new defects and prevent it from failing when there are flaky test results.
We also prioritize the tests, which causes fewer tests to be run which are more likely to fail due to a real defect, which also reduces the number of flaky test results.