Software engineering is programming over time. It's not so hard to write code that is correct today, if that's all that tests did then they wouldn't be worth the effort.
We write tests so that we know whether future changes have broken the system or not.
This article sounds like a reaction to the practice of writing many small, hermetic unit tests, which do little more than recapitulate the code under test. The weakest parts of a system are often in the joints, so the most important tests to write and to run are the integration tests, the ones that tell you with the most confidence whether the code works in the real world or not.
See, for example, this puzzling line from the article:
> Other tests grow obsolete because the target platform is retired, but not the test. There's even less pressure to remove stale tests than useless code. Watch as the workaround for Windows XP is removed from the code, but not the test checking it still works.
How do you remove the code for a feature, but not the test for that feature unless either you aren't running your tests or your tests aren't testing the things that you care about?
Think the question "What's the return on investment on writing tests?" doesn't get asked enough. There's a blind assumption that writing lots of tests is always good and code coverage tools tend to push the idea that you're not done until you reach 100% coverage.
Imagine some incubating startup where you have an initial 4-week runway for development, with the goal of getting a prototype working, so you can get it in the hands of some customers and begin validating a business case. How many tests do you need in this case? And how much time do you allow for it? Most of the code you're going to produce will be thrown away and the more compelling the prototype is in terms of functionality, features and design the stronger the signal you'll get back from potential customers.
To me there's still too much "testing is my religion" amongst developers and not enough "here's the quantified value of me spending 3 days writing tests and here's how that fits in the context of where our business is right now"
Absolutely agree. My biggest lightbulb moments with testing have come when needing to do significant refactoring. Without tests, I feel aimless because it's hard to get feedback about whether the refactoring has 'worked'. With tests there's a nice, tight feedback loop. Not saying it gives any guarantees, in my experience it has dramatically increased how quickly I could improve the code - to the extent where in some cases I wouldn't have even tried in the absence of tests because I wouldn't have been confident I could improve it.
Are you defining unit tests are per class tests? This is the biggest mistake.
You should be testing a unit of behaviour, like a business rule or an effect to be expected. If this involves multiple objects collaborating then so be it. But they should not cross architectural boundaries like http calls, or database calls. If you can only make public api classes accessible, and make all helper classes inaccessible to users. That also forces you test things from only a public api standpoint.
Tests that just check an object calls another object, but exhibits no desired behaviour in its self are pointless, and just couple everything to the implementation. I.e Certain methods, are called in a certain order with specific parameters. When you do that, you make it really hard to change things(just design changes, not desired behaviour changes) without breaking every test.
Small unit tests are valuable iff your small units are well designed and valuable. If they are brittle, temporary components they don't have much value, I agree, but writing mostly integration tests suffers from the combinatorial explosion problem and from making it hard to express succinctly what the important bit about a piece of code is (and besides, almost no test is 100% integrated, you always isolate something, even in full end-to-end tests).
If you have many functional, stateless components, the unit tests can be particularly valuable.
But I think many tests end up being badly written because people don't necessarily ask the fundamental question: "does this increase my confidence in the code?" I think that's the most important consideration when writing tests.
I agree that tests are an insurance against changes. They find regressions and are also lights in the darkness that points to where we have to work on.
I'm moving 3 db fields (let's call them a, b, c) from 3 tables (A, B, C) to a fourth one (D) right now. Adding a, b, c to D is easy, the data migration is also easy (a single update from a CTE coalescing a, b, c from the 3 tables -- the value in a wins over b and over c). And now let's see if anything breaks (it has to.) I run the test suite. Only a dozen errors. Some are where I expected them to happen, some are totally unexpected. There are parts of the code base I forgot about in the last year or never heard about (it's a team of about 5 developers.)
Without tests, no matter the language, typing system, etc any change like that would be a nightmare.
Aside from everything else, writing tests is just as much about writing testable code. Not everything that makes code easier to test makes code better, but often times many things do; after all, nothing is easier to test than a simple, small, pure function with inputs and outputs.
I prefer trying to focus on table driven testing where I can write a single test that exercises the code various ways. This is not applicable to all types of software, but it is wonderful for things like parsers, emitters, algorithms, data structures... things that test well.
Unit tests like this are cheap but make it easy to assert that the code works. If you can assert that your individual functions do what you expect, it makes debugging and understanding software easier.
I like to think of testing as executable debugging. It’s like a debugging session that is executed over and over again. If your tests are difficult to maintain it may say something about what they are asserting or what they are testing.
When people say "testable code" I always wince. Unit tests fucking suck at testing most code. Really suck. They're ok (ok, still not great) for things like parsers and algorithms... which isn't most code.
But, rather than improving test tools to handle more types of code, developers say that they want to completely restructure the code to make it more amenable to the fact that unit tests suck. Unit tests can't handle database? You need "Testable code"! No...
Developers code to a spec, but unit tests suck at encoding a spec except in the few cases you alluded to simply because most specs are not easily, clearly encoded in the form of your turing complete programming language with its mediocre tooling.
The practical upshot of this theoretical brain damage is that when people write unit tests on code where it isn't a suitable tool it tends to be an expensive waste of time. The tests cost time to build, time to maintain and when they fail it means... "oh, you changed some code". Thanks, test.
And then religious unit testing people always argue that it wasn't the tool that was at fault it was you.
Working on projects with excellent integration test tooling really opened my eyes to the possibilities of another world - one without unit tests.
I like to think of tests and the team's emotional relationship to them as a barometer of the extent to which the repo is and has consistently followed best practices in the language. IOW, enforcing unit and functional testing forces developers to follow best practices.
If I find myself dreading writing and maintaining tests, it's a great indicator that the code diverged from best practices at some point in the past, and that the technical debt has just been accumulating ever since. Difficult to test code is code smell telling me that the technical debt is getting out of control or that the code is getting unmaintainable, and this can be difficult to walk back.
Another great indicator is the extent to which the team "detests" mocking. Mocking is great -- when it's trivial to do. When it's not trivial, and elaborate mocking behavior and reflection and testing of private methods, etc. is needed, it's another indicator that best practices have been sacrificed for expediency.
1. Tests aren't necessarily for you, they could be to convince somebody else that your solution is robust enough to be used for their use-cases.
2. Test's aren't necessarily for today. They could be about preventing code regression in the future (some dev comes in and makes "performance" changes for example, not realizing they are breaking the code for others).
3. They can be about testing that the code behaves how you believe it behaves. Sometimes even simple code requires a sanity check.
4. 100% test coverage is likely impossible, but if we do find errors, we can learn from them and try to prevent them happening in the future by adding them to the tests.
5. Tests breaking are a good thing. They inform you that some change you are making is changing the behavior of something you thought to be act reliably.
6. With regards to the same mind making the tests, this could also be a useful tool for ensuring a developer of some code thought of most the edge cases when merging some code. "Ah, i see you didn't add minus numbers to the tests, how are those handled?"
7. We should be able to freely rip out tests and add testing as the requirements of the code base change. Generally though, the goal of much code doesn't really change, despite perhaps how exactly it achieves the task does.
>100% test coverage is likely impossible, but if we do find errors, we can learn from them and try to prevent them happening in the future by adding them to the tests.
You do know there's a way to verify a function to a degree of 100% without writing a single test?
You do also realize that for even a trivial function f(x) = x + 2, the only way to achieve 100% coverage is to write:
assert f(0) == 2
assert f(1) == 3
assert f(2) == 4
assert f(3) == 5
....
assert f(N) == N + 2 where N = infinite.
There are infinite possible inputs and infinite possible outputs so to get 100% coverage you need to write infinite tests. Because your "tests" can never even approach this number most of your "coverage" really amounts to a number close to 0%.
The question is, why does testing seem to kinda sort of work even when our test coverage covers only an amount roughly equal to 0% of all inputs to the program?
It's an interesting question with an interesting answer. Suffice to say "test coverage" is a garbage statistic.
One way to look at testing is that you're taking a statistical sample of a population. You take a small sample and do a statistical measurement on that population and if all tests pass for a sample you assume that the correlations implies that the entire population of inputs will pass all tests.
So in a sense testing is just science. We try to establish correlations among a given observational data set and we assume that this is true the entire population of data including ones that aren't seen.
Except the above isn't actually true either...
The reason is most test writers don't randomly sample test input data. Their methodology is very divergent from the way a statistics expert gathers data. Test writers don't write functions that randomly pick a set of data to test... instead they're highly biased in the tests that they write... for example a typical test set can look like this:
As you can see the above two tests are highly biased with the second test picked in order to deliberately cause an error. Imagine if the test data was randomly picked? It would be highly that the random picker would draw zero as an input test case indicating that statistical sampling may not be the best way to test data....
So if our tests cases are biased then why do tests kind of sort of work? Or do they not actually work? How is the programmer picking a test and how does that influence the overall correctness of their program? Perhaps it's not the test itself or the amount of tests that the programmer is writing but it's more about how the tests influence the way the programmer thinks about the function...?
I would say the last micro service I wrote, (in python) was virtually 100% bug free ever since it went into prod. I also didn't write a single unit test for it. I did do some manual testing on the system but the application itself does not have a suite of unit tests to protect it and it has since had 0 bugs ever since it was placed into prod.
I would still say a testing suite is still good for new programmers diving into an unfamiliar system attempting to change things haphazardly, but in terms of correctness I question this strict almost religious adherence to unit testing.
Two harmful ideas I got out of university: Tests must absolutely cover as much code as possible and if something cannot be solved perfectly then any solution is wrong and useless.
I doubt that this was the intention or that anyone there teaching actually thought that, but the way things were taught to us we ended up thinking we had to test every single getter and setter. And that the traveling salesperson problem is essentially unsolvable because no efficient algorithm exists.
With a tiny bit more nuance you then find out that how much testing is useful depends on the domain/industry quite a lot, and that there are usually plenty of "good enough" solutions to seemingly impossible problems. Sweeping, extreme generalizations are for the inexperienced.
I love integration tests. You can specify your use cases right there in the code (maybe link to some official document), and often you are basically writing usage examples for your API so that someone new to the code can go straight to the tests to get a nice overview over how it's used. Regression tests can save you from looking like an idiot. I'm not going to pretend that testing is on the same level as something like formal verification, but as long as you don't overdo it I think it still has a lot of value.
It's important to remember that our own mental model is a very important part of any project. I've often read complaints about people tracking down a test failure and finding out it's actually a problem with the test and not the implementation, as if that were a waste of time. However, we need to ask the question: why did someone write that test, and do it in that way? If the answer is "because they didn't understand how the system works" then we've just exposed a bug in someone's mental model (often our own). It's not a waste of time to find and fix these failures (either by changing the test, now that we've improved our understanding of the system; or by removing it as superfluous). The article's example of checking for the round-robin algorithm rather than a lack of race conditions seems like one of these cases.
Of course, this isn't the only reason why supposedly-spurious test failures occur. If the answer to "why did someone write this test in this way?" is something like "to increase code coverage" or "they watched a talk that said to do things this way" or something equally silly then that's harder to spin as a positive. Hopefully it might at least expose failures in those development practices?
Unfortunately the developer who has to fix these failures is often someone who already understands, and wouldn't have written the test that way. Perhaps it's exposing a problem in their documentation, coding guidelines, etc.?
My 2 cents regarding testing. Written tests have 2 goals, 1: help to build your code (you verify your code as you write it). 2: Regression testing (new changes have less chances to break your code).
We as devs need to hand over/deploy a verified code. Without tests this means manual testing, without manual testing means not verified code. Manual testing means slow development process.
It's 2020, I think we have more time writing tests than before with all those cool IDEs, tools, frameworks. No tests means laziness and/or cockiness. Imho of course.
That's what you do tests or no tests. Write some code, go to the browser, check it works (assuming it's a web app). I don't think even the most undisciplined of developers just writes code and assume that it works.
>Manual testing means slow development process.
I don't think its necessarily slower, in fact initially it is faster, which is why so many places don't have much in the way of tests. Much of this depends on the size and scope of your project.
I have worked on systems with lots of tests and systems with next to no tests. The tested systems were not "better", they still contained bugs and poor abstractions. The regression tests were useful, but the team required to maintain them was even bigger than the development team.
One thing that doesn't get discussed is that tests provide a way to run bits of code independent of the rest of the (probably over complicated) system. The last two places I have worked, getting a dev environment set up to run the application took a couple of days work at least.
One strategy that I'm really enjoying at a current gig: have the person writing the front-end write the end-to-end tests (or at least the specs) for the person writing the back-end. Just having a fresh set of eyes on the tests has flushed out a lot of edge cases, and tests make a perfect place to iron out small behavioral details. It really solves the "same mind writing the code and tests" issue.
Of course, this isn't always feasible (and is a little unfair to the frontend person in terms of workload).
Wasn't this the point of "real" testing/QA teams filled with SDETs? "Professional testers" were often able to do things like this and enjoyed it. The problem was that testers were often considered lower class and thus had a self-fulfilling prophecy of decreasing talent in most companies until they were eventually cut, but for the orgs that do still have QA teams, the main point is that they bring a fresh set of eyes and are focused on breaking things in ways the designers and engineers couldn't ever think of.
The problem is that at the time you write your stuff it's not always very clear what's well written and what's poorly written. I often encounter old code written by myself that seemed like a really good idea back then but looking at it now it was just bad.
Agreed. The rant to me reads as "Writing good tests is hard. Bad tests are not worth it."
That raises the question how valuable the skill of writing-good-tests is. For quality software or is. For a career unfortunately not in many companies.
I feel in a few years people will look back and laugh at all the useless tests we wrote.
I still haven't seen a test that can prove even an identity function is correct. Yet it's obvious to the programmer.
We lack the technology to properly test software. I like to think tests are a way to implement twice and hope you got it right at least once. But we aren't even to that point yet.
Most tests aren't intended to prove correctness - that is a very high bar. They're instead intended to offer some evidence that increases our level of confidence in correctness.
Techniques that are intended to prove correctness - like for example symbolic execution - easily handle the identity function (of course!).
> I still haven't seen a test that can prove even an identity function is correct.
As with any hypothesis (empirical) testing, software testing is not about proving code is correct, but about striving to falsify that claim by poking at ways it would be likely to fail if it wasn't.
(There are formal analytical methods for proving code correct as opposed to empirical methods of falsifying it, as well, but those aren't tests.)
I wrote an LU decomposition function yesterday.
I then wrote a test that tries to reconstitute the original matrix for the LU decomposition, for a few randomly generated matrices.
That is not 'implement twice and hope you get it right at least once'. It isn't a perfect proof of correctness either, but it gives much more guarantees than a simple double implementation.
Notably, when my code to solve a linear system using LU decomposition failed, I new I didn't need to worry about my decomposition, because those tests still passed.
The crucial difference between tests and implementations is that implementations have to satisfy all of the requirements at once, while tests can check a single thing at a time. As a simple example, if we need to test a 'sort' function for lists, we might write a whole bunch of tests like this:
"Sorting" should "preserve length" in {
forAll() { (l: List[Int]) => assert(sort(l).length == l.length) }
}
"Sorting" should "be idempotent" in {
forAll() { (l: List[Int]) => assert(sort(l) == sort(sort(l))) }
"Sorting" should "put elements in order" in {
forAll() { (l: List[Int]) =>
whenever(l.nonEmpty) {
sort(l).foldLeft((true, l.min))({
case ((result, max), elem) => (result && max <= elem, elem)
}) match {
case (result, max) => assert(result && max == l.max)
}
}
}
}
"Sorting" should "not change elements" in {
forAll() { (l: List[Int) => {
val sorted = sort(l)
assert(l.all(elem => sorted.contains(elem)))
assert(sorted.all(elem => l.contains(elem)))
}
}
Each of these tests can focus on one aspect of sorting and ignore everything else. Each on its own is not enough to give us confidence in the 'sort' function (e.g. the identity function would pass the idempotence test, the same-length test and the same-elements test; returning an empty list would pass the elements-in-order test; etc.), however together they are pretty good.
In contrast, we can't implement the 'sort' function in such a piecemeal way. It needs to take every requirement into account, in case a step which satisfies one requirement invalidates the others. That's also why writing a whole new implementation to test against is best seen as a last resort.
> I like to think tests are a way to implement twice and hope you got it right at least once.
This to me is the biggest thing that makes writing good tests a "grind". For a simple, well-defined function, you can toss up a bunch of inputs and outputs and feel satisfied that all the corner cases work. For any moderately complex behavior, however, so far the only effective way I've found to test the behavior is to effectively duplicate the logic a second time in the testing code, and check to see that I got the same result; which is pretty discouraging when you're doing it.
It's certainly nice once it's done, though, to be able to make a change, re-run your test suite, and have a pretty reasonable confidence that nothing broke.
Imagine assembling a car engine without any QC process, perhaps just visual inspection of the parts. The end users (whoever will drive the car) will be the actual testers.
Also, I agree that's exactly what testing is, in fact I often now test code like that, get someone else to do one simple implementation in Python, a second high-performance implementation in C++/Rust, and compare them.
Then, as long as the same bug doesn't occur in both versions (and that seems very unlikely, as they are implemented with different algorithms), we find all the bugs in both.
In general agree. Tests are needed, but not for the sake of the tests. You need to cover main hot paths. And thats why I prefer integration testing. And ideally automated one, for example using things like GoReplay https://goreplay.org/.
One benefit of testing I haven't seen much discussed yet is that testing can make fixing bugs easier. Many times in my career, there was a bug reported in a function I am not familiar with, based on a very particular set of conditions. There is a lot of work required to simply recreate the conditions that will allow me to debug. If I have written a test, where I have created appropriate fixtures and isolated the logic, I can easily recreate the failure condition and find where my logic is broken.
This part is enough to mark the test as exploratory and not usable for a guide on the real world:
> The experiment was conducted on “live” projects that the teams were developing at that moment.
There are plenty of experiments on those conditions. If they had consistent results, we could trust those, but they are all over the place, what means that confounding factors are more relevant than the thing being studied.
Did the non-TDD team do regular testing, or no testing at all? It doesn't say, and it seems rather important to judge the results..
I think the debate is not whether testing in general reduces defects (which is rather obvious), but if adhering to a stringent Test-Driven Development (TDD) process is better than regular testing (typically adding the tests after writing the code, instead of before).
Starting your implementation with a test that's setting up, what the expected result is, and then implementing the controllers/whatever against that test is pretty satisfying.
In web development it also makes back-end development faster. Switching to a browser, reloading the page and inspecting the console/properties/page, is so slow.
I wasn't really into testing until a certain point, but having the confidence that your business logic still works after doing change requests is a bliss.
No testing it in the browser, no testing it with postman, no checking if mails are send in your dummy smtp server interface etc.
When you don't write software that changes very often or has many use cases, go ahead and leave them out, but the more you take on your shoulders, the better automating the tedious work.
Again, there is a misconception that test prove that a given module works. No, they don't.
Test may serve as some kind of "persistent REPL" to make sure you didn't miss some edge case. But, at least to me, the real value of tests is confidence for refactoring.
Otherwise it is so easy to "simplify" a function but break code due to some implicit type cast, or making it work only for positive values, etc.
I work on a codebase full of crappy unit tests that require more maintenance than the software itself. There's even a test that asserts the dockerfile matches a particular regex. It's made me realise that you should only encode your specifications in the tests, i.e check that all documented features work and that it fails gracefully outside this range. Some may not even be full end-end tests, you just have to be sensible and make them as high-level as feasible.
It seems to me that someone with this philosophy has never tried to change or refactor a subsystem of a very complicated legacy project. Sure, sometimes your tests fail because they're poorly written and test implementation details, but other times they catch the fact that you didn't know about case X that had been discovered years ago and required a minor non-obvious tweak in the code to make work correctly.
I kinda agree with the core issue here: Writing small, hermetic unit tests wont get you far, but unit tests themselves get you only so far.
As Kent C Dodds puts it, Write few tests, mostly integration.
That immediately sat with me. When u model tests based on how your software is supposed to interact, I see instant gains, i'm more confident about the code and have refactored parts of it as well.
The article doesn't mention refactoring, and that's where the value of tests lies.
Given a complete test suite and stable interfaces, you can make substantial changes to existing code and if you break anything you'll know. It's often the difference between making changes that need to be made, and avoiding those changes because there's no safety net.
I'd be curious how the author approaches refactoring.
> One way I like to consider this is to look back after a bug is found, and ask how many tests would have been needed to detect it. The minimal answer is of course one, but we can't reasonably believe that this exact test would be the next one added.
No, you can't. That's why when the bug occurs, you can add that test and then that bug never happens again.
I'm curious what kind of programs the author is writing (and not testing) here.
A few years ago, I implemented tests in a Django website I manage on the side, and am around 90%+ "coverage". Since then, I've been able to upgrade major/minor versions, add new features, and have deployed around 100 updates. Automated testing before deploying has caught numerous small typos and errors before anything went live.
On the other hand, I'm also writing some games right now, and those don't have any tests. I can't see as clear of a reason to implement them.
Would others agree that certain areas of dev are more suited for testing than others?
> Would others agree that certain areas of dev are more suited for testing than others?
I wouldn't agree at all with that. Code are rules written in way that can be translated by programs into a machine executable format.
Saying that code for some areas is more suitable for testing than code for other areas would imply that the rules are different somehow.
I don't see why that would be the case. I'd rather suggest that certain development processes and -methodologies (or lack thereof) may discourage systematic testing.
The rules that govern games are often expressed on a different level and by different people than the implementation. It's therefore hard to come up with reasonable assumptions to test in the first place.
Add to that the goals are often quite different (e.g. correct code vs the software doesn't crash and runs at an acceptable performance) and you may get the impression that tests just aren't well suited for say games.
In reality, the rules of the game, playability and fun need to be tested anyway. So from a cost-perspective it can make sense to skip out on automated code tests as errors and crashes would surface during gameplay tests anyway and overall correctness isn't a requirement for games anyhow.
I've unit tested the core domain of puzzle games and rpg games before. It's like testing business code. It's uniquely suited to that. But they can't tell you if some visual thing looks strange.
It seems like testing in games is a huge problem, in that game QA testing is a big industry. I bet puzzle games and rpg games could gain a huge competitive advantage by testing their core domain. If they're long lasting games with lots of changes coming up.
The author should put this into context of the language being used. In languages with poor separation of interface and implementation, and limited datastructure support, tests do indeed tend to be bad. That's less a fact about testing and more a fact about those languages.
[+] [-] rictic|5 years ago|reply
We write tests so that we know whether future changes have broken the system or not.
This article sounds like a reaction to the practice of writing many small, hermetic unit tests, which do little more than recapitulate the code under test. The weakest parts of a system are often in the joints, so the most important tests to write and to run are the integration tests, the ones that tell you with the most confidence whether the code works in the real world or not.
See, for example, this puzzling line from the article:
> Other tests grow obsolete because the target platform is retired, but not the test. There's even less pressure to remove stale tests than useless code. Watch as the workaround for Windows XP is removed from the code, but not the test checking it still works.
How do you remove the code for a feature, but not the test for that feature unless either you aren't running your tests or your tests aren't testing the things that you care about?
[+] [-] harryf|5 years ago|reply
Imagine some incubating startup where you have an initial 4-week runway for development, with the goal of getting a prototype working, so you can get it in the hands of some customers and begin validating a business case. How many tests do you need in this case? And how much time do you allow for it? Most of the code you're going to produce will be thrown away and the more compelling the prototype is in terms of functionality, features and design the stronger the signal you'll get back from potential customers.
To me there's still too much "testing is my religion" amongst developers and not enough "here's the quantified value of me spending 3 days writing tests and here's how that fits in the context of where our business is right now"
[+] [-] RobinL|5 years ago|reply
[+] [-] UK-Al05|5 years ago|reply
You should be testing a unit of behaviour, like a business rule or an effect to be expected. If this involves multiple objects collaborating then so be it. But they should not cross architectural boundaries like http calls, or database calls. If you can only make public api classes accessible, and make all helper classes inaccessible to users. That also forces you test things from only a public api standpoint.
Tests that just check an object calls another object, but exhibits no desired behaviour in its self are pointless, and just couple everything to the implementation. I.e Certain methods, are called in a certain order with specific parameters. When you do that, you make it really hard to change things(just design changes, not desired behaviour changes) without breaking every test.
[+] [-] Tainnor|5 years ago|reply
If you have many functional, stateless components, the unit tests can be particularly valuable.
But I think many tests end up being badly written because people don't necessarily ask the fundamental question: "does this increase my confidence in the code?" I think that's the most important consideration when writing tests.
[+] [-] pmontra|5 years ago|reply
I'm moving 3 db fields (let's call them a, b, c) from 3 tables (A, B, C) to a fourth one (D) right now. Adding a, b, c to D is easy, the data migration is also easy (a single update from a CTE coalescing a, b, c from the 3 tables -- the value in a wins over b and over c). And now let's see if anything breaks (it has to.) I run the test suite. Only a dozen errors. Some are where I expected them to happen, some are totally unexpected. There are parts of the code base I forgot about in the last year or never heard about (it's a team of about 5 developers.)
Without tests, no matter the language, typing system, etc any change like that would be a nightmare.
[+] [-] collyw|5 years ago|reply
[+] [-] jchw|5 years ago|reply
I prefer trying to focus on table driven testing where I can write a single test that exercises the code various ways. This is not applicable to all types of software, but it is wonderful for things like parsers, emitters, algorithms, data structures... things that test well.
Unit tests like this are cheap but make it easy to assert that the code works. If you can assert that your individual functions do what you expect, it makes debugging and understanding software easier.
I like to think of testing as executable debugging. It’s like a debugging session that is executed over and over again. If your tests are difficult to maintain it may say something about what they are asserting or what they are testing.
[+] [-] pydry|5 years ago|reply
But, rather than improving test tools to handle more types of code, developers say that they want to completely restructure the code to make it more amenable to the fact that unit tests suck. Unit tests can't handle database? You need "Testable code"! No...
Developers code to a spec, but unit tests suck at encoding a spec except in the few cases you alluded to simply because most specs are not easily, clearly encoded in the form of your turing complete programming language with its mediocre tooling.
The practical upshot of this theoretical brain damage is that when people write unit tests on code where it isn't a suitable tool it tends to be an expensive waste of time. The tests cost time to build, time to maintain and when they fail it means... "oh, you changed some code". Thanks, test.
And then religious unit testing people always argue that it wasn't the tool that was at fault it was you.
Working on projects with excellent integration test tooling really opened my eyes to the possibilities of another world - one without unit tests.
[+] [-] jordanpg|5 years ago|reply
If I find myself dreading writing and maintaining tests, it's a great indicator that the code diverged from best practices at some point in the past, and that the technical debt has just been accumulating ever since. Difficult to test code is code smell telling me that the technical debt is getting out of control or that the code is getting unmaintainable, and this can be difficult to walk back.
Another great indicator is the extent to which the team "detests" mocking. Mocking is great -- when it's trivial to do. When it's not trivial, and elaborate mocking behavior and reflection and testing of private methods, etc. is needed, it's another indicator that best practices have been sacrificed for expediency.
[+] [-] postalrat|5 years ago|reply
An identity function should be as testable as they come.
[+] [-] bArray|5 years ago|reply
1. Tests aren't necessarily for you, they could be to convince somebody else that your solution is robust enough to be used for their use-cases.
2. Test's aren't necessarily for today. They could be about preventing code regression in the future (some dev comes in and makes "performance" changes for example, not realizing they are breaking the code for others).
3. They can be about testing that the code behaves how you believe it behaves. Sometimes even simple code requires a sanity check.
4. 100% test coverage is likely impossible, but if we do find errors, we can learn from them and try to prevent them happening in the future by adding them to the tests.
5. Tests breaking are a good thing. They inform you that some change you are making is changing the behavior of something you thought to be act reliably.
6. With regards to the same mind making the tests, this could also be a useful tool for ensuring a developer of some code thought of most the edge cases when merging some code. "Ah, i see you didn't add minus numbers to the tests, how are those handled?"
7. We should be able to freely rip out tests and add testing as the requirements of the code base change. Generally though, the goal of much code doesn't really change, despite perhaps how exactly it achieves the task does.
[+] [-] leafboi|5 years ago|reply
You do know there's a way to verify a function to a degree of 100% without writing a single test?
You do also realize that for even a trivial function f(x) = x + 2, the only way to achieve 100% coverage is to write:
There are infinite possible inputs and infinite possible outputs so to get 100% coverage you need to write infinite tests. Because your "tests" can never even approach this number most of your "coverage" really amounts to a number close to 0%.The question is, why does testing seem to kinda sort of work even when our test coverage covers only an amount roughly equal to 0% of all inputs to the program?
It's an interesting question with an interesting answer. Suffice to say "test coverage" is a garbage statistic.
One way to look at testing is that you're taking a statistical sample of a population. You take a small sample and do a statistical measurement on that population and if all tests pass for a sample you assume that the correlations implies that the entire population of inputs will pass all tests.
So in a sense testing is just science. We try to establish correlations among a given observational data set and we assume that this is true the entire population of data including ones that aren't seen.
Except the above isn't actually true either...
The reason is most test writers don't randomly sample test input data. Their methodology is very divergent from the way a statistics expert gathers data. Test writers don't write functions that randomly pick a set of data to test... instead they're highly biased in the tests that they write... for example a typical test set can look like this:
As you can see the above two tests are highly biased with the second test picked in order to deliberately cause an error. Imagine if the test data was randomly picked? It would be highly that the random picker would draw zero as an input test case indicating that statistical sampling may not be the best way to test data....So if our tests cases are biased then why do tests kind of sort of work? Or do they not actually work? How is the programmer picking a test and how does that influence the overall correctness of their program? Perhaps it's not the test itself or the amount of tests that the programmer is writing but it's more about how the tests influence the way the programmer thinks about the function...?
I would say the last micro service I wrote, (in python) was virtually 100% bug free ever since it went into prod. I also didn't write a single unit test for it. I did do some manual testing on the system but the application itself does not have a suite of unit tests to protect it and it has since had 0 bugs ever since it was placed into prod.
I would still say a testing suite is still good for new programmers diving into an unfamiliar system attempting to change things haphazardly, but in terms of correctness I question this strict almost religious adherence to unit testing.
Food for a thought.
[+] [-] Netcob|5 years ago|reply
I doubt that this was the intention or that anyone there teaching actually thought that, but the way things were taught to us we ended up thinking we had to test every single getter and setter. And that the traveling salesperson problem is essentially unsolvable because no efficient algorithm exists.
With a tiny bit more nuance you then find out that how much testing is useful depends on the domain/industry quite a lot, and that there are usually plenty of "good enough" solutions to seemingly impossible problems. Sweeping, extreme generalizations are for the inexperienced.
I love integration tests. You can specify your use cases right there in the code (maybe link to some official document), and often you are basically writing usage examples for your API so that someone new to the code can go straight to the tests to get a nice overview over how it's used. Regression tests can save you from looking like an idiot. I'm not going to pretend that testing is on the same level as something like formal verification, but as long as you don't overdo it I think it still has a lot of value.
[+] [-] chriswarbo|5 years ago|reply
Of course, this isn't the only reason why supposedly-spurious test failures occur. If the answer to "why did someone write this test in this way?" is something like "to increase code coverage" or "they watched a talk that said to do things this way" or something equally silly then that's harder to spin as a positive. Hopefully it might at least expose failures in those development practices?
Unfortunately the developer who has to fix these failures is often someone who already understands, and wouldn't have written the test that way. Perhaps it's exposing a problem in their documentation, coding guidelines, etc.?
[+] [-] rooam-dev|5 years ago|reply
My 2 cents regarding testing. Written tests have 2 goals, 1: help to build your code (you verify your code as you write it). 2: Regression testing (new changes have less chances to break your code).
We as devs need to hand over/deploy a verified code. Without tests this means manual testing, without manual testing means not verified code. Manual testing means slow development process.
It's 2020, I think we have more time writing tests than before with all those cool IDEs, tools, frameworks. No tests means laziness and/or cockiness. Imho of course.
[+] [-] collyw|5 years ago|reply
That's what you do tests or no tests. Write some code, go to the browser, check it works (assuming it's a web app). I don't think even the most undisciplined of developers just writes code and assume that it works.
>Manual testing means slow development process.
I don't think its necessarily slower, in fact initially it is faster, which is why so many places don't have much in the way of tests. Much of this depends on the size and scope of your project.
I have worked on systems with lots of tests and systems with next to no tests. The tested systems were not "better", they still contained bugs and poor abstractions. The regression tests were useful, but the team required to maintain them was even bigger than the development team.
One thing that doesn't get discussed is that tests provide a way to run bits of code independent of the rest of the (probably over complicated) system. The last two places I have worked, getting a dev environment set up to run the application took a couple of days work at least.
[+] [-] afarrell|5 years ago|reply
Some tests should exist as scaffolding while you are working on a project and then you can throw them out afterwards.
[+] [-] tomlagier|5 years ago|reply
Of course, this isn't always feasible (and is a little unfair to the frontend person in terms of workload).
[+] [-] oldmanhorton|5 years ago|reply
[+] [-] diob|5 years ago|reply
Poorly written tests make refactoring nigh impossible, and usually don't contribute to anything other than code coverage metrics.
Focus on quality, not quantity.
[+] [-] treve|5 years ago|reply
[+] [-] postalrat|5 years ago|reply
High level refactoring means writing new tests. Which often kills the effort before you begin.
[+] [-] spaetzleesser|5 years ago|reply
[+] [-] qznc|5 years ago|reply
That raises the question how valuable the skill of writing-good-tests is. For quality software or is. For a career unfortunately not in many companies.
[+] [-] postalrat|5 years ago|reply
I still haven't seen a test that can prove even an identity function is correct. Yet it's obvious to the programmer.
We lack the technology to properly test software. I like to think tests are a way to implement twice and hope you got it right at least once. But we aren't even to that point yet.
[+] [-] caf|5 years ago|reply
Techniques that are intended to prove correctness - like for example symbolic execution - easily handle the identity function (of course!).
[+] [-] dragonwriter|5 years ago|reply
As with any hypothesis (empirical) testing, software testing is not about proving code is correct, but about striving to falsify that claim by poking at ways it would be likely to fail if it wasn't.
(There are formal analytical methods for proving code correct as opposed to empirical methods of falsifying it, as well, but those aren't tests.)
[+] [-] rocqua|5 years ago|reply
That is not 'implement twice and hope you get it right at least once'. It isn't a perfect proof of correctness either, but it gives much more guarantees than a simple double implementation.
Notably, when my code to solve a linear system using LU decomposition failed, I new I didn't need to worry about my decomposition, because those tests still passed.
[+] [-] chriswarbo|5 years ago|reply
In contrast, we can't implement the 'sort' function in such a piecemeal way. It needs to take every requirement into account, in case a step which satisfies one requirement invalidates the others. That's also why writing a whole new implementation to test against is best seen as a last resort.
[+] [-] donw|5 years ago|reply
Tests exist to help prevent you from changing behavior unknowingly.
[+] [-] gwd|5 years ago|reply
This to me is the biggest thing that makes writing good tests a "grind". For a simple, well-defined function, you can toss up a bunch of inputs and outputs and feel satisfied that all the corner cases work. For any moderately complex behavior, however, so far the only effective way I've found to test the behavior is to effectively duplicate the logic a second time in the testing code, and check to see that I got the same result; which is pretty discouraging when you're doing it.
It's certainly nice once it's done, though, to be able to make a change, re-run your test suite, and have a pretty reasonable confidence that nothing broke.
[+] [-] rooam-dev|5 years ago|reply
[+] [-] CJefferson|5 years ago|reply
Also, I agree that's exactly what testing is, in fact I often now test code like that, get someone else to do one simple implementation in Python, a second high-performance implementation in C++/Rust, and compare them.
Then, as long as the same bug doesn't occur in both versions (and that seems very unlikely, as they are implemented with different algorithms), we find all the bugs in both.
[+] [-] LeonidBugaev|5 years ago|reply
[+] [-] grink71|5 years ago|reply
[+] [-] rooam-dev|5 years ago|reply
IBM team did 40% fewer defects than non-TDD team, Microsoft team: 60–90% (fewer defects than non-TDD fellows).
You can read more here: https://medium.com/crowdbotics/tdd-roi-is-test-driven-develo...
[+] [-] marcosdumay|5 years ago|reply
> The experiment was conducted on “live” projects that the teams were developing at that moment.
There are plenty of experiments on those conditions. If they had consistent results, we could trust those, but they are all over the place, what means that confounding factors are more relevant than the thing being studied.
[+] [-] redbar0n|5 years ago|reply
I think the debate is not whether testing in general reduces defects (which is rather obvious), but if adhering to a stringent Test-Driven Development (TDD) process is better than regular testing (typically adding the tests after writing the code, instead of before).
[+] [-] myspy|5 years ago|reply
In web development it also makes back-end development faster. Switching to a browser, reloading the page and inspecting the console/properties/page, is so slow.
I wasn't really into testing until a certain point, but having the confidence that your business logic still works after doing change requests is a bliss.
No testing it in the browser, no testing it with postman, no checking if mails are send in your dummy smtp server interface etc.
When you don't write software that changes very often or has many use cases, go ahead and leave them out, but the more you take on your shoulders, the better automating the tedious work.
[+] [-] stared|5 years ago|reply
Test may serve as some kind of "persistent REPL" to make sure you didn't miss some edge case. But, at least to me, the real value of tests is confidence for refactoring.
Otherwise it is so easy to "simplify" a function but break code due to some implicit type cast, or making it work only for positive values, etc.
[+] [-] alephu5|5 years ago|reply
[+] [-] ilyanep|5 years ago|reply
[+] [-] kalyantm|5 years ago|reply
[+] [-] aazaa|5 years ago|reply
Given a complete test suite and stable interfaces, you can make substantial changes to existing code and if you break anything you'll know. It's often the difference between making changes that need to be made, and avoiding those changes because there's no safety net.
I'd be curious how the author approaches refactoring.
[+] [-] jasonlfunk|5 years ago|reply
No, you can't. That's why when the bug occurs, you can add that test and then that bug never happens again.
[+] [-] lattalayta|5 years ago|reply
A few years ago, I implemented tests in a Django website I manage on the side, and am around 90%+ "coverage". Since then, I've been able to upgrade major/minor versions, add new features, and have deployed around 100 updates. Automated testing before deploying has caught numerous small typos and errors before anything went live.
On the other hand, I'm also writing some games right now, and those don't have any tests. I can't see as clear of a reason to implement them.
Would others agree that certain areas of dev are more suited for testing than others?
[+] [-] qayxc|5 years ago|reply
I wouldn't agree at all with that. Code are rules written in way that can be translated by programs into a machine executable format.
Saying that code for some areas is more suitable for testing than code for other areas would imply that the rules are different somehow.
I don't see why that would be the case. I'd rather suggest that certain development processes and -methodologies (or lack thereof) may discourage systematic testing.
The rules that govern games are often expressed on a different level and by different people than the implementation. It's therefore hard to come up with reasonable assumptions to test in the first place.
Add to that the goals are often quite different (e.g. correct code vs the software doesn't crash and runs at an acceptable performance) and you may get the impression that tests just aren't well suited for say games.
In reality, the rules of the game, playability and fun need to be tested anyway. So from a cost-perspective it can make sense to skip out on automated code tests as errors and crashes would surface during gameplay tests anyway and overall correctness isn't a requirement for games anyhow.
[+] [-] UK-Al05|5 years ago|reply
It seems like testing in games is a huge problem, in that game QA testing is a big industry. I bet puzzle games and rpg games could gain a huge competitive advantage by testing their core domain. If they're long lasting games with lots of changes coming up.
[+] [-] lmm|5 years ago|reply