While I understand the motivation for disambiguating "checking" from "testing" conceptually, I have some concerns about how this framework may play out in practice. What constitutes a "mechanical" check vs a human "exploration", for example, could likely be a source of endless debate. And is it truly useful to separate the two when most testing undoubtedly involves elements of both? A more pragmatic definition may be preferable to one that sets up a dubious distinction prone to subjective interpretation. Nonetheless, I appreciate the attempt to enrich discussion around strategic testing methodologies at a time when clear terminology is needed to match the complexity of modern software systems.
satisfice|2 years ago
"Mechanical" means algorithmic. It's what Turing meant when he wrote "The idea behind digital computers may be explained by saying that these machines are intended to carry out any operations which could be done by a human computer. The human computer is supposed to be following fixed rules; he has no authority to deviate from them in any detail. We may suppose that these rules are supplied in a book, which is altered whenever he is put on to a new job." See: https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf
Checking is an evaluation activity that can be COMPLETELY automated.
Testing is a social activity done by humans. It cannot be automated because it incorporates social judgement. Social judgement requires collective tacit social knowledge. See: The Shape of Actions, by Harry Collins for a very deep dive into that.
A human exploration is distinguished by the presence of choosing. Humans make choices. You could claim that human choices are an epiphenomenon of bio-mechanics, if you want to. I actually believe that, myself. However, we are speaking of the social/practical realm. In this realm we perceive each other and treat each other as if we are making free choices (although perhaps influenced by known sources of bias). Thus the difference between exploring and checking is that, in every moment, a check is controlled by an explicit script that is knowable in principle; whereas in exploration, a person is forming new thoughts and theories, through a process of real-time sensemaking (otherwise known as learning).
Checking is a process of operating a product and verifying a particular fact. A check does not change the checker.
Exploration is a process of interacting, experiencing, and being transformed by this process. Exploration grows the explorer.
satisfice|2 years ago
Let's say some developer runs his "unit tests" as part of a build process. Let's say that they are automatically kicked off, as part of the pipeline. This developer may say "I've done testing."
I would say, well, we don't know yet. It depends on the answer to these questions:
1. Do you fully understand the unit-level checking code that is being run? Did you write it yourself, for instance, and do you remember what it does and does not do? (IF YES, IT MIGHT BE TESTING, BECAUSE IT'S POSSIBLE THAT YOU COULD MAKE A REASONABLE JUDGMENT ABOUT THE COVERAGE AND VALUE OF THIS PROCESS)
2. Do you fully understand the intent and logic behind the code? In other words, do you understand why the code was written as it was instead of doing something completely different? (IF YES, IT MIGHT BE TESTING, BECAUSE IT'S POSSIBLE THAT YOU COULD MAKE A REASONABLE JUDGMENT ABOUT THE COVERAGE AND VALUE OF THIS PROCESS)
3. Were the automated checks performed correctly? Can you be reasonably sure of this? Did you supervise them in progress or otherwise receive log output that serves as compelling evidence? (IF YES, IT MIGHT BE TESTING, BECAUSE IT'S POSSIBLE THAT YOU COULD MAKE A REASONABLE JUDGMENT ABOUT THE COVERAGE AND VALUE OF THIS PROCESS)
4. Have you reviewed the test results? Did your code collect any metadata or raw data of any kind that may indicate problems beyond those that were the immediate subject of verification? (IF YES, IT MIGHT BE TESTING, BECAUSE IT'S POSSIBLE THAT YOU COULD MAKE A REASONABLE JUDGMENT ABOUT THE COVERAGE AND VALUE OF THIS PROCESS)
So, a checking process becomes testing when a tester applies his judgement to the situation. Otherwise, it's just checking in a vacuum.
Oh, and I'm only talking about the situation where everything is "green." If it's not green, then for testing to be happening, someone must investigate.
An example of checking that is not testing is when my car's "check engine" light comes on. I am not qualified to interpret this event, other than to take me care to a repair shop. When the repair guy looks at it, that check becomes a test, and then he acts on the result of that test.
drewcoo|2 years ago
What is "testing?" Well James (or Cem or . . .) will proclaim the definition in an enthusiastic and self-assured way and then tell you how you're not doing their just-made-up definition. It feels a bit like a self-help seminar. And I'm not just saying that because you also have books available in the lobby.
And the specific claims are over the top, too. "If you didn't watch your 'test' run, it may not have been 'testing'" isn't very helpful. Especially when you're talking about dev unit tests.