Bugs are compelling critters. Through the thousands of hours I have spent hunting, capturing, and describing them to others, I have grown interested in their ecology and that of their natural habitat: software development. The setting, it would be fair to point out, makes me a quality assurance engineer, not an entomologist. But, when I sit down to test, in a way, I am donning hiking boots, grabbing a field kit, and crawling into the underbrush in search of bugs. Granted, there are other possible analogies for Quality Assurance (QA) as a profession: detective, reporter, critic, IRS auditor. But, my favorite is scientist because QA actually does borrow from scientific methodology. And, for one whose college years were dedicated to biology, this is a happy coincidence.
QA engineers are tasked with entering an often strange and complex environment (i.e., a new project or product phase) and observing not only what is known about it, but also what might be lurking in a vast landscape of test cases—combinations of user actions that are like leaves under which bugs might hide. If we don’t turn over the right ones, it’s a good bet that our clients or their customers will. But, reporting isolated observations is not much of a contribution—no more than “I saw a flash of purple!” describes a new species of beetle. QA needs to characterize bugs or, put another way, present a theory of the bug. Some bugs are easy to capture—out in the open and slow moving—while other bugs are concealed in thickets of complexity. Figuring out how to handle them is where testing software can get unexpectedly scientific.
A decade ago, while testing mp3 playback at a chip company, I heard my test machine produce terribly garbled music—not only was it a nasty bug, but an apparently intermittent one. The eventual explanation—a software defect exposed by tiny, corrupted sections of some mp3 files—was not possible until I discovered and reported a key fact: The garbled audio only (and always) happened if playback was started at very specific points within afflicted mp3 files. If I played them from the beginning, they sounded fine. But, how did I know this? In his article, “The Test Case as a Scientific Experiment,” test manager David Coutts argued that “an understanding of science, and of the scientific method, is essential to an understanding of software testing and the methodology of testing.” So, thank you to Charles Darwin here—not for his “survival of the fittest” theory, but for the hypothetico-deductive method. All bug reports, even the difficult ones, start with an observation: something that looks or sounds wrong, where numbers don’t add up, a font is wrong, or an error message appears. Even merely distracting quirks, inconveniences, or fleeting, weird behaviors can count as bug signifiers. The tester checks and rechecks, approaching from different angles and taking notes. At first, the result is just a collection of data. But eventually, a hypothesis might suggest itself; then the prediction is tested to understand if it supports or contradicts the hypothesis. This process is the hypothetico-deductive method, which has been part of science for about 150 years. People use it every day without realizing it, but I like to claim it as part of QA’s field kit.
Still, science tends to seed technology, and we’re not “out there” studying bugs as an academic exercise. We’re playing the role of customer in order to get the experience right for the real thing. Being first to try out unreleased software can be routine, but it can also be surprising and occasionally baffling. This is part of the nature of software development. Good QA, like science, requires creativity—or at least improvisation. Using a cookie-cutter approach, with only a grid of test cases to execute, limits the value that can be brought to the project. The number of tests that can be run is practically infinite, but not all are fruitful for finding important defects. Even the most experienced engineers do not think to report every test case that could expose critical defects. One time, I closed my notebook computer to move to another desk, which was not a planned step in the test cases I was executing, but perhaps should have been. When I reopened the notebook, the screen remained black, and I realized I had a bug.
Test plans are certainly useful for making sure basic components and functions are checked, but QA always benefits from attention to detail informed by creative inquisitiveness. “What if” scenarios unfold best on-the-fly while trying things with the product.
While QA Engineers benefit from scientific and creative thinking, we still need to resist other instincts (and pressures) such as troubleshooting (rather than reporting) problems, going easy on the product, and sticking with familiar methods. Testers may be part engineer, part user, and, some would add, part troublemaker. But one of the biggest qualities of QA is curiosity. My favorite element to being a QA is still the sense of discovery and the satisfaction in tracking down, isolating, and reporting a previously unknown species of bug. Especially if it’s big and scary.