In what kind of situations have people found all-pairs useful? When can we be comfortable that the theory of error that underlies all-pairs is likely to hold true? I find that it’s not that often for me. Am I just over-exercising my tester sense of doubt?
Here’s an example of how I applied All Pairs at a previous company. This was a C++ app that ran on a handful of OSs and DBs. It’s a fairly small and straightforward matrix, but it illustrates some of my thinking about All Pairs testing. The matrix below is NOT what we really did, but from memory I think it’s not far off:
A few things to note:
- Tools to randomly generate an All Pairs matrix are A Good Thing, but I would generally use the generated matrix as a starting point, modifying it as appropriate. In this case, I had a small enough set of options that I had no reason to auto-generate it. Instead, it made sense to think about my context (what combinations do more of our customers use? What do our highest paying customers use?) rather than fill it randomly.
- Many of the combos we didn’t test at all were unsupported or impossible combinations (e.g. Linux+MSSQLServer) , but
- One glaring exception was Solaris 10 on Oracle — a platform we supported but never tested on.
Not testing on one of my supported platforms? Why would I do such a thing? And then go ahead and admit it in public? For a long time we were behind the times on Solaris, supporting older versions but not the latest. There wasn’t much clamor for it – not enough to be losing sales – but a few adventurous customers asked about upgrading to it. In reading Sun’s docs, I found the strongest statement I’ve ever seen about backwards compatibility – guaranteeing that if any software that ran on 9 didn’t work on 10, that Sun would release a patch to fix the incompatibility in 10. At first we just told a few eager customers, “we don’t support 10 yet, but here’s a link to what Sun says. If you want to experiment, feel free and let us know what happens” Several of them chose to try it. Six months later we still didn’t have a Solaris 10 server in house, but we had a good deal of feedback from those customers that all was well from their perspective, we did some further research leading to more encouraging reports about Solaris 10’s backwards compatibility, and finally we decided we were confident enough to declare public support.
Now, I’m sure that there are folks who would say that that was a foolish risk to take. Looking back I’m still happy with the decision…for our particular product, in our particular market, with our particular time and budget limitations.
It’s important to remember that deciding what (not) to test is always risk management. All Pairs is often a useful technique, but of course it doesn’t guarantee that an application will work on all the other platforms. In contrast to the application I’m describing, a friend of mine tests a tool that works at a very low protocol level on a staggering list of OSs, and he’s learned from experience that he really has to run every test on every platform. In my context All Pairs (and Sun’s track record of backward compatibility) mitigated the risk sufficiently that I chose to put my limited time into other tests – and I believe we caught more significant bugs as a result.