3 The Test Catalog in Detail (and By Example)

As mentioned above, the test catalog contains the collected idioms for testing common coding situations. The catalog stores the idioms in such a way that the Clue Extractor and Test Design Tool can find them based on the ``raw'' data types involved (ie: String, int, etc.), or based on the semantic types involved (ie: IP Address, Phone Number).

Take the unassuming IP Address as an example.There are tests and questions about the ``Stringness'' of the data (if it is represented as a string), about the ``IP Addressness'' of the data, and about higher level contextual concepts, such as what subnet mask is associated, in what form the address may be represented, whether or not it's ``classful'' or ``classless'', and so on. This range points out a side benefit of the Test Catalog: it is not simply a place to store test information, it is a shared repository for extensive domain knowledge. Because the catalog contains the sum of all of the knowledge ever stored in it, and because it offers this knowledge in direct correspondence to system objects, methods, and data types, it may prompt pre-emptive revisions to the system well in advance of actual testing.

A simple example is that of the relationship between the IP address and its subnet-mask. A well-stocked test catalog, when encountering an IP Address datum, will add to the list of test specifications notes to the human designer about how IP addresses and subnet masks should ``fit together''. It will offer references for further study, references to well-tested library code to manipulate IP addresses and subnet masks, and, of course, suggested tests themselves.

In the above scenario, the test catalog offers the system designers and implementors a chance to preempt bugs by sending them in the direction of stable library code to solve common problems. In such cases where the library code is insufficient or unusable (whether because of license restrictions, or lack of fitness to the task, or some other reason entirely), the test catalog offers idioms for approaching the problem in new code. Once the code is written, whether by library or by the development team, the test catalog offers a suite of ready-made tests and testing idioms for the creation of new tests.

By allowing experts a single repository for describing idioms, documenting specialized micro-domains, and providing code to both implement and test the idioms, the test catalog allows system designers and implementors who are not experts in the micro-domains to benefit from shared expertise without explicitly ``going in search of'' the wisdom.

It's far easier to roughly describe your system both in its native language and in a semi-formal specification language, then have the test catalog quiz you based on what relevant expertise it finds, than to proactively research every detail with excruciating diligence.