As mentioned above, the test catalog contains the collected idioms for
testing common coding situations. The catalog stores the idioms in
such a way that the Clue Extractor and Test Design Tool can find them
based on the ``raw'' data types involved (ie:
int, etc.), or based on the semantic types involved (ie:
Take the unassuming
IP Address as an example.There are tests
and questions about the ``Stringness'' of the data (if it is
represented as a string), about the ``IP Addressness'' of the data,
and about higher level contextual concepts, such as what subnet mask
is associated, in what form the address may be represented, whether or
not it's ``classful'' or ``classless'', and so on. This range points
out a side benefit of the Test Catalog: it is not simply a place to
store test information, it is a shared repository for extensive domain
knowledge. Because the catalog contains the sum of all of the
knowledge ever stored in it, and because it offers this knowledge in
direct correspondence to system objects, methods, and data types, it
may prompt pre-emptive revisions to the system well in advance of
A simple example is that of the relationship between the IP address and
its subnet-mask. A well-stocked test catalog, when encountering an
IP Address datum, will add to the list of test specifications
notes to the human designer about how IP addresses and subnet masks
should ``fit together''. It will offer references for further study,
references to well-tested library code to manipulate IP addresses and
subnet masks, and, of course, suggested tests themselves.
In the above scenario, the test catalog offers the system designers and implementors a chance to preempt bugs by sending them in the direction of stable library code to solve common problems. In such cases where the library code is insufficient or unusable (whether because of license restrictions, or lack of fitness to the task, or some other reason entirely), the test catalog offers idioms for approaching the problem in new code. Once the code is written, whether by library or by the development team, the test catalog offers a suite of ready-made tests and testing idioms for the creation of new tests.
By allowing experts a single repository for describing idioms, documenting specialized micro-domains, and providing code to both implement and test the idioms, the test catalog allows system designers and implementors who are not experts in the micro-domains to benefit from shared expertise without explicitly ``going in search of'' the wisdom.
It's far easier to roughly describe your system both in its native language and in a semi-formal specification language, then have the test catalog quiz you based on what relevant expertise it finds, than to proactively research every detail with excruciating diligence.