Figure 2 - Replacing broadcast networks
Formal methods provide provable coverage and correctness of the resulting test suite. Nevertheless, they are not very common in practical testing. Most test suites for OSPF conformance testing are designed by reading the standard and extracting the relevant parts. Although the test suites which are developed this way cannot be proven to cover the whole specification, they provide a sufficient coverage of the standard as well. Especially those test suites which are used by many vendors provide a good coverage, because they are used extensively. So the probability is high, that missing or incorrect tests are found soon.
To be able to package and distribute a the test suite with SSFNet, no external devices or additional software can be used. Since these systems are explicitly designed to test real devices, it would require some effort to integrate them into SSFNet -- if this is possible at all. Another negative aspect of these commercial products is their price.
Most vendors put their products into lab testing at some time. One popular testing lab is the InterOperability Lab at University of New Hampshire. The IOL has developed various test suites for different protocols. One of these protocols is OSPF.
The IOL test suite description is publicly available. All the tests of the IOL test suite consist of a test scenario description together with a test procedure. The test purposes and test procedures of the test cases are intuitively clear. DML configurations may be directly obtained from the scenario descriptions. That makes it easy to implement these tests in SSFNet. Since the single test cases are very simple, they can be used for testing during development as well. However, there are some features, that are not covered by the IOL test suite. There are only tests for broadcast networks -- tests specific to point-to-point networks are not taken into account. For most of the test cases, this does not play a role, since the network type only affects the addressing of packets and the decision, when to build an adjacency. The tests for exactly these features have to be repeated, when broadcast networks are implemented. For the remaining tests, broadcast networks can be replaced by a set of point-to-point links (see Figure 2). Other tests, missing from the IOL test suite are tests for the simple calculation of intra area routes, using the Dijkstra algorithm. However, appropriate tests can be added easily to complete the test suite.
Although some tests are missing from the IOL test suite, it is a good basis for practical testing early in the development. It can be easily implemented in the SSFNet environment. Many vendors use the IOL test service -- and with this the IOL test suite -- when testing their products. This implies, that the IOL test suite provides a reasonable coverage and correctness for conformance testing. These considerations led to the decision to use the IOL test suite as a basis for testing the OSPF implementation in SSFNet. Whenever necessary, additions or modifications are made to meet the specific requirements in the SSFNet environment.