Figure 1 - SSFNet testing architecture
The ISO 9646 standard provides basic concepts for conformance testing of protocols. It defines standardized test architectures and procedures. A protocol implementation usually provides services at a particular layer and uses services of the layer below. The protocol is called an N-service provider. The lower layer is the N-1 service. All protocol implementations must be tested as black boxes. Tests are driven by the observation of input/output behavior and not by the observation of internal states. A component, which stimulates and observes an implementation at the level of the N-1 service is called a lower tester (LT). A component, which uses the N-service provided by the implementation itself to test the implementation, is called an upper tester (UT). These concepts can be easily transferred into conformance testing within the SSFNet framework. Since the OSI methods are independent of any particular system, the test procedure needs not to be modified. The following abstract test architecture can be used to test arbitrary protocols in SSFnet:
In SSFNet, the protocol implementation (IUT) is realized as a protocol session. The upper tester (UT) can also be realized as a protocol session in the protocol graph of he IUT (e.g. the UT for a TCP session could be a HTTP session). The lower testers (LT) are placed in their own protocol graphs and can also be implemented in appropriate protocol sessions. The data exchanged between the testers and the IUT are protocol messages. The underlying service can be any lower level protocol that has already been implemented and tested in SSNet (e.g. IP, UDP or TCP). A closer look at the OSI test procedure shows, that there are four main phases:
In the OSI testing methodology, each test case has a preamble, which brings the IUT from an initial state into an appropriate state to run the test (e.g. establish a connection). After that, the test is run. Finally, there is a postamble, which brings the IUT back into the initial state. In the SSFNet framework, this postamble can be omitted, since each test case is executed in a different simulator run. That guarantees, that the IUT is definitely in the initial state, when a test begins.
The execution of the test suite will produce a set of test logs. Since the implementation is tested as a black box, the only source of information for the evaluation of the test results are the packets exchanged between the testers and the IUT, which can be observed at the PCOs. Therefore the testers must log these packets in an appropriate way (e.g. dump packets in {\em tcpdump} format). However, for some tests (e.g. verification of routing tables in routing protocols) it could be useful to have logs of internal datastructures as well. For this, the simulation log can be used as a source of information as well. Since the test suites should be independent of a specific implementation in order to reuse them for other implementations of the same protocol, this possibility should be sparely used.
After the execution of the test suite, the logs must be analyzed and a verdict must be assigned. This can either be done manually or automatically. When a user installs SSFNet on a machine, he might want to validate if SSFNet works properly in this environment. Since a user does not necessarily have knowledge on how to analyze the logs, the automatic evaluation of logs should be preferred.