Design of the Test Suite
The IOL test suite can be implemented with regards to the concepts discussed before. For
OSPF testing, an upper tester is not needed, since OSPF does not provide any service to an
upper layer. There are multiple lower testers, depending of the test case. Each tester can be
represented by a router in the DML configuration. These routers run different protocol sessions,
dependent of their function during the test. Most testers run normal protocol sessions (e.g. IP,
ICMP and OSPF) in order to model the behavior of a normal router. This is sufficient to test
most parts of the OSPF specification. Nevertheless, some tests require a non-standard behavior
to verify, that a router correctly reacts on invalid or erroneous packets. These behaviours
must be modeled in specialized testers, which must extend or modify the behavior of the standard
protocol sessions.
For the OSPF test suite, three classes of non-standard behavior are needed for testers.
In general, all topologies are configured statically in SSFNet, which is sufficient for
current applications. In order to test the behavior of OSPF when the topology changes
(i.e. links fail, etc.), these topology changes must be simulated by a tester. The second
class of test behavior is to introduce simple errors into all packets of the same type
(e.g. modify the network mask in all hello packets). Such behavior can be used to test
how OSPF deals with packets, that contain invalid values in some fields. The third class
of behaviors is needed to model more complex tests. Some tests require the tester to be
aware of the state of an adjacency and to modify only cetain packets or originate new
ones dependent of this state (e.g. modify the sequence number of the fifth database
description packet). For some test cases it may also be necessary to combine these
behaviors. So, all the test behaviors are implemented in appropriate protocol sessions,
which can be combined in a protocol graph. All the protocol sessions for the testers
are implemented in the SSF.OS.OSPFv2.test package.
Structure of the Test Suite
The OSPF functionality can be divided into several areas, which are mostly independent of
each other. These areas are
- the hello protocol,
- database exchange and flooding,
- origination of different LSA types and
- routing calculations.
Additionally, {\em general configuration and packet formatting} issues can be seen as a
separate group. The IOL test suite is divided in exactly these groups. So the test suite
can be seen as a collection of five subsidiary test suites, which can be executed mostly
separate from the others. Each of these five test suites again consists of several test
cases.
Structure of the Test Cases
Each of the test cases consists of a test setup and a test procedure. The
test setup describes the network topology for this test. The test procedure describes
in detail, how the routers should behave and which results should be observed. A basic
DML configuration can be directly derived from the test setup. A detailed configuration
and required behavior of the components can be obtained from the test procedure. During
the simulation of a test scenario, OSPF packets must be logged, where necessary. For the
evaluation of the test results, these logs can then be analyzed and a verdict can be
assigned. In summary, the execution of each test case is divided into two phases:
- simulation of the scenario
- evaluation the logfiles and assignment of a verdict
The evaluation of results is done automatically by a PERL script which is provided with
every test case. These scripts extract the necessary information from the logfiles of a
test scenario and generate a verdict out of this information.
Assigning Verdicts
There are three possible verdicts, which can be assigned to every test case. PASS
means, that the implementation performed exactly as expected. FAIL is assigned,
when there were offences against the specification. The INCONCLUSIVE verdict is
assigned, if it is not clear if the implementation complies to the specification. This
can be the case when log files are missing or some prerequisites for the test are not
fulfilled (e.g. when testing sequence numbering in the database description process,
but no database description packets are sent at all). The evaluation of logfiles strongly
depends on the test cases, so every test case has its own evaluation script. For each
of the test cases a file named \texttt{report.log} is produced, which contains information
about which test steps passed or failed. This file also contains the final verdict for
the test case. To be able to evaluate the reports automatically, the final verdict is
contained in a line which is formatted as follows:
### VERDICT for <test case name>: <PASS|FAIL|INCONCLUSIVE> ###
When all test cases of a test suite have been executed, all the verdicts must be examined to
assign a final verdict for this test suite. When all subsidiary test suites have been
executed, the verdicts for the particular test suites are examined and a final conformance
statement is produced. Another PERL script (\texttt{verdict.pl}) is used for the assignment
of verdicts for a test suite.
verdict.pl takes the names of the test cases as parameters. The reports of all
test cases specified in the command line are scanned. If this report exists, it is
scanned for the above line, which contains the verdict for the test case. If the file
does not exist or if it doesn't contain a line as specified above, the test case result
is considered INCONCLUSIVE. The syntax of verdict.pl is as follows:
verdict.pl <test_suite_name> <report_file> <test1> ... <testn>
The final verdict for the test suite is generated as follows: If all test cases have passed,
the whole test suite gets a PASS verdict. If one or more FAIL verdicts have been
found, the whole test suite gets a FAIL verdict. If there are one or more
INCONCLUSIVE verdicts, but no FAIL verdict, the whole test suite's verdict
will be INCONCUSIVE. verdict.pl also generates an output, which contains a
verdict line as specified above. This output can again be redirected into a file called
report.log. So verdict.pl can also be used to produce the final conformance
statement.
The Testsuite Dictionary
Many of the test scenarios require the same or similar configurations for some components.
These common configurations of components are stored in the testsuite dictionary
testsuite.dml to keep the test case DML configurations simple. The dictionary
consists of several parts:
- The interfaces part defines some common interface types (e.g. 100MBit or 1GBit).
- The protocols part defines common configurations for ProtocolSessions.
- Common router configurations are contained in the routers section.
- The \em networks part finally contains several networks of different sizes.
In the routers section there are common router configurations with different numbers of
interfaces running different protocol configurations. Some test scenarios of the IOL test
suite require the router to send a minimum number of LSAs. There are two approaches for
realizing such scenarios. A tester could generate "virtual" LSAs to fill its link state
database. In order to leave the OSPF session unchanged, this would require to simulate the
complete building of an adjacency with a router connected to a virtual network. That means,
that the tester has to implement parts of the OSPF protocol itself. The second approach to
this problem uses the fact, that large networks can be generated easily in SSFNet. So, the
router which has its link state database to be filled is simply connected to a network, in
which a sufficient number of LSAs are generated (see Figure 3). In the test dictionary,
there are networks with 8, 57 and 400 routers, which can be used for this purpose.
Figure 3 - Filling a router's link state database
Last update: 2002-02-05, Dirk Jacob (dirk@d-jacob.net)