Test Results
The execution of the test suite generates one of three possible verdicts for
each test case. PASSED is assigned, when the OSPF implementation behaves
exactly as required by the specification. FAIL is assigned, whenever an
offence against the specification is found. INCONCLUSIVE is used for test
cases, where no verdict can be assigned. This is the case for all tests cases
which test parts of the specification which are not yet implemented. The test cases,
which only test features, that are not
implemented yet are not considered in the following part of this section, since the
result is clear. Nevertheless, these tests have to be performed and evaluated once
the according features are implemented in OSPF. Table
3 shows an overview of test results for the current OSPF
implementation version 0.1.5.
Some of the test cases are INCONCLUSIVE at
the moment. These test cases indicate, that they test both, features that are implemented,
but also features that are not implemented yet. The test steps for the features that were
implemented must have got a PASS verdict, since the complete result would have been
a FAIL otherwise. The PASS verdicts indicate, that the according features
have been implemented according to the RFC. The most interesting results are the
test cases, which have FAIL verdicts. These are now discussed in detail.
Test 1.1 -- hello_addressing_and_timing
This test case verifies, if hello packets are sent periodically and addressed to
the multicast address AllSPFRouters. As discussed before, multicast is
not yet available. OSPF
packets are always unicast in the current OSPF version, which causes this test
to fail.
Test 6.2 -- equal_cost_routes
This additional test case is used to verify, that the set of next-hops is
calculated correctly, when there are multiple least-cost routes to a
destination. For the evaluation of the routing tables, the correct routing table
is calculated manually. The test shows, that one route has an additional next
hop, although the destination can only be reached at a higher cost, when this
next-hop is used.
Test 6.3 -- routes_use
This test case verifies, that the routing tables are activated correctly when
they have been calculated. Therefore, a TCP connection is established between
two hosts. According to the log, the client host sends lots of TCP SYN packets,
but receives no response. A look at the simulation log shows, that IP is
dropping packets due to invalid IP addresses. This indicates, that there is a
problem in the OSPF/IP interaction.
Test 6.4 -- table_change
Since this test case uses the same scenario as test number 6.2, it can be
expected, that the same error occurs. Indeed, the additional next hop appears in
this scenario as well. After the routing table is updated, it still exists in
the routing table. When the bug in the next-hop calculation is fixed, this test
will probably pass as well.
Earlier Results
As mentioned before, the test suite has two main purposes. It can be used for
conformance testing and for regression testing as well. Therefore, the tests are
executed, everytimes there is a new version of OSPF. Besides of the results
presented above, the test suite helped uncovering a number of bugs during the
development of OSPF. Because the test suite covers the most parts of the RFC,
performing the tests guarantees, that wide parts of the actual code are exected
at some time. Especially the handling of exceptions or code dealing with special
configurations can be tested in this way. Indeed, the execution of the test
cases helped finding some unhandled exceptions. Mostly these were "standard"
bugs, like unhandled NullPointerExceptions,
ConcurrentModificationExceptions or casts to wrong classes, which could
be fixed easily. This helped to make the code more stable. Surprisingly only a
small number of errors was related to misunderstanding the RFC.
One error for example, caused most of the Flooding tests to fail. When an LSA
has to be flooded, it is set on the retransmission list. Then the a flood timer
is started, which sends all LSAs from this list, when it fires. Then this timer
must be set again, since SSFNet timers are single shot timers. Again, a
"standard" bug caused the whole flooding procedure to fail. When the
retransmission list is empty, the timer is not restarted. But the timer object
exists further on. When the timer had now to be restarted, it was checked for a
null value, instead of testing, whether the timer is cancelled. Because
of this, the flooding timer was never restarted and no LSAs are flooded at all.
After this bug was fixed, most flooding tests were passed.
These experiences show, how useful extensive testing is for the development
process. Using the appropriate tests helps finding errors early. Detailed test
cases help restricting the possible sources of errors to well defined regions of
the source code.
Complete test results
| Test. No. |
Verdict |
Problems / Remarks |
| Hello Protocol Tests |
|
1.1 | FAIL | missing multicast support |
|
1.10 | PASS | |
|
1.11 | PASS | automatic address assignment |
|
1.12 | PASS | |
| Flooding and Adjacency Tests |
|
2.2 | INCONCLUSIVE | virtual links not supported |
|
2.3 | PASS | |
|
2.4 | PASS | |
|
2.5 | PASS | |
|
2.6 | PASS | |
|
2.7 | PASS | |
|
2.8 | PASS | |
|
2.9 | INCONCLUSIVE | AS-external LSAs not yet supported |
|
2.10 | PASS | |
|
2.16 | PASS | |
|
2.17 | PASS | |
|
2.18 | PASS | |
|
2.19 | PASS | |
|
2.20 | PASS | |
|
2.21 | PASS | |
| LSA Tests |
|
3.22 | PASS | |
| Configuration and Formatting Tests |
|
5.1 | PASS | implicit |
|
5.2 | PASS | implicit |
|
5.9 | PASS | implicit |
|
5.10 | PASS | implicit |
|
5.11 | PASS | implicit |
|
5.12 | PASS | implicit |
|
5.14 | PASS | |
| Additional Tests |
|
6.1 | PASS | |
|
6.2 | FAIL | wrong next-hop calculation |
|
6.3 | FAIL | no packet forwarding |
|
6.4 | FAIL | same error as 6.2 |
Table 5
Last update: 2002-02-06, Dirk Jacob (dirk@d-jacob.net)