Documentation and Testing

Software Engineering texts are filled with methodologies and advice about how best to design a software system. Although methods may differ, a couple of things that all software engineering texts agree on are documentation and testing.

Steam XML provides for documentation and testing to be integrated with the IDE or generated to be viewed separately. The test results and documentation can be displayed within the development environment itself, or they can be viewed in HTML on separate web pages.

Water provides for documentation and testing through the <doc /> and <test /> classes. The example below is meant to show how closely documentation and testing can be tied to individual methods in a particular class.

Declaring a Method

Figure 1 shows a method called <return_string_keys /> that loops through the string_keyed fields of an object and returns the key names as strings in a vector.

<!-- ----------------------------------------------------------------
  M E T H O D _ D E C L A R A T I O N S
----------------------------------------------------------------- -->
<!-- ----------------------------------------------------------------
  R E T U R N _ S T R I N G _ K E Y S
----------------------------------------------------------------- -->
<defmethod return_string_keys  >

   <if> _environment.<has "_subject" />.<not />
        <error "Method <return_string_keys /> expects a _subject."  /> 
   </if>

    _subject.<for_each include=string_key combiner=insert >
       key
    </for_each>

</defmethod>

<!-- ----------------------------------------------------------------
       E N D _ O F _ M E T H O D _ D E C L A R A T I O N S
----------------------------------------------------------------- -->

Figure 1

Internal Documentation

There might be several Water methods used in the short example above about which a programmer would include internal documentation. Water uses standard HTML comment tags <!-- --> to include common internal documentation, as seen at the top and bottom of Figure 1.

Figure 2 shows how the first figure might be documented using internal documentation tags.

<!-- ----------------------------------------------------------------
  R E T U R N _ S T R I N G _ K E Y S
---------------------------------------------------------------------
  This method returns the string_keyed fields of any object passed 
  to it.
----------------------------------------------------------------- -->
<defmethod return_string_keys  >

<!-- If method call has no subject, throw an error.    -->

   <if> _environment.<has "_subject" />.<not />
        <error "Method <return_string_keys /> expects a _subject."  /> 
   </if>

<!-- Loop through the string_keys, returning a vector. -->
<!-- If there are no string_keys, the vector will be   -->
<!-- empty.                                            -->

    _subject.<for_each include=string_key combiner=insert >
       key
    </for_each>

</defmethod>

Figure 2

To the discerning eye, the internal documentation in Figure 2 contains several comments that give clues to what functionality should be tested and documented. Good programmers will take care to include examples of the expected functionality, and test cases that cover the expected behavior of the method in all foreseeable circumstances.

Test Cases

In order to ensure that the <return_string_keys /> method will perform according to specification in any particular implementation of the Water language, test cases can be written and run against the method.

Steam XML allows for these test cases to be included in the same file as the actual method declaration, or to be stored in a separate file. This choice is very convenient. I prefer to keep test cases stored in a separate file, and I give the test case file the same name as the method file but with a "_f_test.h2o" extention.

Figure 3 shows a set of test cases for the <return_string_keys /> method.

<!-- ----------------------------------------------------------------
  T E S T _ C A S E S
----------------------------------------------------------------- -->
<!-- ----------------------------------------------------------------
  R E T U R N _ S T R I N G _ K E Y S
----------------------------------------------------------------- -->
thing.return_string_keys.<test name="return string_keys only"
   result=<vector "last_name" "first_name" />
   known_bug=false >
   <thing first_name="Merrick" last_name="Stemen" "a_vector_item" />.
   <return_string_keys />
</test>

thing.return_string_keys.<test name="return empty vector for _subject integer"
   result=<vector />
   known_bug=false >
   5.<return_string_keys />
</test>

thing.return_string_keys.<test name="return empty vector on no string keys"
   result=<vector />
   known_bug=false >
   <thing "vector_time_1" "vector_item_2" />.
   <return_string_keys />
</test>

thing.return_string_keys.<test name="error on no _subject"
   result=<error "Method <return_string_keys /> expects a _subject."  /> 
   known_bug=false >
   <return_string_keys />
</test>

<!-- ----------------------------------------------------------------
              E N D _ O F _ T E S T _ C A S E S
----------------------------------------------------------------- -->

Figure 3

The known_bug Field

The test cases in Figure 3 all include a name, result, and a known_bug field. The purpose of these fields may become more apparent when I show an example of what the output looks like from running one of these tests.

Figure 4

Figure 4 shows the output if the first test is passed successfully. Figure 5 shows the output if the known_bug field is set to true for the first test case. Because the test succeeds, the output notifies us that the expected bug is now fixed.

   <thing first_name="Merrick" last_name="Stemen" "a_vector_item" />.
   <return_string_keys />

<vector "last_name" "first_name" />

Figure 5

Figure 6a shows Steam XML's output for a test that fails when known_bug is true, and Figure 6b shows the output when known_bug is false. For this example, I have changed the result field to expect a different result.

Figure 6a

   <thing first_name="Merrick" last_name="Stemen" "a_vector_item" />.
   <return_string_keys />

<vector "last_name" "first_name" "a_vector_item" />

<vector "last_name" "first_name"/>

Figure 6b

Generating a Test Report

Once you have developed a test suite you would like to run against your code, Steam XML provides a facility for viewing the results of your test cases under the Test > Show Test Report menu. Figure 7 shows a test report generated by Steam XML including the error introduced in the previous examples.

   <thing first_name="Merrick" last_name="Stemen" "a_vector_item" />.
   <return_string_keys />

<vector "last_name" "first_name" "a_vector_item" />

<vector "last_name" "first_name"/>

Figure 7

For the Record

The error introduced into the example above as a known_bug is in fact a bad test case. It was used here to illustrate what happens when code is bad. However, it also serves as a reminder that:

"A bad test case can cause just as much trouble as bad code."

Perfect code can be brought into doubt by bad test cases, so be careful when writing test cases that they neither flag false positives, nor allow bad code to escape without a warning.

"External" Documentation

While internal documentation is important to allowing another programmer to understand the inner workings of your code, external documentation allows other programmers to understand what to expect when they make use of your code.

The internal documentation is for the programmer who views your code as a clear box, and the external documentation is for the programmer who views your code as a black box.

Figure 8 is an example of external documentation for the <return_string_keys /> method using Water's <doc /> feature. Figure 9 shows the incredible documentation generated by Steam XML from the combination of the method declaration and the <doc /> tag.

<!-- ----------------------------------------------------------------
  D O C U M E N T A T I O N
----------------------------------------------------------------- -->
<!-- ----------------------------------------------------------------
  R E T U R N _ S T R I N G _ K E Y S
----------------------------------------------------------------- -->
thing.return_string_keys.<doc>

  This method returns a vector of the 
  string_keys in the <b>_subject</b> object.

  <Example name="Default example" result=<vector "last_name" "first_name" /> >
     <thing first_name="Merrick" last_name="Stemen" />.
     <return_string_keys />
  </Example>

  The returned vector can be made more HTML-friendly by following a call 
  to this method with a call to <join <br /> />

  <Example name="Formatting a result for HTML" result="last_name<br />first_name" >
     <thing first_name="Merrick" last_name="Stemen" />.
     <return_string_keys />.<join <br /> />
  </Example>

  The string_keys in the vector can be sorted using the <sort /> 
  method.

  <Example name="Sorting the result" result="first_name<br />last_name" >
     <thing first_name="Merrick" last_name="Stemen" />.
     <return_string_keys />.<sort />.<join <br /> />
  </Example>


</doc>

<!-- ----------------------------------------------------------------
            E N D _ O F _ D O C U M E N T A T I O N
----------------------------------------------------------------- -->

Figure 8

Note: In Figure 9, the links related to the <return_string_keys > method will not work because they are out-of-context in this page.

     <thing first_name="Merrick" last_name="Stemen" />.
     <return_string_keys />
  

<vector "last_name" "first_name" />

     <thing first_name="Merrick" last_name="Stemen" />.
     <return_string_keys />.<join <br /> />
  

     <thing first_name="Merrick" last_name="Stemen" />.
     <return_string_keys />.<sort />.<join <br /> />
  

Figure 9

A Final Word About Tests and Docs

Testing and Documentation are necessary parts of any complete software design. When used correctly, they help to make your code more reliable and accessible.

Steam XML makes creating and applying test cases easy and convenient. It also makes including both internal and external documentation easy and convenient.

Now that you have seen it done, get out there and write some great code!