|
|
SIGDOC Newsletter
June 2005
:: Volume 6, Number 2
Features
An Introduction to Documentation Testing
By Rob Pierce
robertp@us.ibm.com
When developing software or developing technical documentation, it is
imperative to verify the accuracy and completeness of what you are planning
to deliver. Testing is a fundamental best practice in software development.
And, while testing your software is imperative, the best of quality assurance
intentions will fail if the following variables are not accounted for
in advance of any actual testing effort:
- Planning
- Designing
- Scheduling
- Staffing
A lack of time, resources, or formal test cases planned to be followed
can lead to application instability, late system delivery, or
a higher number of defects delivered to end users and customers. For the information
developer, a lack of formal testing may result in similar errors
and
issues with a Help system or context sensitive Help within an
application.
Typically, the last phase in the software development cycle is
testing, and this phase in the development cycle is the most
commonly shortened
or neglected phase.
back to top Testing starts with a test plan. In addition to establishing
the resource and scheduling variables for the testing effort,
a test
plan should identify
the scope and level of granularity of what will be tested. The
first step is to identify the types of tests to plan for and
follow. For example,
the primary types of tests to plan for include:
- Unit tests
A unit test is the smallest piece of software that can be compiled
and tested. For a programming language, a unit might be one method
or one class in an application programming interface (API). Each unit is
typically the smallest level of component in a complete system. Each unit
test
can be run in isolation from the context of the entire system.
For an information development project or deliverable, one topic
file might comprise a unit test. The content would be validated
for completeness
and accuracy. Another example is to test a fix to a given topic
file to resolve a change request (also known as a documentation
defect) that
needs to be validated before the change request is closed.
- Functional tests
A functional test applies to a specific function or use case for an
application. Functional tests check for correct behavior of a collection
of
components that comprise one function or use case. For example, testing
the functionality
to generate a report or create a query would be examples of
functional tests.
For an information development project or deliverable, a documented
use case to be followed would be a functional test. The use
case, or procedure,
would be followed for completeness and accuracy. For example,
following a procedure to generate a report or create a query
would be examples
of functional tests.
A component might be a chapter of a book, or a section of a
Help system. Or, it might be one complete installation procedure
for
a product. A
component test might be to review a new section or to follow
a procedure to verify completeness and accuracy. Other examples
of
testing documentation
components might be to ensure that new content is correctly
integrated into an existing deliverable or to validate that
information
is available through a keyword search or index entry.
- System
tests
Testing of complete scenarios that span multiple components of an application.
System tests can only be run on a complete system.
For an information development project or deliverable, a complete
Help system being tested for correct links, search capabilities,
and indexing,
is an example of a system test for an information development
project or deliverable. Another example would be to check a
use case plus
all links to related information of other topic types. For
example, test
a use case and validate that all of the references or links
to additional information such as conceptual and reference
topics
are all correct.
- Performance tests
Performance tests are a form of system tests that determine if performance
requirements are met. Performance has a significant effect
on user perceptions of a product. Slow performance degrades the usability of
a
product.
Performance tests for an information development deliverable would
test the time for a Help system to launch or the time for a
search result
to display. For example, if a Help system displays 90 seconds
after it is invoked on a commonly used browser, then a performance issue
would
be identified and need to be corrected.
back to top
As the examples illustrate, testing software and documentation
need not be separated. In fact, testing them together may accelerate
testing efforts,
since good documentation can assist the quality engineers to
understand the context and follow the procedures to testing
the software. For example, to complete a functional test for a complex installation
of a product,
if documentation is available that describes the procedure,
then the tester can try following the documentation to both
validate
that the
use case (installing the product) documentation as well as
the actual function (that the product correctly installs if
you follow
the correct
procedure).
Each test can be viewed as a component of a whole
test plan just as each portion of the software or information can be
viewed
as a component to
the whole deliverable. As such, each test can be viewed as
a test case. A test plan contains test cases. And each test
case
would
be one of the
described test types.
To summarize, testing includes:
- test plans, that outline high-level areas of coverage and
non-coverage for features and components.
- test schedules, that list
timeframes and assignments.
- test cases, that provide details of input
and expected output, including history of execution, failures, and
configurations.
- Defects, submitted by testers for results of tests
that fail, tracked in a defect or change management database.
- test
status or metrics, that provide details of how much has been accomplished,
and how much is remaining as well as a list of any issues
of concern.
back to top
Quality engineering groups follow test plans that include a set
of agreed-upon use cases to test functionality. In addition to quality engineering tests are usability
tests. Usability testing may be viewed as complementary
or distinct
from functional,
system, and performance testing, depending on the product
or organization.
A use case might be contained in one component, Help topic,
or procedure, or it might span multiple topics, procedures
or components,
especially
if the use case involves an integration between different
products. Since use cases presumably follow more closely
how a product
will be used by
customers, use case testing is likely the most critical
and valuable to follow to ensure that documentation is
accurate,
complete,
and can be optimally retrieved. However, because use case
testing requires
the
most time and lack of automation, it requires the most
resources in terms of staffing.
Information developers and documentation deliverables can
help software development use-case testing if use cases
are documented
and testers
can follow the documentation to validate the use case.
In this manner, the documentation can also be validated
in the
process
of testing use
cases. For example, if a new Java method is documented
and the new information is integrated into a given component,
or documented
class, along with
appropriate index entries using terms common to customers
or other users, then complete testing will include:
- validating that the method is correctly documented
- checking to verify that the new content is placed in the correct context
and can be easily located
- ensuring that a provided code example not only compiles but works
to provide a solution for a specified use case, if applicable.
If a
software development functional test is planned for a given use case
that requires the use of the
newly documented
method,
then a tester
may indirectly complete the information development
testing in the process of validating the actual use
case.
Another example is having a test engineer install
a product and being required to follow the documentation
explicitly.
This can
help identify
issues or areas for improvement. In many organizations
this type of testing is a requirement before documents
are approved
for
distribution.
Perhaps designing test plans that capture both aspects
of a product – its
functionality and its documentation – in one
comprehensive set of test cases, ensures the most
usable of all deliverables.
back to top |
