gs/doc/Testing.htm

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Using and writing Ghostscript testing scripts</title>
<!-- $Id: Testing.htm,v 1.37 2005/10/20 19:46:23 ray Exp $ -->
<link rel="stylesheet" type="text/css" href="gs.css" title="Ghostscript Style">
</head>

<body>
<!-- [1.0 begin visible header] ============================================ -->

<!-- [1.1 begin headline] ================================================== -->

<h1>Using and writing Ghostscript testing scripts</h1>

<!-- [1.1 end headline] ==================================================== -->

<!-- [1.2 begin table of contents] ========================================= -->

<!-- [1.2 end table of contents] =========================================== -->

<h2>Table of contents</h2>

<blockquote><ul>
<li><a href="#General_overview">General overview</a>
<li><a href="#Running_tests">Running tests</a>
<ul>
<li><a href="#Individual_tests">Individual tests</a>
<li><a href="#Regression_testing">Regression testing</a>
</ul>
<li><a href="#Writing_new_tests">Writing new tests</a>
</ul></blockquote>

<!-- [1.3 begin hint] ====================================================== -->

<p>
This document describes how to use the scripts located in the <a
href="../toolbin/tests">toolbin/tests</a> directory, and conventions for
writing new testing scripts.

<p>For other information, see the <a href="Readme.htm">Ghostscript
overview</a> and the documentation related to <a
href="Maintain.htm">maintaining Ghostscript</a>.

<!-- [1.3 end hint] ======================================================== -->

<hr>

<!-- [1.0 end visible header] ============================================== -->

<!-- [2.0 begin contents] ================================================== -->

<h2><a name="General_overview"></a>General overview</h2>

<p>
The test scripts discussed here are written in Python, a language whose
implementation is freely available from <a class="offsite"
href="http://www.python.org"><tt>http://www.python.org</tt></a>.  The
scripts require Python version 2.1 or later.

<h2><a name="Running_tests"></a>Running tests</h2>

<p>
On Unix and Linux systems, test scripts written in Python can be executed
directly simply by typing their name into the shell, e.g.,

<blockquote><pre>
toolbin/tests/check_source.py
</pre></blockquote>

<p>
On other systems, it may be necessary to invoke Python explicitly, e.g.,

<blockquote><pre>
python toolbin/tests/check_source.py
</pre></blockquote>

<p>
The test scripts will print information about any failures that occur.

<h3><a name="Individual_tests"></a>Individual tests</h3>

<p>
The individual test scripts are named
<b><tt>toolbin/tests/check_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if they
do not run Ghostscript) or
<b><tt>toolbin/tests/gscheck_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if
they do run Ghostscript).  Any script with such a name can be run
individually, and is also normally run as part of regression testing
(described in the next section).

<p>
We don't list the individual test scripts here, because any such
documentation would inevitably be out of date most of the time.  Each of
these scripts contains documentation about what it tests: we suggest you
read the documentation in the scripts.

<h3><a name="Regression_testing"></a>Regression testing</h3>

<p>
We run a nightly regression test to discover any obvious problems caused by
code checked in the previous day.  Here is a list of the scripts and
supporting files that make up the regression test.

<h4>Top-level scripts</h4>

<dl>
<dl>
<dt><a href="../toolbin/tests/dump_testdb"><tt>dump_testdb</tt></a>
<dd>A debugging script that will print the contents of the database defined
by gsconf.testdatadb.
</dl>

<dl>
<dt><a href="../toolbin/tests/make_testdb"><tt>make_testdb</tt></a>
<dd>This script creates an initial test database.  It uses gsconf.baselinegs
to create raster data from the test files and computes their MD5 sums and
stores them in the gsconf.testdatadb database.
</dl>

<dl>
<dt><a href="../toolbin/tests/make_two_versions"><tt>make_two_versions</tt></a>
<dd>A helper script to make two versions of a particular file for visual
diffing or manual analysis.  When a test fails nightly regression, this is
generally the first investigative step.
</dl>

<dl>
<dt><a href="../toolbin/tests/make_two_pdfversions"><tt>make_two_pdfversions</tt></a>
<dd>Same as above, except for pdfwrite regressions.
</dl>

<dl>
<dt><a href="../toolbin/tests/testdiff"><tt>testdiff</tt></a>
<dd>this script provides the difference between two sets of regression results.
if end date is omitted, the current date will be used.
</dl>

<dl>
<dt><a href="../toolbin/tests/revert_baseline"><tt>revert_baseline</tt></a>
<dd>In cases where a baseline has been accidentally updated, this script
will revert the database entry to the MD5 sum computed with
gsconf.baselinegs.
</dl>

<dl>
<dt><a href="../toolbin/tests/revert_pdfbaseline"><tt>revert_pdfbaseline</tt></a>
<dd>Same as above, except for pdfwrite baselines.
</dl>

<dl>
<dt><a href="../toolbin/tests/run_nightly"><tt>run_nightly</tt></a>
<dd>This is the control script (usually invoked by cron) that controls the
nightly test run.  It's responsible for checking the latest code out of CVS,
building a new Ghostscript to compare with and launching the test suite via
run_regression.
</dl>

<dl>
<dt><a href="../toolbin/tests/run_regression"><tt>run_regression</tt></a>
<dd>This script runs the full gamut of regression tests using files from
gsconf.comparedir.  It differentiates files by extension and controls what
tests get run and with what options.
</dl>

<dl>
<dt><a href="../toolbin/tests/update_baseline"><tt>update_baseline</tt></a>
<dd>This script is invoked to update the MD5 sum in the test database when a
nightly regression is really a progression.  Generally after noticing that
the output from make_two_versions is acceptable or better, this script is
run to log the changes to the database.
</dl>

<dl>
<dt><a href="../toolbin/tests/update_pdfbaseline"><tt>update_pdfbaseline</tt></a>
<dd>Same as above, except for pdfwrite baselines.
</dl>

<dl>
<dt><a href="../toolbin/tests/update_specific"><tt>update_specific</tt></a> <tt><em>cvs-date-spec</em></tt>
<dd>Update a series of baselines to a specific datestamp; accepts a series of "flag filename"
lines on stdin, where flag is either N or P to specify whether the normal or pdfwrite baselines
should be updated.
<dl>

<dt><a href="../toolbin/tests/get_baselines"><tt>get_baselines</tt></a>
<dd>This script returns the change log for the test database starting
at a given date.
</dl>
</dl>

<h4>Support files</h4>

<dl>

<dl>
<dt><a href="../toolbin/tests/fuzzy.c"><tt>fuzzy.c</tt></a>
<dd>A fuzzy comparison tool appropriate for tests where exact binary matches
aren't appropriate.
</dl>

<dl>
<dt><a href="../toolbin/tests/cmpi.py"><tt>cmpi.py</tt></a>
<dd>An image comparison tool for exploring the differences between two
raster files. It can switch between the two, display only the differences
and localize, highlight and jump between areas where differences occur.
This can be a useful way of examining regression difference to decide
if they are progressions or regressions. Requires python imaging and
python Tkinter.
</dl>

<dl>
<dt><a href="../toolbin/tests/gsconf.py"><tt>gsconf.py</tt></a>
<dd>This is the configuration framework for the scripts above.  It reads
configuration files and makes available those configuration options to
the rest of the testing framework.
</dl>

<dl>
<dt><a href="../toolbin/tests/testing.cfg.example"><tt>testing.cfg.example</tt></a>
<dd>This is an example configuration file for the scripts above.  It controls
where files are found, where Ghostscript executables are and the location of
the test database.  Most test configuration will be in this file.  It must
be copied to testing.cfg in order for the tests to find it.
</dl>

<dl>
<dt><a href="../toolbin/tests/testing.cfg"><tt>testing.cfg</tt></a>
<dd>This is the configuration file for the scripts above.  It controls
where files are found, where Ghostscript executables are and the location of
the test database.  Most test configuration will be in this file. The file
testing.cfg.example above can be used as a template.
</dl>

<dl>
<dt><a href="../toolbin/tests/gstestgs.py"><tt>gstestgs.py</tt></a>
<dd>This provides classes for running tests that actually execute
Ghostcript.
</dl>

<dl>
<dt><a href="../toolbin/tests/gsparamsets.py"><tt>gsparamsets.py</tt></a>
<dd>What parameters tests get run with by default are stored in this file.
Between this configuration information and the information in gsconf.py all
configurable testing parameters should be covered.
</dl>

<dl>
<dt><a href="../toolbin/tests/gsutil.py"><tt>gsutil.py</tt></a>
<dd>Various utility routines used by the regression test scripts.
</dl>

<dl>
<dt><a href="../toolbin/tests/gssum.py"><tt>gssum.py</tt></a>
<dd>Helper functions that compute, compare and store MD5 sums.
</dl>

<dl>
<dt><a href="../toolbin/tests/rasterdb.py"><tt>rasterdb.py</tt></a>
<dd>Hepler functions for accessing the raster database.
</dl>

</dl>

<h2><a name="Writing_new_tests"></a>Writing new tests</h2>

<p>
Some of Ghostscript's test scripts follow a set of conventions that allow
them to be run either stand-alone or as part of a suite; in particular, they
can be run as part of the nightly regression test suite.  In this section,
we provide pointers to documentation on how to write new tests that follow
these conventions, since that will make them the most useful.

<p>
The test scripts are based on Python's <b><tt>unittest</tt></b> module.  We
suggest that if you are not familiar with this module, you read the
documentation, which is available at <a class="offsite"
href="http://www.python.org/doc/current/lib/module-unittest.html">http://www.python.org/doc/current/lib/module-unittest.html</a>.

<p>
Ghostscript specializes the <b><tt>unittest</tt></b> module by defining
subclasses, which all individual tests use in place of those in
<b><tt>unittest</tt></b>.  These subclasses are defined in <a
href="../toolbin/tests/gstestutils.py">toolbin/tests/gstestutils.py</a>.

<p>
Since code documentation separate from the code itself is always out of
date, we have decided to maintain the primary documentation for writing new
tests in <a href="../toolbin/tests/gstestutils.py">gstestutils.py</a> rather
than here in a separate document.  Please read that file for more
information.

<!-- [2.0 end contents] ==================================================== -->

<!-- [3.0 begin visible trailer] =========================================== -->
<hr>

<p>
<small>Copyright &copy; 2002 artofcode LLC.  All rights reserved.</small>

<p>
<small>This file is part of AFPL Ghostscript.  See the <a
href="Public.htm">Aladdin Free Public License</a> (the "License") for full
details of the terms of using, copying, modifying, and redistributing AFPL
Ghostscript.</small>

<p>
<small>Ghostscript version 8.53, 20 October 2005

<!-- [3.0 end visible trailer] ============================================= -->

</body>
</html>