1*593dc095SDavid du Colombier<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> 2*593dc095SDavid du Colombier<html> 3*593dc095SDavid du Colombier<head> 4*593dc095SDavid du Colombier<title>Using and writing Ghostscript testing scripts</title> 5*593dc095SDavid du Colombier<!-- $Id: Testing.htm,v 1.37 2005/10/20 19:46:23 ray Exp $ --> 6*593dc095SDavid du Colombier<link rel="stylesheet" type="text/css" href="gs.css" title="Ghostscript Style"> 7*593dc095SDavid du Colombier</head> 8*593dc095SDavid du Colombier 9*593dc095SDavid du Colombier<body> 10*593dc095SDavid du Colombier<!-- [1.0 begin visible header] ============================================ --> 11*593dc095SDavid du Colombier 12*593dc095SDavid du Colombier<!-- [1.1 begin headline] ================================================== --> 13*593dc095SDavid du Colombier 14*593dc095SDavid du Colombier<h1>Using and writing Ghostscript testing scripts</h1> 15*593dc095SDavid du Colombier 16*593dc095SDavid du Colombier<!-- [1.1 end headline] ==================================================== --> 17*593dc095SDavid du Colombier 18*593dc095SDavid du Colombier<!-- [1.2 begin table of contents] ========================================= --> 19*593dc095SDavid du Colombier 20*593dc095SDavid du Colombier<!-- [1.2 end table of contents] =========================================== --> 21*593dc095SDavid du Colombier 22*593dc095SDavid du Colombier<h2>Table of contents</h2> 23*593dc095SDavid du Colombier 24*593dc095SDavid du Colombier<blockquote><ul> 25*593dc095SDavid du Colombier<li><a href="#General_overview">General overview</a> 26*593dc095SDavid du Colombier<li><a href="#Running_tests">Running tests</a> 27*593dc095SDavid du Colombier<ul> 28*593dc095SDavid du Colombier<li><a href="#Individual_tests">Individual tests</a> 29*593dc095SDavid du Colombier<li><a href="#Regression_testing">Regression testing</a> 30*593dc095SDavid du Colombier</ul> 31*593dc095SDavid du Colombier<li><a href="#Writing_new_tests">Writing new tests</a> 32*593dc095SDavid du Colombier</ul></blockquote> 33*593dc095SDavid du Colombier 34*593dc095SDavid du Colombier<!-- [1.3 begin hint] ====================================================== --> 35*593dc095SDavid du Colombier 36*593dc095SDavid du Colombier<p> 37*593dc095SDavid du ColombierThis document describes how to use the scripts located in the <a 38*593dc095SDavid du Colombierhref="../toolbin/tests">toolbin/tests</a> directory, and conventions for 39*593dc095SDavid du Colombierwriting new testing scripts. 40*593dc095SDavid du Colombier 41*593dc095SDavid du Colombier<p>For other information, see the <a href="Readme.htm">Ghostscript 42*593dc095SDavid du Colombieroverview</a> and the documentation related to <a 43*593dc095SDavid du Colombierhref="Maintain.htm">maintaining Ghostscript</a>. 44*593dc095SDavid du Colombier 45*593dc095SDavid du Colombier<!-- [1.3 end hint] ======================================================== --> 46*593dc095SDavid du Colombier 47*593dc095SDavid du Colombier<hr> 48*593dc095SDavid du Colombier 49*593dc095SDavid du Colombier<!-- [1.0 end visible header] ============================================== --> 50*593dc095SDavid du Colombier 51*593dc095SDavid du Colombier<!-- [2.0 begin contents] ================================================== --> 52*593dc095SDavid du Colombier 53*593dc095SDavid du Colombier<h2><a name="General_overview"></a>General overview</h2> 54*593dc095SDavid du Colombier 55*593dc095SDavid du Colombier<p> 56*593dc095SDavid du ColombierThe test scripts discussed here are written in Python, a language whose 57*593dc095SDavid du Colombierimplementation is freely available from <a class="offsite" 58*593dc095SDavid du Colombierhref="http://www.python.org"><tt>http://www.python.org</tt></a>. The 59*593dc095SDavid du Colombierscripts require Python version 2.1 or later. 60*593dc095SDavid du Colombier 61*593dc095SDavid du Colombier<h2><a name="Running_tests"></a>Running tests</h2> 62*593dc095SDavid du Colombier 63*593dc095SDavid du Colombier<p> 64*593dc095SDavid du ColombierOn Unix and Linux systems, test scripts written in Python can be executed 65*593dc095SDavid du Colombierdirectly simply by typing their name into the shell, e.g., 66*593dc095SDavid du Colombier 67*593dc095SDavid du Colombier<blockquote><pre> 68*593dc095SDavid du Colombiertoolbin/tests/check_source.py 69*593dc095SDavid du Colombier</pre></blockquote> 70*593dc095SDavid du Colombier 71*593dc095SDavid du Colombier<p> 72*593dc095SDavid du ColombierOn other systems, it may be necessary to invoke Python explicitly, e.g., 73*593dc095SDavid du Colombier 74*593dc095SDavid du Colombier<blockquote><pre> 75*593dc095SDavid du Colombierpython toolbin/tests/check_source.py 76*593dc095SDavid du Colombier</pre></blockquote> 77*593dc095SDavid du Colombier 78*593dc095SDavid du Colombier<p> 79*593dc095SDavid du ColombierThe test scripts will print information about any failures that occur. 80*593dc095SDavid du Colombier 81*593dc095SDavid du Colombier<h3><a name="Individual_tests"></a>Individual tests</h3> 82*593dc095SDavid du Colombier 83*593dc095SDavid du Colombier<p> 84*593dc095SDavid du ColombierThe individual test scripts are named 85*593dc095SDavid du Colombier<b><tt>toolbin/tests/check_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if they 86*593dc095SDavid du Colombierdo not run Ghostscript) or 87*593dc095SDavid du Colombier<b><tt>toolbin/tests/gscheck_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if 88*593dc095SDavid du Colombierthey do run Ghostscript). Any script with such a name can be run 89*593dc095SDavid du Colombierindividually, and is also normally run as part of regression testing 90*593dc095SDavid du Colombier(described in the next section). 91*593dc095SDavid du Colombier 92*593dc095SDavid du Colombier<p> 93*593dc095SDavid du ColombierWe don't list the individual test scripts here, because any such 94*593dc095SDavid du Colombierdocumentation would inevitably be out of date most of the time. Each of 95*593dc095SDavid du Colombierthese scripts contains documentation about what it tests: we suggest you 96*593dc095SDavid du Colombierread the documentation in the scripts. 97*593dc095SDavid du Colombier 98*593dc095SDavid du Colombier<h3><a name="Regression_testing"></a>Regression testing</h3> 99*593dc095SDavid du Colombier 100*593dc095SDavid du Colombier<p> 101*593dc095SDavid du ColombierWe run a nightly regression test to discover any obvious problems caused by 102*593dc095SDavid du Colombiercode checked in the previous day. Here is a list of the scripts and 103*593dc095SDavid du Colombiersupporting files that make up the regression test. 104*593dc095SDavid du Colombier 105*593dc095SDavid du Colombier<h4>Top-level scripts</h4> 106*593dc095SDavid du Colombier 107*593dc095SDavid du Colombier<dl> 108*593dc095SDavid du Colombier<dl> 109*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/dump_testdb"><tt>dump_testdb</tt></a> 110*593dc095SDavid du Colombier<dd>A debugging script that will print the contents of the database defined 111*593dc095SDavid du Colombierby gsconf.testdatadb. 112*593dc095SDavid du Colombier</dl> 113*593dc095SDavid du Colombier 114*593dc095SDavid du Colombier<dl> 115*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/make_testdb"><tt>make_testdb</tt></a> 116*593dc095SDavid du Colombier<dd>This script creates an initial test database. It uses gsconf.baselinegs 117*593dc095SDavid du Colombierto create raster data from the test files and computes their MD5 sums and 118*593dc095SDavid du Colombierstores them in the gsconf.testdatadb database. 119*593dc095SDavid du Colombier</dl> 120*593dc095SDavid du Colombier 121*593dc095SDavid du Colombier<dl> 122*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/make_two_versions"><tt>make_two_versions</tt></a> 123*593dc095SDavid du Colombier<dd>A helper script to make two versions of a particular file for visual 124*593dc095SDavid du Colombierdiffing or manual analysis. When a test fails nightly regression, this is 125*593dc095SDavid du Colombiergenerally the first investigative step. 126*593dc095SDavid du Colombier</dl> 127*593dc095SDavid du Colombier 128*593dc095SDavid du Colombier<dl> 129*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/make_two_pdfversions"><tt>make_two_pdfversions</tt></a> 130*593dc095SDavid du Colombier<dd>Same as above, except for pdfwrite regressions. 131*593dc095SDavid du Colombier</dl> 132*593dc095SDavid du Colombier 133*593dc095SDavid du Colombier<dl> 134*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/testdiff"><tt>testdiff</tt></a> 135*593dc095SDavid du Colombier<dd>this script provides the difference between two sets of regression results. 136*593dc095SDavid du Colombierif end date is omitted, the current date will be used. 137*593dc095SDavid du Colombier</dl> 138*593dc095SDavid du Colombier 139*593dc095SDavid du Colombier<dl> 140*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/revert_baseline"><tt>revert_baseline</tt></a> 141*593dc095SDavid du Colombier<dd>In cases where a baseline has been accidentally updated, this script 142*593dc095SDavid du Colombierwill revert the database entry to the MD5 sum computed with 143*593dc095SDavid du Colombiergsconf.baselinegs. 144*593dc095SDavid du Colombier</dl> 145*593dc095SDavid du Colombier 146*593dc095SDavid du Colombier<dl> 147*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/revert_pdfbaseline"><tt>revert_pdfbaseline</tt></a> 148*593dc095SDavid du Colombier<dd>Same as above, except for pdfwrite baselines. 149*593dc095SDavid du Colombier</dl> 150*593dc095SDavid du Colombier 151*593dc095SDavid du Colombier<dl> 152*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/run_nightly"><tt>run_nightly</tt></a> 153*593dc095SDavid du Colombier<dd>This is the control script (usually invoked by cron) that controls the 154*593dc095SDavid du Colombiernightly test run. It's responsible for checking the latest code out of CVS, 155*593dc095SDavid du Colombierbuilding a new Ghostscript to compare with and launching the test suite via 156*593dc095SDavid du Colombierrun_regression. 157*593dc095SDavid du Colombier</dl> 158*593dc095SDavid du Colombier 159*593dc095SDavid du Colombier<dl> 160*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/run_regression"><tt>run_regression</tt></a> 161*593dc095SDavid du Colombier<dd>This script runs the full gamut of regression tests using files from 162*593dc095SDavid du Colombiergsconf.comparedir. It differentiates files by extension and controls what 163*593dc095SDavid du Colombiertests get run and with what options. 164*593dc095SDavid du Colombier</dl> 165*593dc095SDavid du Colombier 166*593dc095SDavid du Colombier<dl> 167*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/update_baseline"><tt>update_baseline</tt></a> 168*593dc095SDavid du Colombier<dd>This script is invoked to update the MD5 sum in the test database when a 169*593dc095SDavid du Colombiernightly regression is really a progression. Generally after noticing that 170*593dc095SDavid du Colombierthe output from make_two_versions is acceptable or better, this script is 171*593dc095SDavid du Colombierrun to log the changes to the database. 172*593dc095SDavid du Colombier</dl> 173*593dc095SDavid du Colombier 174*593dc095SDavid du Colombier<dl> 175*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/update_pdfbaseline"><tt>update_pdfbaseline</tt></a> 176*593dc095SDavid du Colombier<dd>Same as above, except for pdfwrite baselines. 177*593dc095SDavid du Colombier</dl> 178*593dc095SDavid du Colombier 179*593dc095SDavid du Colombier<dl> 180*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/update_specific"><tt>update_specific</tt></a> <tt><em>cvs-date-spec</em></tt> 181*593dc095SDavid du Colombier<dd>Update a series of baselines to a specific datestamp; accepts a series of "flag filename" 182*593dc095SDavid du Colombierlines on stdin, where flag is either N or P to specify whether the normal or pdfwrite baselines 183*593dc095SDavid du Colombiershould be updated. 184*593dc095SDavid du Colombier<dl> 185*593dc095SDavid du Colombier 186*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/get_baselines"><tt>get_baselines</tt></a> 187*593dc095SDavid du Colombier<dd>This script returns the change log for the test database starting 188*593dc095SDavid du Colombierat a given date. 189*593dc095SDavid du Colombier</dl> 190*593dc095SDavid du Colombier</dl> 191*593dc095SDavid du Colombier 192*593dc095SDavid du Colombier<h4>Support files</h4> 193*593dc095SDavid du Colombier 194*593dc095SDavid du Colombier<dl> 195*593dc095SDavid du Colombier 196*593dc095SDavid du Colombier<dl> 197*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/fuzzy.c"><tt>fuzzy.c</tt></a> 198*593dc095SDavid du Colombier<dd>A fuzzy comparison tool appropriate for tests where exact binary matches 199*593dc095SDavid du Colombieraren't appropriate. 200*593dc095SDavid du Colombier</dl> 201*593dc095SDavid du Colombier 202*593dc095SDavid du Colombier<dl> 203*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/cmpi.py"><tt>cmpi.py</tt></a> 204*593dc095SDavid du Colombier<dd>An image comparison tool for exploring the differences between two 205*593dc095SDavid du Colombierraster files. It can switch between the two, display only the differences 206*593dc095SDavid du Colombierand localize, highlight and jump between areas where differences occur. 207*593dc095SDavid du ColombierThis can be a useful way of examining regression difference to decide 208*593dc095SDavid du Colombierif they are progressions or regressions. Requires python imaging and 209*593dc095SDavid du Colombierpython Tkinter. 210*593dc095SDavid du Colombier</dl> 211*593dc095SDavid du Colombier 212*593dc095SDavid du Colombier<dl> 213*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/gsconf.py"><tt>gsconf.py</tt></a> 214*593dc095SDavid du Colombier<dd>This is the configuration framework for the scripts above. It reads 215*593dc095SDavid du Colombierconfiguration files and makes available those configuration options to 216*593dc095SDavid du Colombierthe rest of the testing framework. 217*593dc095SDavid du Colombier</dl> 218*593dc095SDavid du Colombier 219*593dc095SDavid du Colombier<dl> 220*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/testing.cfg.example"><tt>testing.cfg.example</tt></a> 221*593dc095SDavid du Colombier<dd>This is an example configuration file for the scripts above. It controls 222*593dc095SDavid du Colombierwhere files are found, where Ghostscript executables are and the location of 223*593dc095SDavid du Colombierthe test database. Most test configuration will be in this file. It must 224*593dc095SDavid du Colombierbe copied to testing.cfg in order for the tests to find it. 225*593dc095SDavid du Colombier</dl> 226*593dc095SDavid du Colombier 227*593dc095SDavid du Colombier<dl> 228*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/testing.cfg"><tt>testing.cfg</tt></a> 229*593dc095SDavid du Colombier<dd>This is the configuration file for the scripts above. It controls 230*593dc095SDavid du Colombierwhere files are found, where Ghostscript executables are and the location of 231*593dc095SDavid du Colombierthe test database. Most test configuration will be in this file. The file 232*593dc095SDavid du Colombiertesting.cfg.example above can be used as a template. 233*593dc095SDavid du Colombier</dl> 234*593dc095SDavid du Colombier 235*593dc095SDavid du Colombier<dl> 236*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/gstestgs.py"><tt>gstestgs.py</tt></a> 237*593dc095SDavid du Colombier<dd>This provides classes for running tests that actually execute 238*593dc095SDavid du ColombierGhostcript. 239*593dc095SDavid du Colombier</dl> 240*593dc095SDavid du Colombier 241*593dc095SDavid du Colombier<dl> 242*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/gsparamsets.py"><tt>gsparamsets.py</tt></a> 243*593dc095SDavid du Colombier<dd>What parameters tests get run with by default are stored in this file. 244*593dc095SDavid du ColombierBetween this configuration information and the information in gsconf.py all 245*593dc095SDavid du Colombierconfigurable testing parameters should be covered. 246*593dc095SDavid du Colombier</dl> 247*593dc095SDavid du Colombier 248*593dc095SDavid du Colombier<dl> 249*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/gsutil.py"><tt>gsutil.py</tt></a> 250*593dc095SDavid du Colombier<dd>Various utility routines used by the regression test scripts. 251*593dc095SDavid du Colombier</dl> 252*593dc095SDavid du Colombier 253*593dc095SDavid du Colombier<dl> 254*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/gssum.py"><tt>gssum.py</tt></a> 255*593dc095SDavid du Colombier<dd>Helper functions that compute, compare and store MD5 sums. 256*593dc095SDavid du Colombier</dl> 257*593dc095SDavid du Colombier 258*593dc095SDavid du Colombier<dl> 259*593dc095SDavid du Colombier<dt><a href="../toolbin/tests/rasterdb.py"><tt>rasterdb.py</tt></a> 260*593dc095SDavid du Colombier<dd>Hepler functions for accessing the raster database. 261*593dc095SDavid du Colombier</dl> 262*593dc095SDavid du Colombier 263*593dc095SDavid du Colombier</dl> 264*593dc095SDavid du Colombier 265*593dc095SDavid du Colombier<h2><a name="Writing_new_tests"></a>Writing new tests</h2> 266*593dc095SDavid du Colombier 267*593dc095SDavid du Colombier<p> 268*593dc095SDavid du ColombierSome of Ghostscript's test scripts follow a set of conventions that allow 269*593dc095SDavid du Colombierthem to be run either stand-alone or as part of a suite; in particular, they 270*593dc095SDavid du Colombiercan be run as part of the nightly regression test suite. In this section, 271*593dc095SDavid du Colombierwe provide pointers to documentation on how to write new tests that follow 272*593dc095SDavid du Colombierthese conventions, since that will make them the most useful. 273*593dc095SDavid du Colombier 274*593dc095SDavid du Colombier<p> 275*593dc095SDavid du ColombierThe test scripts are based on Python's <b><tt>unittest</tt></b> module. We 276*593dc095SDavid du Colombiersuggest that if you are not familiar with this module, you read the 277*593dc095SDavid du Colombierdocumentation, which is available at <a class="offsite" 278*593dc095SDavid du Colombierhref="http://www.python.org/doc/current/lib/module-unittest.html">http://www.python.org/doc/current/lib/module-unittest.html</a>. 279*593dc095SDavid du Colombier 280*593dc095SDavid du Colombier<p> 281*593dc095SDavid du ColombierGhostscript specializes the <b><tt>unittest</tt></b> module by defining 282*593dc095SDavid du Colombiersubclasses, which all individual tests use in place of those in 283*593dc095SDavid du Colombier<b><tt>unittest</tt></b>. These subclasses are defined in <a 284*593dc095SDavid du Colombierhref="../toolbin/tests/gstestutils.py">toolbin/tests/gstestutils.py</a>. 285*593dc095SDavid du Colombier 286*593dc095SDavid du Colombier<p> 287*593dc095SDavid du ColombierSince code documentation separate from the code itself is always out of 288*593dc095SDavid du Colombierdate, we have decided to maintain the primary documentation for writing new 289*593dc095SDavid du Colombiertests in <a href="../toolbin/tests/gstestutils.py">gstestutils.py</a> rather 290*593dc095SDavid du Colombierthan here in a separate document. Please read that file for more 291*593dc095SDavid du Colombierinformation. 292*593dc095SDavid du Colombier 293*593dc095SDavid du Colombier<!-- [2.0 end contents] ==================================================== --> 294*593dc095SDavid du Colombier 295*593dc095SDavid du Colombier<!-- [3.0 begin visible trailer] =========================================== --> 296*593dc095SDavid du Colombier<hr> 297*593dc095SDavid du Colombier 298*593dc095SDavid du Colombier<p> 299*593dc095SDavid du Colombier<small>Copyright © 2002 artofcode LLC. All rights reserved.</small> 300*593dc095SDavid du Colombier 301*593dc095SDavid du Colombier<p> 302*593dc095SDavid du Colombier<small>This file is part of AFPL Ghostscript. See the <a 303*593dc095SDavid du Colombierhref="Public.htm">Aladdin Free Public License</a> (the "License") for full 304*593dc095SDavid du Colombierdetails of the terms of using, copying, modifying, and redistributing AFPL 305*593dc095SDavid du ColombierGhostscript.</small> 306*593dc095SDavid du Colombier 307*593dc095SDavid du Colombier<p> 308*593dc095SDavid du Colombier<small>Ghostscript version 8.53, 20 October 2005 309*593dc095SDavid du Colombier 310*593dc095SDavid du Colombier<!-- [3.0 end visible trailer] ============================================= --> 311*593dc095SDavid du Colombier 312*593dc095SDavid du Colombier</body> 313*593dc095SDavid du Colombier</html> 314