testr¶

The testr package provides a framework for unit, regression, and integration testing of a set of packages within a production runtime environment. The primary component of testr is lightweight testing script and file structure definition that simple but can support complex testing as needed. The secondary component is a couple of helper functions that make it easier to define and run Python package unit tests. This includes a test_helper module with some helper functions.

Package testing: unit, regression, integration¶

The use case here is a production environment where one has the capability (and indeed requirement) to build and test within a testing environment prior to deploying to production. The starting assumption is that the testr package is installed in the test environment and that all new or updated packages have also been installed to the environment.

Package testing and definition in the testr framework is done entirely by convention rather than configuration. What this means is that there is a certain directory structure and file naming convention that must be followed and which controls how testing is done. There are no configuration files at all.

The following definitions are provided to give a general feel for the different styles of testing supported here. The definitions are by no means rigorous and there is certainly overlap.

Unit testing refers to dedicated tests of individual package functionality that are conceptually independent of the environment and result in an immediate pass/fail for each test.

Regression testing refers to tests that typically produce output files which are then compared against reference outputs. The testr package contains functions to facilitate this process by munging the outputs to remove uninteresting differences (like time stamps or the user who ran a script).

Integration testing refers to ensuring that the packages work within an updated environment. This overlaps with unit and regression testing, but is linked to potential cross-package issues. A key example is updating a core package that may produce unexpected results or issue deprecation warnings downstream that are not desirable.

Basic structure¶

When testr is installed it creates a console script run_testr that can be run from the command line in your environment testing directory. This directory can be anywhere, but as we will see the ideal setup is to have this be a git / GitHub repository.

A full-featured example of the environment testing setup is in the ska_testr GitHub repository. A more minimal example would be:

get_version_id            # Executable script or program
packages/
  py_package/             # py_package is a Python package
    helper_script.py      # Helper script called by test_regress.sh
    test_unit.py          # Run built-in unit tests for my
    test_regress.sh       # Additional regression tests for integration testing
    post_regress.py       # Copy regression output files to the regress/ directory
    post_check_logs.py    # Check logs for "warning" or "error"

  other_package/          # Some package without unit tests
    test_regress_long.sh  # Long-running regression test that isn't run every time
    post_regress_long.py  # Copy regression output files to the regress/ directory

get_version_id¶

The get_version_id file must be an executable script or program that produces a single line of output corresponding to a unique version identifier for the environment. This is used to organize the outputs by version and later to compare the regression outputs. In the Ska runtime environment this script produces a version identifier like 0.18-r609-0d91665.

Packages¶

The packages directory must contain sub-directories corresponding to each package being explicitly tested. For Python packages the directory name should be the same as the importable package name. This allows the simple unit testing idiom (shown later) to infer the package name to import.

Test scripts¶

Test scripts are located within the package sub-directory. The key concept to understand here is that the tests consist entirely of scripts or executable files that have the following properties:

File name begins with test_.
File is either a Python script (test_*.py), Bash script (test_*.sh), or an executable file.
When run the test file returns a successful exit status (0) for PASS and a non-successful exit status (not 0) for FAIL.
Tests are run in alphabetical order.

Those are the only rules, so the testing can run the gamut from simple to arbitrarily complex.

Post-processing scripts¶

After the tests are run then any post-processing scripts in the package directory will be run (again in alphabetical order). These are exactly the same as test files except:

File name begins with post_.
Post-processing scripts will be run after all the test scripts are done.

A key point is that the post-processing scripts can generate failures. For instance a common task is checking the output log files for unexpected warnings that do not generate a failure.

Environment variables¶

Several environment variables are defined prior to spawning jobs that run the test or post-processing scripts.

Name	Definition
TESTR_REGRESS_DIR	Regression output directory for this package
TESTR_INTERPRETER	Interpreter that will be used (python, bash, or None)
TESTR_PACKAGE	Package name
TESTR_PACKAGES_REPO	URL root for cloning package repo
TESTR_FILE	Script file name
TESTR_OUT_DIR	Testing output directory for this package

Filename conventions¶

There are no required conventions beyond the previously mentioned rules, but following some simple conventions will make life easier and more organized. This is especially true because of the way that tests are selected based on the file name and the --include and --exclude arguments of run_testr. The table below shows keywords that are included in the file name, separated by underscores.

Keywords	Meaning
unit	Run built-in unit tests
regress	Regression tests that compare to reference outputs
git	Test that requires cloning a git repo to run
long	Long-running test that does not need to be run every time

Standard filenames and examples of the keywords are shown in the example directory structure above.

Running the tests¶

The run_testr command has the following options:

$ run_testr --help
usage: run_testr [-h] [--test-spec TEST_SPEC] [--root ROOT]
                 [--outputs-dir OUTPUTS_DIR] [--include INCLUDES]
                 [--exclude EXCLUDES] [--collect-only]
                 [--packages-repo PACKAGES_REPO] [--overwrite]

optional arguments:
  -h, --help            show this help message and exit
  --test-spec TEST_SPEC
                        Test include/exclude specification (default=None)
  --root ROOT           Directory containing standard testr configuration
  --outputs-dir OUTPUTS_DIR
                        Directory containing all output package test
                        runs. Absolute, or relative to CWD
  --include INCLUDES    Include tests that match glob pattern
  --exclude EXCLUDES    Exclude tests that match glob pattern
  --collect-only        Collect tests but do not run
  --packages-repo PACKAGES_REPO
                        Base URL for package git repos
  --overwrite           Overwrite existing outputs directory instead of
                        deleting

For the example directory structure, doing run_testr (with no custom options) would run the tests, reporting test status for each test and then finish with a summary of test status like so:

*************************************************
***    Package           Script        Status ***
*** ------------- -------------------- ------ ***
*** other_package test_regress_long.sh   pass ***
*** other_package post_regress_long.py   pass ***
***    py_package      test_regress.sh   pass ***
***    py_package         test_unit.py   pass ***
***    py_package   post_check_logs.py   pass ***
***    py_package      post_regress.py   pass ***
*************************************************

The working directory would contain the following new sub-directories. The first step in processing is to copy the test and post-process scripts from packages into outputs/<version_id>, where <version_id> is the output of get_version_id. The scripts are then run from that directory, so they are free to write outputs directly into the current directory or any sub-directories therein.

# Testing and post-process scripts and outputs
outputs/
  last -> 0.18-r609-0d91665
  0.18-r609-0d91665/
    logs/
      all_tests.json        # Master log file in JUnit's XML format
      test.log              # Master log file of test processing and results
      py_package/
        helper_script.py
        test_unit.py
        test_unit.py.log    # Log file from running test_unit.py
        test_regress.sh
        test_regress.sh.log # Log file
        post_regress.py
        post_regress.py.log # Log file
        post_check_logs.py
        post_check_logs.py.log
        out.dat             # Example data file from test_regress.sh
        index.html          # Example web page from test_regress.sh
      other_package/
        test_regress_long.sh
        test_regress_long.sh.log
        post_regress_long.py
        post_regress_long.py.log
        big_data.dat        # More data

    # Regression outputs, copied from outputs/ by post_regress* scripts
    regress/
      py_package/
        out.dat             # Example data file from test_regress.sh
        index.html          # Example web page from test_regress.sh
      other_package/
        big_data.dat        # More data

Selecting tests¶

The --include and --exclude options can be used to select specific tests and post-process scripts. The default is to include all files.

The values for both options are a comma-separated list of shell file “glob” patterns that are used to match files. The selection filtering first finds all files that match the --include argument (which defaults to *), and then removes any files that match the --exclude argument (which defaults to None).

The glob patterns are compared to the full filename that includes the package name and test name, for instance py_package/test_regress.sh.

For practical convenience, any specified values (e.g. py_package) automatically have a * appended. Some examples:

$ run_testr --include=py_package  # py_package (but also py_package_2)
$ run_testr --include=py_package/  # py_package only
$ run_testr --exclude=py_package/  # exclude py_package but run all others
$ run_testr --exclude='*_long'    # exclude long tests
$ run_testr --exclude='*_long,*_unit' # exclude long and unit tests

Comparing regression outputs¶

In this example the 0.18-r609-0d91665 regress outputs might correspond to the current production installation. Then suppose you create a test environment with version id 0.19-r610-1b65555 and run the full test suite. Then you can do:

$ cd regress
$ diff -r 0.18-r609-0d91665/ 0.19-r610-1b65555/
...

A key point is that effort should be made to clean the regression outputs to strip them of uninteresting diffs like a date or the user that ran the tests. See the post_regress.py example for more discussion on that.

One option from here is to copy the regress outputs into a git-versioned repo in a new branch and then use GitHub to do comparisons.

Test specification files¶

One might like to maintain a set of unit and regression tests in a single testing repository but to run somewhat different combinations of those tests in different circumstances. For instance one testing environment might not have the necessary resources to run all tests. The tests themselves might manage this (i.e. pytest “skip”) but it may be simpler and more explicit to manage these lists of tests directly.

The --test-spec command line option provides a way to do this. If you provide an option like --test-spec=HEAD then the following happens:

A file in the current directory named HEAD is opened and read:
- It must contain a list of include / exclude specifications like those for the -include and -exclude options.
- The exclude specifications are preceded by a minus sign (-).
- Blank lines or those starting with # are ignored.
The test specification file include and exclude files are appended to any provided on the command line.
The regression outputs are put into a sub-directory HEAD within the regress directory.

Example test specification file:

# Includes
acisfp_check
dpa_check

# Excludes
-*_long*

File recipes¶

Here we show some example testing files taken from ska_testr GitHub repository in order to illustrate some useful recipes or idioms.

test_unit.py¶

Python packages that have built-in (inline) unit tests (see Python testing helpers) can be tested with the following:

import testr
testr.testr()

This uses pytest to run the included tests for the package. By default it provides flags to run in verbose mode with all test code output copied to the console. This allows any stray warnings to be emitted.

You can do the same thing without using the testr.testr() function as follows. The important thing is to raise an exception if there were test failures:

import xija
n_fail = xija.test('-v', '-s', '-k not test_minusz')
if n_fail > 0:
    raise ValueError(str(n_fail) + ' test failures')

test_unit_git.sh¶

Python packages that do not have inline unit tests or for other reasons require the source repo for unit testing can use the following recipe:

/usr/bin/git clone ${TESTR_PACKAGES_REPO}/${TESTR_PACKAGE}
cd ${TESTR_PACKAGE}
git checkout master
py.test -v -s

This illustrates some environment variables that are always defined when running test code.

post_regress.py¶

Here is an example from the Ska dpa_check test that copies a few key outputs into the regress directory after stripping out the user name and run time.

from testr.packages import make_regress_files

regress_files = ['out/index.rst',
                 'out/run.dat',
                 'out/states.dat',
                 'out/temperatures.dat']

clean = {'out/index.rst': [(r'^Run time.*', '')],
         'out/run.dat': [(r'#.*py run at.*', '')]}

make_regress_files(regress_files, clean=clean)

post_check_logs.py¶

Here is an example again from the Ska dpa_check test that checks the test processing log files for any occurrences of warning or error. In this case there are a couple of lines where this is expected, so those matches are ignored. Any other matches will raise an exception and signal a FAIL for this package:

from testr.packages import check_files

check_files('test_*.log',
            checks=['warning', 'error'],
            allows=['99% quantile value of', 'in output at out'])

test_regress_long.sh¶

This is an example that is not really a recipe but a demonstration of doing a complex regression test that requires some setup and uses several scripts to regenerate database tables from scratch. It then uses a helper script those to write a subset of database contents in a clean ASCII format for regression comparisons. This is in the kadi package.

# Get three launcher scripts manage.py update_events and update_cmds
/usr/bin/git clone ${TESTR_PACKAGES_REPO}/kadi

# Copy some scripts to the current directory
cd kadi
cp manage.py update_events update_cmds ltt_bads.dat ../
cd ..
rm -rf kadi

export KADI=$PWD
./manage.py syncdb --noinput

START='2015:001'
STOP='2015:030'

./update_events --start=$START --stop=$STOP
./update_cmds --start=$START --stop=$STOP

# Write event and commands data using test database
./write_events_cmds.py --start=$START --stop=$START --data-root=events_cmds

Another complicated example is in the Ska.engarchive package. This one is slightly different because it generates new database values and then immediately compares with the current production database.

Summary Logs¶

Testr produces a summary log which includes all tests run. It parses the logs produced by pytest, and tests are grouped in suites following the test hierarchy. Tests that do not use pytest are grouped in a test suite at the top level. The log is written in JSON format and looks something like the following:

{
  "test_suite": {
    "name": "Quaternion-tests",
    "package": "Quaternion",
    "test_cases": [
      {
        "name": "post_check_logs.py",
        "file": "Quaternion/post_check_logs.py",
        "timestamp": "2020:06:16T09:43:13",
        "log": "Quaternion/post_check_logs.py.log",
        "status": "fail",
        "failure": {
          "message": "post_check_logs.py failed",
          "output": null
        }
      }
    ],
    "timestamp": "2020:06:16T09:43:13",
    "properties": {
      "system": "Darwin",
      "architecture": "64bit",
      "hostname": "saos-MacBook-Pro.local",
      "platform": "Darwin-19.5.0",
      "package": "Quaternion",
      "package_version": "3.5.2.dev9+g7ee8b10.d20200616",
      "t_start": "2020:06:16T09:43:13",
      "t_stop": "2020:06:16T09:43:14",
      "regress_dir": null,
      "out_dir": "Quaternion"
    }
  },
  "test_suites": [
    {
      "test_cases": [
        {
          "name": "test_shape",
          "classname": "Quaternion.tests.test_all",
          "file": "Quaternion/tests/test_all.py",
          "line": "43",
          "status": "pass"
        },
        {
          "name": "test_init_exceptions",
          "classname": "Quaternion.tests.test_all",
          "file": "Quaternion/tests/test_all.py",
          "line": "50",
          "failure": {
            "message": "Exception: Unexpected exception here",
            "output": "def test_init_exceptions():\n>       raise Exception('Unexpected exception here')\nE       Exception: Unexpected exception here\n\nQuaternion/tests/test_all.py:52: Exception"
          },
          "status": "fail"
        },
        {
          "name": "test_from_q",
          "classname": "Quaternion.tests.test_all",
          "file": "Quaternion/tests/test_all.py",
          "line": "83",
          "skipped": {
            "message": "no way of currently testing this",
            "output": "Quaternion/tests/test_all.py:83: <py._xmlgen.raw object at 0x7f9ca044fb38>"
          },
          "status": "skipped"
        }
      ],
      "name": "Quaternion-pytest",
      "properties": {
        "system": "Darwin",
        "architecture": "64bit",
        "hostname": "saos-MacBook-Pro.local",
        "platform": "Darwin-19.5.0",
        "package": "Quaternion",
        "package_version": "3.5.2.dev9+g7ee8b10.d20200616",
        "t_start": "2020:06:16T09:43:11",
        "t_stop": "2020:06:16T09:43:13",
        "regress_dir": null,
        "out_dir": "Quaternion"
      },
      "log": "Quaternion/test_unit.py.log",
      "hostname": "saos-MacBook-Pro.local",
      "timestamp": "2020:06:16T09:43:11",
      "package": "Quaternion",
      "file": "Quaternion/test_unit.py"
    }
  ]
}

Python testing helpers¶

The modules testr.runner and testr.setup_helper provide the infrastructure to enable easy and uniform running of pytest test functions in two ways:

By importing the package (locally or installed) and doing <package>.test(*args, **kwargs)
Within a local development repo using python setup.py test --args='arg1 kwarg2=val2' where arg1 and kwarg2 are valid pytest arguments.

`init.py`¶

Typical usage within the package __init__.py file:

def test(*args, **kwargs):
    '''
    Run py.test unit tests.
    '''
    import testr
    return testr.test(*args, **kwargs)

This will run any tests that included as a test sub-package in the package distribution. The following package layout shows an example of such inlined tests:

setup.py   # your setuptools Python package metadata
mypkg/
    __init__.py
    appmodule.py
    ...
    tests/
        test_app.py
        ...

A key advantage of providing inline tests is that they can be run post-install in the production environment, providing positive confirmation that the actual installed package is working as expected. See the next section for an example setup.py file which shows including this inline test sub-package.

`setup.py`¶

Typical usage in setup.py:

from setuptools import setup

try:
    from testr.setup_helper import cmdclass
except ImportError:
    cmdclass = {}

setup(name='my_package',
      packages=['my_package', 'my_package.tests'],
      tests_require=['pytest'],
      cmdclass=cmdclass,
      )

API¶

testr.runner¶

Provide a test() function that can be called from package __init__.

exception testr.runner.TestError[source]¶

testr.runner.get_full_version(calling_frame_globals, calling_frame_filename)[source]¶: Return a full version which includes git info if the module was imported from the git repo source directory.

testr.runner.test(*args, **kwargs)[source]¶

Run py.test unit tests for the calling package with specified args and kwargs.

This temporarily changes to the directory above the installed package directory and effectively runs py.test <packagename> <args> <kwargs>.

If the kwarg raise_exception=True is provided then any test failures will result in an exception being raised. This can be used to make a shell-level failure.

Parameters

*args – positional args to pass to pytest
raise_exception – test failures raise an exception (default=False)
package_from_dir – set package name from parent directory name (default=False)
verbose – run pytest in verbose (-v) mode (default=False)
show_output – run pytest in show output (-s) mode (default=False)
**kwargs – additional keyword args to pass to pytest

Returns

number of test failures

testr.runner.testr(*args, **kwargs)[source]¶

Run py.test unit tests for the calling package. This just calls the test() function but includes defaults that are more appropriate for integrated package testing using run_testr.

Parameters

*args – positional args to pass to pytest
raise_exception – test failures raise an exception (default=True)
package_from_dir – set package name from parent directory name (default=True)
verbose – run pytest in verbose (-v) mode (default=True)
show_output – run pytest in show output (-s) mode (default=True)
**kwargs – additional keyword args to pass to pytest

Returns

number of test failures

testr.setup_helper¶

Define a test runner command class suitable for use in setup.py so that python setup.py test runs tests via pytest.

class testr.setup_helper.PyTest(dist, **kw)[source]¶

Test runner command class suitable for use in setup.py so that python setup.py test runs tests via pytest.

initialize_options()[source]¶

Set default values for all the options that this command supports. Note that these defaults may be overridden by other commands, by the setup script, by config files, or by the command-line. Thus, this is not the place to code dependencies between options; generally, ‘initialize_options()’ implementations are just a bunch of “self.foo = None” assignments.

This method must be implemented by all command classes.

testr.test_helper¶

Provide helper functions that are useful for unit testing.

testr.test_helper.has_dirs(*paths)[source]¶

All of the paths exist and each is a directory

Input path(s) can contain ~ (home dir) or environment variables like $SKA or ${SKA}.

testr.test_helper.has_paths(*paths)[source]¶

All of the paths exist

Input path(s) can contain ~ (home dir) or environment variables like $SKA or ${SKA}.

testr.test_helper.has_sybase()[source]¶

Return True if the system apparently can run Sybase queries from Python.

In detail, this is True if the SYBASE and SYBASE_OCS env variables are set and the correct Python shared object library exists on the system.

testr.test_helper.on_head_network()[source]¶

Return True if the system is apparently on the HEAD network.

This looks for a list of subnets corresponding to linux machines listed in the HEAD nodeinfo database. Last updated on 2022-08-30. It also requires that the path /proj/sot/ska exists (because there are machines on those subnets that do not have /proj/sot/ska on their filesystem).

testr¶

Package testing: unit, regression, integration¶

Basic structure¶

get_version_id¶

Packages¶

Test scripts¶

Post-processing scripts¶

Environment variables¶

Filename conventions¶

Running the tests¶

Selecting tests¶

Comparing regression outputs¶

Test specification files¶

File recipes¶

test_unit.py¶

test_unit_git.sh¶

post_regress.py¶

post_check_logs.py¶

test_regress_long.sh¶

Summary Logs¶

Python testing helpers¶

`init.py`¶

`setup.py`¶

API¶

testr.runner¶

testr.setup_helper¶

testr.test_helper¶

Page Contents

This Page

Navigation

testr¶

Package testing: unit, regression, integration¶

Basic structure¶

get_version_id¶

Packages¶

Test scripts¶

Post-processing scripts¶

Environment variables¶

Filename conventions¶

Running the tests¶

Selecting tests¶

Comparing regression outputs¶

Test specification files¶

File recipes¶

test_unit.py¶

test_unit_git.sh¶

post_regress.py¶

post_check_logs.py¶

test_regress_long.sh¶

Summary Logs¶

Python testing helpers¶

__init__.py¶

setup.py¶

API¶

testr.runner¶

testr.setup_helper¶

testr.test_helper¶

`init.py`¶

`setup.py`¶