Testing

GVEC has a set of testcases and test logic contained within test-CI.

• The test logic is implemented with pytest.

• The tests can be executed locally using pytest with many options to customize which tests should be run (see Usage).

• A predefined set of tests can be executed using ctest, after the cmake install process. Simply change to the build directory, and execute:

  ctest -T test --output-on-failure -R
  

• A CI pipeline with automatic tests is configured for the repository, using shared MPCDF gitlab runners to execute the tests.

  • More details on the CI setup see: Continuous Integration.

  • The CI manages different builds of the code, then calls pytest for running them and checking the results (requires python >3.10 to be installed!).

Usage

• Call pytest or python -m pytest in the test-CI directory or the gvec root directory to run automatic tests.

• With the -m "(MARKER_A or MARKER_B) and MARKER_C" option tests can be selected or deselected based on their markers.

  • Currently, these custom markers specify a testgroup:

    • example: runs all testcases from test-CI/examples/*

    • shortrun: runs all testcases from test-CI/examples/*, saved in RUNDIR/shortrun/*, changing the parameters to one iteration, and no visualization.

    • debugrun: runs all testcases from test-CI/examples/*, saved in RUNDIR/debugrun/*, changing the parameters to one iteration and trigger additional testing during runtime.

  • Additionally, the marker restart exists, that executes all testcases that are named “BASENAME_restart”, which have a dependency on the base testcase named “BASENAME”. The base testcase must be run before!

  • Currently, these custom markers specify a stage:

    • run_stage: runs all testcases of all testgroups (or only of those specified).

    • regression_stage: runs a comparison of the RUNDIR and the runs of a reference in directory REFDIR, for all testcases of all testgroups (or only of those specified). Depends on a run_stage executed before!

    • post_stage: runs post-processing/visualization of the tests in RUNDIR, saved into POSTDIR/post, for all testcases of all testgroups (or only of those specified). Depends on a run_stage executed before!

    • converter_stage: runs post-processing using all converters of the tests in RUNDIR, saved into POSTDIR/converter-name, for all testcases of all testgroups (or only of those specified). Depends on a run_stage executed before!

  • Notes:

    • Only one stage should be specified in a `pytest`` run!

    • Testgroups are exclusive. Testgroups are combined with a stage using and.

  • Examples:

    • -m "run_stage": runs all tests of all testgroups

    • -m "run_stage and example": runs tests marked with example

    • -m "run_stage and (not example)": run all tests except those marked with example

    • -m "run_stage and (example or shortrun)": runs tests marked with example and also the tests marked with shortrun

    • -m "post_stage and shortrun": runs post-processing on the tests saved in RUNDIR/shortrun

• The relevant paths (relative or absolute) are supplied to pytest with:

  • --builddir="PATH/DIRNAME": a directory containing the build that was generated by cmake. Default is gvec_root/build

  • --rundir="PATH/DIRNAME": a directory for the runs. Default is gvec_root/test-CI/run

  • --refdir="PATH/DIRNAME": a reference directory, if --refdir is not specified, the regression tests are skipped

  • --postdir="PATH/DIRNAME": a directory for the post-processing. Default is gvec_root/test-CI/post

• With the -k "(CASE_A or CASE_B) and (not CASE_C)" option tests can be selected or deselected based on their names, which includes the python function name of the test, and the folder name of the testcase.

• With --dry-run, only the folders and parameter files are setup, but nothing is executed or compared. Good for testing the pytest command.

More Examples

• Simulate a test run, without actually executing the tests. All tests are displayed and run folders and files are created, along with a dry-run output:

  python -m pytest -v -m "run_stage" --dry-run
  

• Run all end2end tests using BUILDDIR/bin/gvec and store the results at RUNDIR:

  python -m pytest -v -m "run_stage" --builddir="BUILDDIR" --rundir="RUNDIR"
  

• Run the file comparison between RUNDIR and REFDIR

  python -m pytest -v -m "regression_stage" --builddir="BUILDDIR" --rundir="RUNDIR" --refdir="REFDIR"`
  

Details

• Details on the gitlab CI setup are found at Continuous Integration

• the main python script for all tests is in test-CI/test_all.py

• Large input files should be stored in test-CI/data and the examples should contain a relative symbolic link

  • When filling a RUNDIR, a link RUNDIR/data -> test-CI/data is created. In this way the relative link should still work.

• The test logic is contained in test_all.py:test_examples but relies on other functions:

  • the test logic for a successful GVEC run is in helpers.py:assert_empty_stderr and assert_finished_stdout

  • the test logic for file comparison in helpers.py:assert_equal_files