# Testing

GVEC has a set of testcases and test logic contained within `test-CI`.

* The test logic is implemented with [pytest](https://docs.pytest.org).
* The tests can be executed locally using *pytest* with many options to customize which tests should be run (see [](#usage)).
* A predefined set of tests can be executed using *ctest*, after the [cmake install process](/user/install). Simply change to the build directory, and execute:
    ```bash
    ctest -T test --output-on-failure -R
    ```
* A CI pipeline with automatic tests is configured for the [](http://gitlab.mpcdf.mpg.de/gvec-group/gvec) repository, using shared MPCDF gitlab runners to execute the tests.
    * More details on the CI setup see: [](pipeline).
    * The CI manages different builds of the code, then calls pytest for running them and checking the results (requires `python >3.10` to be installed!).

## Usage

* Call `pytest` or `python -m pytest` in the `test-CI` directory or the gvec root directory to run automatic tests.
* With the `-m "(MARKER_A or MARKER_B) and MARKER_C"` option tests can be selected or deselected based on their markers.
    * Currently, these custom markers specify a **testgroup**:
        * `example`: runs all testcases from `test-CI/examples/*`
        * `shortrun`: runs all testcases from `test-CI/examples/*`, saved in `RUNDIR/shortrun/*`, changing the parameters to one iteration, and no visualization.
        * `debugrun`: runs all testcases from `test-CI/examples/*`, saved in `RUNDIR/debugrun/*`, changing the parameters to one iteration and trigger additional testing during runtime.
    * Additionally, the marker `restart` exists, that executes all testcases that are named "BASENAME_restart", which have a dependency on the base testcase named "BASENAME". The base testcase must be run before!
    * Currently, these custom markers specify a **stage**:
        * `run_stage`: runs all testcases of all testgroups (or only of those specified).
        * `regression_stage`: runs a comparison of the `RUNDIR` and the runs of a reference in directory `REFDIR`, for all testcases of all testgroups (or only of those specified). Depends on a `run_stage` executed before!
        * `post_stage`: runs post-processing/visualization of the tests in `RUNDIR`, saved into `POSTDIR/post`, for all testcases of all testgroups (or only of those specified). Depends on a `run_stage` executed before!
        * `converter_stage`: runs post-processing using all converters of the tests in `RUNDIR`, saved into `POSTDIR/converter-name`, for all testcases of all testgroups (or only of those specified). Depends on a `run_stage` executed before!
    * **Notes:**
        * Only one stage should be specified in a `pytest`` run!
        * Testgroups are exclusive. Testgroups are combined with a stage using `and`.
    * **Examples:**
        * `-m "run_stage"`: runs all tests of all testgroups
        * `-m "run_stage and example"`: runs tests marked with `example`
        * `-m "run_stage and (not example)"`: run all tests except those marked with `example`
        * `-m "run_stage and (example or shortrun)"`: runs tests marked with `example` and also the tests marked with `shortrun`
        * `-m "post_stage and shortrun"`: runs post-processing on the tests saved in `RUNDIR/shortrun`
* The relevant paths (relative or absolute) are supplied to `pytest` with:
    * `--builddir="PATH/DIRNAME"`: a directory containing the build that was generated by cmake. **Default** is `gvec_root/build`
    * `--rundir="PATH/DIRNAME"`: a directory for the runs. **Default** is `gvec_root/test-CI/run`
    * `--refdir="PATH/DIRNAME"`: a reference directory, if `--refdir` **is not specified, the regression tests are skipped**
    * `--postdir="PATH/DIRNAME"`: a directory for the post-processing. **Default** is `gvec_root/test-CI/post`
* With the `-k "(CASE_A or CASE_B) and (not CASE_C)"` option tests can be selected or deselected based on their names, which includes the python function name of the test, and the folder name of the testcase.
* With `--dry-run`, only the folders and parameter files are setup, but nothing is executed or compared. Good for testing the `pytest` command.

### More Examples

* Simulate a test run, without actually executing the tests. All tests are displayed and run folders and files are created, along with a dry-run output:
  ```bash
  python -m pytest -v -m "run_stage" --dry-run
  ```
* Run all end2end tests using `BUILDDIR/bin/gvec` and store the results at `RUNDIR`:
  ```bash
  python -m pytest -v -m "run_stage" --builddir="BUILDDIR" --rundir="RUNDIR"
  ```
* Run the file comparison between `RUNDIR` and `REFDIR`
  ```bash
  python -m pytest -v -m "regression_stage" --builddir="BUILDDIR" --rundir="RUNDIR" --refdir="REFDIR"`
  ```

## Details

* Details on the gitlab CI setup are found at <project:pipeline.md>
* the main python script for all tests is in `test-CI/test_all.py`
* Large input files should be stored in `test-CI/data` and the examples should contain a *relative* symbolic link
    * When filling a `RUNDIR`, a link `RUNDIR/data -> test-CI/data` is created. In this way the relative link should still work.
* The test logic is contained in `test_all.py:test_examples` but relies on other functions:
    * the test logic for a successful GVEC run is in `helpers.py:assert_empty_stderr` and `assert_finished_stdout`
    * the test logic for file comparison in `helpers.py:assert_equal_files`