Testing methodology

ionics-fits is heavily tested using both unit tests and fuzzing.

Unit Tests

  • run using poe test

  • to run a subset of tests use the -k flag e.g. poe test -k "rabi" to only run tests with the word “rabi” in their name. For more information about configuring pytest see the documentation

  • all tests must pass before a PR can be merged into master as well as static analysis based on pytype, run using poe types (Linux only)

  • PRs to add new models will only be merged once they have full test coverage

  • unit tests aim to provide good coverage over the space of “reasonable datasets”. There will always be corner-cases where the fits fail and that’s fine; the aim here is to cover the main cases users will hit in the wild

  • when a user hits a case in the wild where the fit fails unexpectedly (i.e. we think the fit code should have handled it), a regression test based on the failing dataset should be added

  • unit tests should be deterministic. Synthetic datasets should be included in the test rather than randomly generated at run time. Tip: while writing a test it’s fine to let the test code generate datasets for you. Once you’re happy, run the test in verbose mode and copy the dataset from the log output

Fuzzing

  • fuzzing is non-deterministic (random parameter values, randomly generated datasets) exploration of the parameter space.

  • used when developing / debugging fits, but not automatically run by CI

  • run with poe fuzz (see --help for details)

  • fit failures during fuzzing are not automatically considered a bug; unlike unit tests, fuzzing explores the “unreasonable” part of parameter space as well. Indeed, a large part of the point of fuzzing is to help the developer understand what should be considered “reasonable” (this information should end up in the documentation for the fuzzed model).

  • fuzzing is considered a tool to help developers finding corner-cases. We don’t aim for the same level of code coverage with fuzzing that we do with the unit tests which should, for example, cover ever code path in every parameter estimator