test_perimeter

TODO: Maybe keep this FS for the overall "test perimeter" concept, but add separate one for each.

Test perimeters

There are the following test perimeters (from min to max):

  • test_fast_slim_max_mocked

    These are the thinnest perimeter. It tries to mock every call outside the function or class.

  • test_fast_fat_min_mocked

    This tries to mock all external OS calls (like os.execv or file IO). The code still executes within the same OS process with the test runner. This allows running (almost) end-to-end tests in debug mode. The tests are also very fast relatively to test_slow_integrated.

  • test_slow_integrated

    This is the thickest perimeter. It does not mock anything and calls external commands. It is relatively slow (especially, for installation of packages via pip), but provides the confidence that things are not broken in realistic conditions.

    It is disabled by default unless CI env var is set to "truthy" value.

PyCharm test configurations

To run tests in PyCharm, there are two configurations:

test_fast_fat_min_mocked limitations

Due to the nature of the protoprimer, this test mode affects some prod code.

To make it explicit, all such prod code extensions work with EnvVar.var_PROTOPRIMER_MOCKED_RESTART.

The main condition: the venv of the test runner is not escape-able. This means proto_code cannot be selected automatically (proto_code will always be from the venv of the test runner). This means EnvVar.var_PROTOPRIMER_PROTO_CODE must also be set for such tests for the bootstrap process to complete.

Another obstacle is python process (re-)starts (via switch_python function) - this does not happen in tests (otherwise, test runner would die). Any prod code which assumes single execution per python process has to be changed to support such tests.

Test run detection

In general, some prod code works undesireably under tests - to avoid that, see is_test_run function to change test and prod code behavior.