Skip to content

Commit

Permalink
add docs on running pytest under TSAN
Browse files Browse the repository at this point in the history
  • Loading branch information
ngoldbaum committed Feb 17, 2025
1 parent ea8b92f commit 3515809
Showing 1 changed file with 44 additions and 0 deletions.
44 changes: 44 additions & 0 deletions docs/debugging.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,8 @@ CC=clang-18 CXX=clang++-18 python -m pip install -v . --no-build-isolation -Cset
TSAN_OPTIONS=halt_on_error=1 python -m pytest test.py
```

See [the TSAN documentation](https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags) for a full listing of options accepted by TSAN.

- To add TSAN suppressions (written in a file: `tsan-suppressions`):

```bash
Expand All @@ -380,4 +382,46 @@ race:dnnl_sgemm
export TSAN_OPTIONS="suppressions=$PWD/tsan-suppressions" python -m pytest test.py
```

### Running pytest tests under TSAN

By default, pytest [captures all output from
tests](https://docs.pytest.org/en/stable/how-to/capture-stdout-stderr.html),
this means that you might only see output like `ThreadSanitizer: reported 2
warnings`, but with no accompanying report with details about the warning.

To ensure that pytest doesn't capture any output from ThreadSanitizer, you can
pass `-s` (short for `--show-capture`) to your pytest invocation.

The [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) plugin can also
sometimes be problematic if a test runner happens to crash during
execution. While `pytest-xdist` does have some support for detecting crashed
worker, it is not foolproof and the authors of this guide have observed hangs on
CI due to pytest-xdist not properly handling a worker failing. The
`pytest-xdist` plugin also [makes it impossible to obtain stdout from a test
runner](https://github.com/pytest-dev/pytest-xdist/issues/82), so there is no
way to see TSAN output if there is an issue. This can lead to hangs on CI
machines with no accompanying error report to explain the nature of the
hang. For that reason we suggest uninstalling `pytest-xdist` from your
environment to ensure it isn't used. If you need to use `pytest-xdist` to make
the tests complete in a reasonable amount of time, we suggest using
[`pytest-timeout`](https://pypi.org/project/pytest-timeout/) to ensure hung
tests eventually exit, particularly on CI.

TSAN includes a check to ensure allocators never fail. This can lead to runtime
crashes if a test happens to try allocating a very large block of memory
specifically to ensure such an allocation does fail correctly. Set
`allocator_may_return_null=1` in `TSAN_OPTIONS` to avoid this.

If a TSAN warning is detected, the exit code of the running process will be set
to a nonzero value (66, by default). If for some reason that is problematic in
your test suite then you can set `exitcode=0` in `TSAN_OPTIONS` to make TSAN
quit "successfully" if a warning is detected. For example, you might set this if
a subprocess returning a nonzero exit code unexpectedly breaks a test.

Altogether, a pytest invocation using TSAN might look like:

```
$ TSAN_OPTIONS='allocator_may_return_null=1 halt_on_error=1' pytest -s
```

[^1]: This feature is not correctly working on `lldb` after CPython 3.12.

0 comments on commit 3515809

Please sign in to comment.