Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add docs on running pytest under TSAN #138

Merged
merged 4 commits into from
Feb 17, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/debugging.md
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,8 @@ CC=clang-18 CXX=clang++-18 python -m pip install -v . --no-build-isolation -Cset
# CC=clang-18 CXX=clang++-18 python -m pip install -v . --no-build-isolation -Csetup-args=-Db_sanitize=thread -Csetup-args=-Dbuildtype=debugoptimized
```

## Running python under TSAN

### Useful TSAN options

- By default TSAN reports warnings. To stop execution on TSAN errors, use:
Expand All @@ -365,6 +367,8 @@ CC=clang-18 CXX=clang++-18 python -m pip install -v . --no-build-isolation -Cset
TSAN_OPTIONS=halt_on_error=1 python -m pytest test.py
```

See [the TSAN documentation](https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags) for a full listing of options accepted by TSAN.

- To add TSAN suppressions (written in a file: `tsan-suppressions`):

```bash
Expand All @@ -380,4 +384,51 @@ race:dnnl_sgemm
export TSAN_OPTIONS="suppressions=$PWD/tsan-suppressions" python -m pytest test.py
```

### Running pytest tests under TSAN

By default, pytest [captures all output from
tests](https://docs.pytest.org/en/stable/how-to/capture-stdout-stderr.html),
this means that you might only see output like `ThreadSanitizer: reported 2 warnings`, but with no accompanying report with details about the warning.

To ensure that pytest doesn't capture any output from ThreadSanitizer, you can
pass `-s` (short for `--show-capture`) to your pytest invocation.

Some authors of this guide have observed hangs running pytest with
`halt_on_error=1`. If you observe hangs, try setting `halt_on_error=0` in
TSAN_OPTIONS.

The [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) plugin can also
sometimes be problematic if a test runner happens to crash during
execution. While `pytest-xdist` does have some support for detecting crashed
worker, it is not foolproof and the authors of this guide have observed hangs on
CI due to pytest-xdist not properly handling a worker failing due to a TSAN
error.

The `pytest-xdist` plugin also [makes it impossible to obtain stdout from
a test runner](https://github.com/pytest-dev/pytest-xdist/issues/82), so there
is no way to see TSAN output if there is an issue. This can lead to hangs on CI
machines with no accompanying error report to explain the nature of the
hang. For that reason we suggest uninstalling `pytest-xdist` from your
environment to ensure it isn't used. If you need to use `pytest-xdist` to make
the tests complete in a reasonable amount of time, we suggest using
[`pytest-timeout`](https://pypi.org/project/pytest-timeout/) to ensure hung
tests eventually exit, particularly on CI.

TSAN includes a check to ensure allocators never fail. This can lead to runtime
crashes if a test happens to try allocating a very large block of memory
specifically to ensure such an allocation does fail correctly. Set
`allocator_may_return_null=1` in `TSAN_OPTIONS` to avoid this.

If a TSAN warning is detected, the exit code of the running process will be set
to a nonzero value (66, by default). If for some reason that is problematic in
your test suite then you can set `exitcode=0` in `TSAN_OPTIONS` to make TSAN
quit "successfully" if a warning is detected. For example, you might set this if
a subprocess returning a nonzero exit code unexpectedly breaks a test.

Altogether, a pytest invocation using TSAN might look like:

```
$ TSAN_OPTIONS='allocator_may_return_null=1 halt_on_error=1' pytest -s
```

[^1]: This feature is not correctly working on `lldb` after CPython 3.12.