You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: test-runner/README.md
+20-5
Original file line number
Diff line number
Diff line change
@@ -30,6 +30,7 @@ A test is defined as a set of commands to be executed along with their associate
30
30
|[volumes](https://github.com/compose-spec/compose-spec/blob/master/spec.md#volumes)| Optional[List[[Volume](utils/test.py#L13)]]| A list of volumes to be mounted when running the test in a container. |
31
31
|[env](https://github.com/compose-spec/compose-spec/blob/master/spec.md#environment)| Optional[Dict[str, str]]| A list of environment variables to be set when the test is running. |
32
32
| mask | Optional[List[str]]| A list of keys to [mask](#masking) in the test output. |
33
+
| performance | Optional[str]| Check test performance thresholds in the format `perf/path/to/model.yaml:test-id`|
33
34
| notebook | Optional[str]| A flag indicating whether the test utilizes a [jupyter notebook](#notebook-test). |
34
35
| serving | Optional[str]| A flag indicating whether a [serving test](#serving-test) should be invoked. |
@@ -75,12 +76,12 @@ In the example above, the first output will be `hello`, and the second output wi
75
76
76
77
Masking is a feature that allows you to hide sensitive information in the logs generated by the test runner. This is useful when you want to prevent benchmark information from being publicly exposed.
77
78
78
-
To enable masking, add the `mask` parameter to your `tests.yaml` file as a list of strings. Each string should be a key whose value you want to mask without any kind of delimiter.
79
+
To enbable masking, add the `mask` parameter to your `tests.yaml` file as a list of strings. Each string should be a key whose value you want to mask without any kind of delimiter.
79
80
80
-
By default, masking is not enabled. To enable masking, use the `-m` flag when running the test runner application.
81
+
By default, masking is enabled. To disable masking, add `"mask": [false]` to your `.actions.json` file.
81
82
82
83
```bash
83
-
python -m -f path/to/tests.yaml
84
+
python -f path/to/tests.yaml
84
85
```
85
86
86
87
```bash
@@ -92,6 +93,21 @@ test:
92
93
93
94
In the example above, the output will be `hello:***`
94
95
96
+
#### Performance Thresholds
97
+
98
+
You can utilize performance thresholds stored in another github repository by providing the `PERF_REPO` environment variable in GitHub's `org-name/repo-name` format.
A notebook test is a special type of test designed to run Jupyter notebooks. This is indicated by setting the notebook attribute to `True` in the test definition. When a test is marked as a notebook test, the command specified in the cmd attribute is expected to be [papermill](https://github.com/nteract/papermill) command. If papermill is not already installed in the provided `image` property, then it will be installed.
@@ -139,7 +155,7 @@ For more options, see the `--help` output below:
0 commit comments