Skip to content

Commit 9bd9c37

Browse files
committed
Document our policy about performance testing environment and epochs
1 parent 73be50d commit 9bd9c37

File tree

1 file changed

+11
-1
lines changed

1 file changed

+11
-1
lines changed

docs/team/ci.md

+11-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Continuous Integration
22

3-
## Testing
3+
## Correctness Testing
44

55
MMTK core runs CI tests *before* a pull request is merged.
66

@@ -11,3 +11,13 @@ MMTk core sets up two sets of tests, the *minimal tests* and the *extended tests
1111
* Extended tests only run for a pull request if the pull request is tagged with the label `PR-extended-testing`. This set of tests
1212
may take hours, and usually include integration tests with bindings which run the language implementation's standard test suite
1313
as much as possible.
14+
15+
## Performance Testing
16+
17+
We conduct performance testing for each MMTk core commit after it has been merged.
18+
19+
### Testing Environment and Epochs
20+
21+
We track the performance of MMTk over years. Naturally, changes in the testing environment (hardware or software) and methodology are sometimes necessary. Each time we make such a change, it marks the start of a new *epoch* in our performance evaluation.
22+
23+
Since changes in the testing environment can significantly impact performance, we do not directly compare performance results across different epochs. Within an epoch, we ensure that **MMTk does not experience performance regressions**, and **we only update the testing environment when there is no performance regression in the current epoch**.

0 commit comments

Comments
 (0)