Skip to content

Commit 2eb4ecc

Browse files
authored
Improve execution time of E2E tests (#5763)
## Description This PR further improves the execution time of E2E tests based on the approach introduced in #5757. The overall execution time of the whole E2E test suit (except tests that `require_contract_deplyment`) on my machine went down to almost unbelievable ~12.5 minutes 😄 The PR: - creates reduced versions of `std` by copying latest `std` files, instead of checking-in a copy of those files. This ensures testing against exactly the same code as in `std`. - either fully removes `std` dependencies from `should_fail` tests or replaces them with reduced versions of `std`. - replaces additional `std` dependencies in `should_pass` tests with reduced versions of `std`. - removes dependencies to `core` in tests (mostly `should_pass/language`) that do not need any `core` or `std` features. A note on the change done in `should_pass/stdlib` tests. Some of those tests are changed to use reduced versions of `std` which sounds contradictory, because they should, well, test `std`. However, the first point mentioned above ensures that they test exactly the same code given in the full `std`. ## Checklist - [ ] I have linked to any relevant issues. - [x] I have commented my code, particularly in hard-to-understand areas. - [ ] I have updated the documentation where relevant (API docs, the reference, and the Sway book). - [x] I have added tests that prove my fix is effective or that my feature works. - [ ] I have added (or requested a maintainer to add) the necessary `Breaking*` or `New Feature` labels where relevant. - [x] I have done my best to ensure that my PR adheres to [the Fuel Labs Code Review Standards](https://github.com/FuelLabs/rfcs/blob/master/text/code-standards/external-contributors.md). - [x] I have requested a review from the relevant team or maintainers.
1 parent c02d50c commit 2eb4ecc

File tree

358 files changed

+1017
-2297
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

358 files changed

+1017
-2297
lines changed

sway-lsp/tests/lib.rs

+1
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,7 @@ fn show_ast() {
289289
}
290290

291291
#[test]
292+
#[ignore = "`struct_field_access` test doesn't depend on `core` anymore which makes this test fail because the dependency graph is not the expected one."]
292293
fn visualize() {
293294
run_async!({
294295
let server = ServerState::default();

test/src/e2e_vm_tests/README.md

+10
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,13 @@
1+
# Writing end-to-end VM tests
2+
3+
In order to minimize compilation time of individual tests, strive to reduce dependencies in tests.
4+
5+
To achieve that, follow these guidelines:
6+
- Use `implicite-std = false` if dependency on `core` is not needed. This is often possible when testing `should_pass/language` features.
7+
- Do not use `std` just to conveniently get an arbitrary type or trait. E.g., if a test requires an arbitrary type or trait, go with `struct Dummy {}` or `trait Trait {}` instead of importing `Option` or `Hash`.
8+
- If `std` functionality is needed, import the minimal [reduced `std` library](reduced_std_libs/README.md) that provides the functionality.
9+
- Import the full `std` only if the provided [reduced `std` libraries](reduced_std_libs/README.md) do not provide required types.
10+
111
# Running end-to-end VM tests
212

313
This document assumes you have `fuel-core` running on the default port.

test/src/e2e_vm_tests/mod.rs

+6-2
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ use std::collections::HashSet;
1717
use std::io::stdout;
1818
use std::io::Write;
1919
use std::str::FromStr;
20+
use std::time::Instant;
2021
use std::{
2122
collections::HashMap,
2223
path::{Path, PathBuf},
@@ -659,6 +660,7 @@ pub async fn run(filter_config: &FilterConfig, run_config: &RunConfig) -> Result
659660
let mut number_of_tests_failed = 0;
660661
let mut failed_tests = vec![];
661662

663+
let start_time = Instant::now();
662664
for (i, test) in tests.into_iter().enumerate() {
663665
let cur_profile = if run_config.release {
664666
BuildProfile::RELEASE
@@ -714,6 +716,7 @@ pub async fn run(filter_config: &FilterConfig, run_config: &RunConfig) -> Result
714716

715717
number_of_tests_executed += 1;
716718
}
719+
let duration = Instant::now().duration_since(start_time);
717720

718721
if number_of_tests_executed == 0 {
719722
if let Some(skip_until) = &filter_config.skip_until {
@@ -747,7 +750,7 @@ pub async fn run(filter_config: &FilterConfig, run_config: &RunConfig) -> Result
747750
} else {
748751
tracing::info!("_________________________________");
749752
tracing::info!(
750-
"Sway tests result: {}. {} total, {} passed; {} failed; {} disabled",
753+
"Sway tests result: {}. {} total, {} passed; {} failed; {} disabled [test duration: {}]",
751754
if number_of_tests_failed == 0 {
752755
"ok".green().bold()
753756
} else {
@@ -756,7 +759,8 @@ pub async fn run(filter_config: &FilterConfig, run_config: &RunConfig) -> Result
756759
total_number_of_tests,
757760
number_of_tests_executed - number_of_tests_failed,
758761
number_of_tests_failed,
759-
disabled_tests.len()
762+
disabled_tests.len(),
763+
util::duration_to_str(&duration)
760764
);
761765
if number_of_tests_failed > 0 {
762766
tracing::info!("{}", "Failing tests:".red().bold());
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
*.sw
2+
!lib.sw
3+
!prelude.sw
4+
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Reduced Versions of the Sway Standard Library
2+
3+
This folder contains reduced versions of the Sway Standard Library meant to be used in tests that only need a limited `std` functionality. Until we get incremental compilation or reuse of a once-compiled `std` across tests, we want to minimize dependencies on `std`. The reason for this is the significant increase in test compilation time when `std` gets compiled as a test dependency.
4+
5+
In general, when writing a test try to avoid depending on `std` unless the test actually needs `std` functionality.
6+
7+
If the test depends only on a narrow subset of `std`, try first to include one of the following reduced versions of `std` as dependency. The compilation time of reduced versions is negligible compared to the compile time of the whole `std`.
8+
9+
If none of the reduced versions contain the modules needed by the test, include the whole `std` from the `sway-lib-std`.
10+
11+
## Content of the Reduced Versions
12+
13+
Each reduced version brings a small additional functionality on top of the previous one. The versions are listed below, ordered by increasing functionality.
14+
15+
### `assert` (in `sway-lib-std-assert`)
16+
Contains:
17+
- asserting
18+
- logging
19+
- reverting
20+
21+
### `option-result` (in `sway-lib-std-option-result`)
22+
Contains:
23+
- everything available in `assert`
24+
- `Option`
25+
- `Result`
26+
27+
### `vec` (in `sway-lib-std-vec`)
28+
Contains:
29+
- everything available in `option-result`
30+
- `Vec`
31+
- `Iterator` trait
32+
- `From` and `Into` traits
33+
34+
### `conversions` (in `sway-lib-std-conversions`)
35+
Contains:
36+
- everything available in `vec`
37+
- intrinsics
38+
- `Bytes`
39+
- bytes conversions
40+
- array conversions
41+
- primitive conversions

test/src/e2e_vm_tests/reduced_std_libs/sway-lib-std-assert/README.md

-7
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
assert.sw
2+
error_signals.sw
3+
logging.sw
4+
revert.sw

test/src/e2e_vm_tests/reduced_std_libs/sway-lib-std-assert/src/assert.sw

-124
This file was deleted.

test/src/e2e_vm_tests/reduced_std_libs/sway-lib-std-assert/src/error_signals.sw

-37
This file was deleted.

test/src/e2e_vm_tests/reduced_std_libs/sway-lib-std-assert/src/logging.sw

-33
This file was deleted.

test/src/e2e_vm_tests/reduced_std_libs/sway-lib-std-assert/src/revert.sw

-73
This file was deleted.

0 commit comments

Comments
 (0)