You need to install some dependencies. For Ubuntu you can run:
sudo apt-get install clang lldYou also need to uncomment these lines in the cargo config.toml file.
[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
rustflags = ["-Clink-arg=-fuse-ld=lld", "-Clink-arg=-Wl,--no-rosegment"]Follow the flamegraph instructions for installation.
Apart from running the tracker you will need to run some request if you want to profile services while they are processing requests.
You can use the aquatic UDP load test script.
To generate the graph you will need to:
- Build the tracker for profiling.
- Run the aquatic UDP load test.
- Run the tracker with flamegraph and profiling configuration.
cargo build --profile=release-debug --bin=profiling
./target/release/aquatic_udp_load_test -c "load-test-config.toml"
sudo TORRUST_TRACKER_CONFIG_TOML_PATH="./share/default/config/tracker.udp.benchmarking.toml" /home/USER/.cargo/bin/flamegraph -- ./target/release-debug/profiling 60NOTICE: You need to install the aquatic_udp_load_test program.
The output should be like the following:
Loading configuration file: `./share/default/config/tracker.udp.benchmarking.toml` ...
Torrust successfully shutdown.
[ perf record: Woken up 23377 times to write data ]
Warning:
Processed 533730 events and lost 3 chunks!
Check IO/CPU overload!
[ perf record: Captured and wrote 5899.806 MB perf.data (373239 samples) ]
writing flamegraph to "flamegraph.svg"
NOTICE: You need to provide the absolute path for the installed flamegraph app if you use sudo. Replace /home/USER/.cargo/bin/flamegraph with the location of your installed flamegraph app. You can run it without sudo but you can get a warning message like the following:
WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,
check /proc/sys/kernel/kptr_restrict and /proc/sys/kernel/perf_event_paranoid.
Samples in kernel functions may not be resolved if a suitable vmlinux
file is not found in the buildid cache or in the vmlinux path.
Samples in kernel modules won't be resolved at all.
If some relocation was applied (e.g. kexec) symbols may be misresolved
even with a suitable vmlinux or kallsyms file.
Couldn't record kernel reference relocation symbol
Symbol resolution may be skewed if relocation was used (e.g. kexec).
Check /proc/kallsyms permission or run as root.
Loading configuration file: `./share/default/config/tracker.udp.benchmarking.toml` ...
And some bars in the graph will have the unknown label.
You need to:
- Build an run the tracker for profiling.
- Make requests to the tracker while it's running.
Build and the binary for profiling:
RUSTFLAGS='-g' cargo build --release --bin profiling \
&& export TORRUST_TRACKER_CONFIG_TOML_PATH="./share/default/config/tracker.udp.benchmarking.toml" \
&& valgrind \
--tool=callgrind \
--callgrind-out-file=callgrind.out \
--collect-jumps=yes \
--simulate-cache=yes \
./target/release/profiling 60NOTICE: You should make requests to the services you want to profile. For example, using the UDP load test.
After running the tracker with <valgrind it generates a file callgrind.out
that you can open with kcachegrind.
kcachegrind callgrind.out
Profiling tools:
Talks about profiling:
- Profiling Rust Programs with valgrind, heaptrack, and hyperfine.
- RustConf 2023 - Profiling async applications in Rust by Vitaly Bragilevsky.
- Profiling Code in Rust - by Vitaly Bragilevsky - Rust Linz, December 2022.
- Xdebug 3 Profiling: 2. KCachegrind tour.
Many thanks to Vitaly Bragilevsky and others for sharing the talks about profiling.