diff --git a/docs/adv/troubleshooting/test_command.md b/docs/adv/troubleshooting/test_command.md
new file mode 100644
index 0000000000..014a944efd
--- /dev/null
+++ b/docs/adv/troubleshooting/test_command.md
@@ -0,0 +1,161 @@
+---
+sidebar_position: 3
+description: Troubleshoot issues spotted by the test command
+---
+
+# Test Commands
+
+This page aims to give guidance on the causes, and potential for troubleshooting or improvement, of failed tests or low test scores from the [Charon Test commands](../../run/prepare/test-command.mdx).
+
+## Peers
+
+### Charon Peers
+
+#### Ping
+
+- Peers might have not started their nodes or are not reachable.
+
+#### PingMeasure
+
+- Peer might be too far away (geographically) from you.
+- If the connection to the peer is indirect, the route is from your node, to the relay, to the peer. Meaning you are measuring the travel time from you to the relay, and from the relay to the peer: (your node -> relay -> peer). This means, even if your peer's node is right next to yours, if the connection is being transmitted through a relay far away, the latency between your nodes might be too high to be effective.
+- Your general network latency to the public internet might be high. Verify with the [`charon test infra`](../../run/prepare/test-command.mdx#test-machine-and-network-performance) tests.
+- If the connection to the peer is indirect, there is a potential that the relay might be overloaded or under-resourced, consider adding [alternative relays](../../adv/security/risks.md#risk-obol-hosting-the-relay-infrastructure), or preferably [opening charon's p2p port](../../learn/charon/networking.mdx#libp2p-relays-and-peer-discovery) to the internet to establish direct peer to peer connections.
+
+#### PingLoad
+
+Same causes as PingMeasure test apply here.
+
+#### DirectConn
+
+- Your or your peer's port might not be publicly exposed.
+- Your or your peer's port might be behind a firewall.
+- Your or your peer's port might be behind a strict NAT gateway.
+
+### Charon Relays
+
+#### PingRelay
+
+- Relay might be down or un-conctactable for other reasons.
+
+#### PingMeasureRelay
+
+- Relay might be under heavy load.
+- Your network latency might be high. Verify with the `charon test infra` tests.
+
+### Self
+
+#### Libp2pTCPPortOpenTest
+
+- There might be another process running on the designated port (tcp/3610 by default).
+- The process might have died.
+
+## Beacon
+
+#### Ping
+
+- Beacon node might not be started or is not reachable.
+
+#### PingMeasure
+
+- Beacon node might be too far away (geographically) from you.
+- Your network latency might be high. Verify with the `charon test infra` tests.
+
+#### Version
+
+- The beacon node version is not compatible with charon.
+
+#### IsSynced
+
+- Beacon node is not synced to the network.
+
+#### PeerCount
+
+- Beacon node does not have enough peers. This may result in slower fetching and broadcasting of slots and duties.
+
+#### PingLoad
+
+This is a load test, to enable it add the `--load-test` flag.
+
+Same causes as PingMeasure test apply here.
+
+#### Simulation
+
+This is a load test, to enable it add the `--load-test` flag.
+
+Same causes as PingMeasure test apply here and additionally:
+
+- The infrastructure on which the beacon node runs (amount of RAM, disk IOPS) might not be enough to handle the number of simulated validators supplied in this test.
+
+## Validator
+
+#### Ping
+
+- Validator client might not be started or is not reachable.
+
+#### PingMeasure
+
+- Validator client might be too far away (geographically) from the charon client. Generally a low latency between a validator client and its charon client is important for timely signing.
+
+#### PingLoad
+
+Same causes as PingMeasure test apply here.
+
+## MEV
+
+#### Ping
+
+- MEV relay might not be started or is not reachable.
+
+#### PingMeasure
+
+- MEV relay might be too far away (geographically) from you.
+- Your network latency might be high. Verify with the `charon test infra` tests.
+
+#### CreateBlock
+
+Same causes as PingMeasure test apply here and additionally:
+
+- MEV relay might be too slow in block production.
+
+#### CreateMultipleBlocks
+
+Same causes as CreateBlock test apply here.
+
+## Infra
+
+#### DiskWriteSpeed
+
+- Read more in our [Deployment Best Practices](../../run/prepare/deployment-best-practices#hardware-specifications).
+
+#### DiskWriteIOPS
+
+- Read more in our [Deployment Best Practices](../../run/prepare/deployment-best-practices#hardware-specifications).
+
+#### DiskReadSpeed
+
+- Read more in our [Deployment Best Practices](../../run/prepare/deployment-best-practices#hardware-specifications).
+
+#### DiskReadIOPS
+
+- Read more in our [Deployment Best Practices](../../run/prepare/deployment-best-practices#hardware-specifications).
+
+#### AvailableMemory
+
+- Your available memory (RAM) is not enough to run Charon. The minimum available memory should be 2GB, the recommended available memory is 4GB. Note that this test is a best estimate, as memory availability can be hard to predict, particularly if the command is run in a virtualised environment (i.e.: a Docker container).
+
+#### TotalMemory
+
+- Your total memory (RAM) may not be enough to run a full validating node. The recommended minimum total memory is 16GB. Specialised, or optimised deployments can use less RAM than the recommended minimum, but may require some monitoring to assert sufficient stability and performance. Read more in our [Deployment Best Practices](../../run/prepare/deployment-best-practices#hardware-specifications)
+
+#### InternetLatency
+
+- Your internet latency to the nearest server is too high. Latency is expected to be at least less than 50ms and at best less than 20ms.
+
+#### InternetDownloadSpeed
+
+- Your internet download speed from the nearest test server is too low. Download speed is expected to be at least above 10Mb/s and at best above 50Mb/s.
+
+#### InternetUploadSpeed
+
+- Your internet upload speed to the nearest test server is too low. Upload speed is expected to be at least above 10Mb/s and at best above 50Mb/s.
diff --git a/docs/run/prepare/deployment-best-practices.md b/docs/run/prepare/deployment-best-practices.md
index be41e6aaa6..fc998c791d 100644
--- a/docs/run/prepare/deployment-best-practices.md
+++ b/docs/run/prepare/deployment-best-practices.md
@@ -17,6 +17,8 @@ The following specifications are recommended for bare metal machines for cluster
- A CPU with 4+ cores, favouring high clock speed over more cores. ( >3.0GHz and higher or a cpubenchmark [single thread](https://www.cpubenchmark.net/singleThread.html) score of >2,500)
- 16GB of RAM
- 2TB+ free SSD disk space (for mainnet)
+- 1000 read/write SSD IOPS
+- 500MB/s read/write SSD speed
- 10mb/s internet bandwidth
### Recommended Specs for extremely large clusters
@@ -24,6 +26,8 @@ The following specifications are recommended for bare metal machines for cluster
- A CPU with 8+ physical cores, with clock speeds >3.5Ghz
- 32GB+ RAM (depending on the EL+CL clients)
- 4TB+ NVMe storage
+- 2000 read/write SSD IOPS
+- 1000MB/s read/write SSD speed
- 25mb/s internet bandwidth
An NVMe storage device is **highly recommended for optimal performance**, offering nearly 10x more random read/writes per second than a standard SSD.
@@ -68,7 +72,7 @@ Cluster sizes that allow for Byzantine Fault Tolerance are recommended as they a
MEV relays are configured at the Consensus Layer or MEV-boost client level. Refer to our [guide](../../run/start/quickstart-builder-api.mdx) to ensure all necessary configuration has been applied to your clients. As with all validators, low latency during proposal opportunities is extremely important. By default, MEV-Boost waits for all configured relays to return a bid, or will timeout if any have not returned a bid within 950ms. This default timeout is generally too slow for a distributed cluster (think of this time as additive to the time it takes the cluster to come to consensus, both of which need to happen within a 2 second window for optimal proposal broadcasting). It is likely better to only list relays that are located geographically near your node, so that once all relays respond (e.g. in < 50ms) your cluster will move forward with the proposal.
-Use Charon's [`test mev` command](../../run/prepare/test-command.md#test-mev-relay) to test a number of your preferred relays, and select the two or three relays with the lowest latency to your node(s), you do not need to have the same relays on each node in a cluster.
+Use Charon's [`test mev` command](../../run/prepare/test-command.mdx#test-mev-relay) to test a number of your preferred relays, and select the two or three relays with the lowest latency to your node(s), you do not need to have the same relays on each node in a cluster.
## Client Diversity
diff --git a/docs/run/prepare/test-command.md b/docs/run/prepare/test-command.md
deleted file mode 100644
index 5ba71bf363..0000000000
--- a/docs/run/prepare/test-command.md
+++ /dev/null
@@ -1,145 +0,0 @@
----
-sidebar_position: 3
-description: Test the performance of a candidate Distributed Validator Cluster setup.
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-# Test a Cluster
-
-:::warning
-The `charon alpha test` command is in an alpha state and is subject to change until it is made available as `charon test` in a future version.
-:::
-
-Charon test commands are designed to help you evaluate the performance and readiness of your candidate cluster. It allows you to test your connection to other Charon peers, the performance of your beacon node(s), and the readiness of your validator client. It prints a performance report to the standard output (which can be omitted with the `--quiet` flag) and a machine-readable TOML format of the report if the `--output-toml` flag is set.
-
-## Test your connection to peers
-
-Run tests towards other Charon peers to evaluate the effectiveness of a potential cluster setup. The command sets up a libp2p node, similarly to what Charon normally does. This test command **has to be running simultaneously with the other peers**. After the node is up it waits for other peers to get their nodes up and running, retrying the connection every 3 seconds. The libp2p node connects to relays (configurable with `p2p-relays` flag) and to other libp2p nodes via TCP. Other peer nodes are discoverable by using their ENRs. Note that for a peer to be successfully discovered, it needs to be connected to the same relay. After completion of the test suite the libp2p node stays alive (duration configurable with `keep-alive` flag) for other peers to continue testing against it. The node can be forcefully stopped as well.
-
-To be able to establish direct connection, you have to ensure:
-
-- Your machine is publicly accessible on the internet or at least a specific port is.
-- You add flag `p2p-tcp-address` (i.e.: `127.0.0.1:9001`) flag and the port specified in it is free and publicly accessible.
-- You add the flag `p2p-external-ip` (i.e.: `8.8.8.8`) and specify your public IP.
-
-If all points are satisfied by you and the other peers, you should be able to establish a direct TCP connection between each other. Note that a relay is still required, as it is used for peer discovery.
-
-At least 1 ENR is required to be supplied to the `--enrs` flag.
-
-### Pre-requisites
-
-- [Create an ENR](../../learn/charon/charon-cli-reference#creating-an-enr-for-charon).
-- Share your ENR with the other peers which will test with you.
-- Obtain the ENRs of the other peers with which you will test.
-
-### Run
-
-```shell
-charon alpha test peers \
- --enrs="enr:-HW4QNDXi9MzdH9Af65g20jDfelAJ0kJhclitkYYgFziYHXhRFF6JyB_CnVnimB7VxKBGBSkHbmy-Tu8BJq8JQkfptiAgmlkgnY0iXNlY3AyNTZrMaEDBVt5pk6x0A2fjth25pjLOEE9DpqCG-BCYyvutY04TZ,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY"
-```
-
-### Run with Docker
-
-```shell
-docker run -v /Users/obol/charon/.charon:/opt/charon/.charon obolnetwork/charon:v1.1.2 alpha test peers \
- --enrs="enr:-HW4QNDXi9MzdH9Af65g20jDfelAJ0kJhclitkYYgFziYHXhRFF6JyB_CnVnimB7VxKBGBSkHbmy-Tu8BJq8JQkfptiAgmlkgnY0iXNlY3AyNTZrMaEDBVt5pk6x0A2fjth25pjLOEE9DpqCG-BCYyvutY04TZs,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY"
-```
-
-## Test your beacon node
-
-Run tests towards beacon node(s), to evaluate its effectiveness for a Distributed Validator cluster. The beacon node is usually the instance getting the most load, especially with high number of validators serviced by the validator client and Charon. If the beacon node is self-hosted (which is highly recommended), the flag `--enable-load-test` can be added.
-
-At least 1 endpoint is required to be supplied to the `--endpoints` flag.
-
-### Pre-requisites
-
-- Running beacon node(s) towards which tests will be executed.
-
-### Run
-
-```shell
-charon alpha test beacon \
- --endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com"
-```
-
-### Run with Docker
-
-```shell
-docker run obolnetwork/charon:v1.1.2 alpha test beacon \
- --endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com"
-```
-
-## Test your validator client
-
-Run tests towards your validator client, to evaluate its effectiveness for a Distributed Validator cluster.
-
-Default endpoint for validator and port is used at `127.0.0.1:3600`. This can be changed by supplying different endpoint to the `--validator-api-address` flag.
-
-### Pre-requisites
-
-- Running validator client towards which tests will be executed.
-
-### Run
-
-```shell
-charon alpha test validator
-```
-
-### Run with Docker
-
-```shell
-docker run obolnetwork/charon:v1.1.2 alpha test validator
-```
-
-## Test MEV relay
-
-Run tests towards MEV relay(s), to evaluate its effectiveness for a Distributed Validator cluster. If a MEV-Boost instance is configured for the validator node, it is of utmost importance that the connection to it is fast. If not, the chance of missing a block proposal increases significantly, because of a slow building of the block from the MEV.
-
-At least 1 endpoint is required to be supplied to the `--endpoints` flag.
-
-### Pre-requisites
-
-- Running MEV relay(s) towards which tests will be executed.
-
-### Run
-
-```shell
-charon alpha test mev \
- --endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,.io,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz"
-```
-
-### Run with Docker
-
-```shell
-docker run -it obolnetwork/charon:v1.1.2 alpha test mev --endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xb0b07cd0abef743db4260b0ed50619cf6ad4d82064cb4fbec9d3ec530f7c5e6793d9f286c4e082c0244ffb9f2658fe88@bloxroute.regulated.blxrbdn.com,https://0xb3ee7afcf27f1f1259ac1787876318c6584ee353097a50ed84f51a1f21a323b3736f271a895c7ce918c038e4265918be@relay.edennetwork.io,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@titanrelay.xyz"
-```
-
-## Test your machine and network performance
-
-Run tests towards your machine and network, to evaluate their effectiveness for a Distributed Validator cluster. While Charon does not require highly performant hardware, the network connectivity of the machine it is running on should be good. In comparison, the beacon node requires not only good connectivity, but also performant storage with high input-output operations per second, enough memory and also good network connectivity. This test aims to address all those requirements and give a good overview of the system.
-
-### Pre-requisites
-
-The storage tests require `fio` to be installed. `fio` is already pre-installed in the charon Docker image if you choose to [run with Docker](#run-with-docker-4). Read more about `fio` [here](https://fio.readthedocs.io/en/latest/fio_doc.html).
-
-### Run
-
-```shell
-charon alpha test performance
-```
-
-### Run with Docker
-
-```shell
-docker run obolnetwork/charon:v1.1.2 alpha test performance
-```
-
-For proper examination of the disk performance, it is advised to run the tests in a Docker volume.
-
-```shell
-docker run -v ${HOME}/charon/test:/opt/charon/test obolnetwork/charon:v1.1.2 alpha test performance \
- --disk-io-test-file-dir=/opt/charon/test
-```
diff --git a/docs/run/prepare/test-command.mdx b/docs/run/prepare/test-command.mdx
new file mode 100644
index 0000000000..3a945b5160
--- /dev/null
+++ b/docs/run/prepare/test-command.mdx
@@ -0,0 +1,662 @@
+---
+sidebar_position: 3
+description: Test the performance of a candidate Distributed Validator Cluster setup.
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Test a Cluster
+
+Charon test commands are designed to help you evaluate the performance and readiness of your candidate cluster. It allows you to test your connection to other Charon peers, the performance of your beacon node(s), the readiness of your validator client, the performance of the MEV relays you will be using and the infrastructure on which you will run the cluster. It prints a performance report to the standard output (which can be omitted with the `--quiet` flag) and a machine-readable JSON format of the report if the `--output-json` flag is set.
+
+
+
+Test all
+ Intended for running tests across all categories. Each flag should have a prefix for its category (i.e.: the flag `--endpoints` from the beacon tests becomes `--beacon-endpoints`). For details about each category refer to their respective sections.
+
+
+
+
+ Regular tests intended for relatively fast run, without putting any major load on any tested system.
+ Based on which stage you are with your cluster creation, some steps are easened.
+
+
+
+ Pre-requisites
+
+ - [ENR private key](../../learn/charon/charon-cli-reference#creating-an-enr-for-charon).
+ - Peers' ENRs, supplied to the `--peers-enrs` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+
+ Example run
+
+ ```shell
+ charon alpha test all \
+ --peers-enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY" \
+ --beacon-endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com" \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz"
+ ```
+
+
+ Pre-requisites
+
+ - Cluster definition file, supplied to the `--peers-definition-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+
+ Example run
+
+ ```shell
+ charon alpha test all \
+ --peers-definition-file="./.charon/cluster-definition.json" \
+ --beacon-endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com" \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz"
+ ```
+
+
+ Pre-requisites
+
+ - Cluster lock file, supplied to the `--peers-lock-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+
+ Example run
+
+ ```shell
+ charon alpha test all \
+ --peers-lock-file="./.charon/cluster-lock.json" \
+ --beacon-endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com" \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz"
+ ```
+
+
+
+
+
+
+ Load tests intended for more time consuming run. Beacon nodes are put under heavy load. MEV relays are required to create real blocks.
+ Based on which stage you are with your cluster creation, some steps are easened.
+
+
+
+ Pre-requisites
+
+ - [ENR private key](../../learn/charon/charon-cli-reference#creating-an-enr-for-charon).
+ - Peers' ENRs, supplied to the `--peers-enrs` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--mev-beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ charon alpha test all \
+ --peers-enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY" \
+ --beacon-endpoints="http://lighthouse:5052/" \
+ --beacon-load-test \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --mev-beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com" \
+ --mev-load-test
+ ```
+
+
+
+ Pre-requisites
+
+ - Cluster definition file, supplied to the `--peers-definition-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--mev-beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ charon alpha test all \
+ --peers-definition-file="./.charon/cluster-definition.json" \
+ --beacon-endpoints="http://lighthouse:5052/" \
+ --beacon-load-test \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --mev-beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com" \
+ --mev-load-test
+ ```
+
+
+
+ Pre-requisites
+
+ - Cluster lock file, supplied to the `--peers-lock-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--mev-beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ charon alpha test all \
+ --peers-lock-file="./.charon/cluster-lock.json" \
+ --beacon-endpoints="http://lighthouse:5052/" \
+ --beacon-load-test \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --mev-beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com" \
+ --mev-load-test
+ ```
+
+
+
+
+
+
+
+
+Test connection to peers
+
+Run tests towards other Charon peers to evaluate the effectiveness of a potential cluster setup. The command sets up a libp2p node, similarly to what Charon normally does. This test command **has to be run simultaneously with the other peers**. After the node is up it waits for other peers to get their nodes up and running, retrying the connection every 3 seconds. The libp2p node connects to relays (configurable with `p2p-relays` flag) and to other libp2p nodes via TCP. Other peer nodes are discoverable by using their ENRs. Note that for a peer to be successfully discovered, it needs to be connected to the same relay. After completion of the test suite the libp2p node stays alive (duration configurable with `keep-alive` flag) for other peers to continue testing against it. The node can be forcefully stopped as well.
+
+To be able to establish direct connection, you have to ensure:
+
+- Your machine is publicly accessible on the internet or at least a specific port is.
+- You add flag `p2p-tcp-address` (i.e.: `127.0.0.1:9001`) flag and the port specified in it is free and publicly accessible.
+- You add the flag `p2p-external-ip` (i.e.: `8.8.8.8`) and specify your public IP.
+
+If all points are satisfied by you and the other peers, you should be able to establish a direct TCP connection between each other. Note that a relay is still required, as it is used for peer discovery.
+
+Based on which stage you are with your cluster creation, some steps are easened.
+
+
+
+ Pre-requisites
+
+ - [ENR private key](../../learn/charon/charon-cli-reference#creating-an-enr-for-charon).
+ - Peers' ENRs, supplied to the `--enrs` flag.
+
+ Example run
+
+ ```shell
+ charon alpha test peers \
+ --enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY"
+ ```
+
+
+ Pre-requisites
+
+ - Cluster definition file, supplied to the `--definition-file` flag.
+
+ Example run
+
+ ```shell
+ charon alpha test peers \
+ --definition-file="./.charon/cluster-definition.json"
+ ```
+
+
+ Pre-requisites
+
+ - Cluster lock file, supplied to the `--lock-file` flag.
+
+ Example run
+
+ ```shell
+ charon alpha test peers \
+ --lock-file="./.charon/cluster-lock.json"
+ ```
+
+
+
+
+Test beacon node
+
+Run tests on beacon node(s), to evaluate their effectiveness for a Distributed Validator cluster. The beacon node is usually the client doing the most work in a validating stack, especially with a high number of validators being serviced by the validator client(s) and Charon(s) that depend on it.
+
+
+
+
+ Regular tests intended for relatively fast run, without putting any major load on any tested system.
+
+ Pre-requisites
+
+ - Running beacon node(s) towards which tests will be executed, supplied to `--endpoints` flag.
+
+ Example run
+
+ ```shell
+ charon alpha test beacon \
+ --endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com"
+ ```
+
+
+
+ Load tests intended for more time consuming run. Beacon nodes are put under heavy load.
+
+ These tests include simulated workloads for an increasing number of validators, and the process takes some time (approximately ~33 minutes). It is normal to observe some warnings during the simulations.
+
+ A file with detailed results about simulations done is saved at the current working directory (configurable by `--simulation-file-dir` flag).
+
+ Pre-requisites
+
+ - Running beacon node(s) towards which tests will be executed, supplied to `--endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+
+ Example run
+
+ ```shell
+ charon alpha test beacon \
+ --endpoints="http://lighthouse:5052/" \
+ --load-test
+ ```
+
+
+
+
+Test validator client
+
+Run tests towards your validator client, to evaluate its effectiveness for a Distributed Validator cluster.
+
+Default endpoint for validator and port is used at `127.0.0.1:3600`. This can be changed by supplying different endpoint to the `--validator-api-address` flag.
+
+Pre-requisites
+
+- Running validator client towards which tests will be executed.
+
+Example run
+
+ ```shell
+ charon alpha test validator
+ ```
+
+Test MEV relay
+
+Run tests towards MEV relays, to evaluate their effectiveness for a Distributed Validator cluster. If MEV-Boost clients are configured for the distributed validator nodes, it is of utmost importance that the relays they connect to are fast and reliable. If not, the chance of missing a block proposal increases significantly. Supplying `--beacon-node-endpoint` and `--load-test` flags allows the test to ask relays for real MEV headers, increasing the accuracy (and duration) of this test.
+
+At least 1 endpoint is required to be supplied to the `--endpoints` flag.
+
+
+
+ Pre-requisites
+
+ - Running MEV relay(s) towards which tests will be executed.
+
+ Example run
+
+ ```shell
+ charon alpha test mev \
+ --endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz"
+ ```
+
+
+
+
+ Pre-requisites
+
+ - Running MEV relay(s) towards which tests will be executed.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ charon alpha test mev \
+ --endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --load-test \
+ --beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com"
+ ```
+
+
+
+
+
+Test machine and network performance
+
+Run tests of your machine and network, to evaluate their effectiveness for a Distributed Validator cluster. Distributed Validators need stable, low-latency, internet, a reasonable amount of RAM, and a highly performant disk drive for storage. This test aims to analyse these requirements to give an overview of the systems suitability.
+
+Pre-requisites
+
+The storage tests require `fio` to be installed on your machine. Read more about `fio` [here](https://fio.readthedocs.io/en/latest/fio_doc.html).
+
+Example run
+
+ ```shell
+ charon alpha test infra
+ ```
+
+
+
+
+Test all
+
+Intended for running tests across all categories. Each flag should have a prefix for its category (i.e.: the flag `--endpoints` from the beacon tests becomes `--beacon-endpoints`). For details about each category refer to their respective sections.
+
+
+
+
+ Regular tests intended for relatively fast run, without putting any major load on any tested system.
+ Based on which stage you are with your cluster creation, some steps are easened.
+
+
+
+ Pre-requisites
+
+ - [ENR private key](../../learn/charon/charon-cli-reference#creating-an-enr-for-charon).
+ - Peers' ENRs, supplied to the `--peers-enrs` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+
+ Example run
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test all \
+ --peers-enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY" \
+ --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \
+ --beacon-endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com" \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --infra-disk-io-test-file-dir="/opt/charon/test"
+ ```
+
+
+
+ Pre-requisites
+
+ - Cluster definition file, supplied to the `--peers-definition-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test all \
+ --peers-definition-file="/opt/charon/test/.charon/cluster-definition.json" \
+ --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \
+ --beacon-endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com" \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --infra-disk-io-test-file-dir="/opt/charon/test"
+ ```
+
+
+ Pre-requisites
+
+ - Cluster lock file, supplied to the `--peers-lock-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test all \
+ --peers-lock-file="/opt/charon/test/.charon/cluster-lock.json" \
+ --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key"
+ --beacon-endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com" \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --infra-disk-io-test-file-dir="/opt/charon/test"
+ ```
+
+
+
+
+
+
+ Load tests intended for more time consuming run. Beacon nodes are put under heavy load. MEV relays are required to create real blocks.
+ Based on which stage you are with your cluster creation, some steps are easened.
+
+
+
+ Pre-requisites
+
+ - [ENR private key](../../learn/charon/charon-cli-reference#creating-an-enr-for-charon).
+ - Peers' ENRs, supplied to the `--peers-enrs` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--mev-beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test all \
+ --peers-enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY" \
+ --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \
+ --beacon-endpoints="http://lighthouse:5052/" \
+ --beacon-simulation-file-dir="/opt/charon/test" \
+ --beacon-load-test \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --mev-beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com" \
+ --mev-load-test \
+ --infra-disk-io-test-file-dir="/opt/charon/test"
+ ```
+
+
+
+ Pre-requisites
+
+ - Cluster definition file, supplied to the `--peers-definition-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--mev-beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test all \
+ --peers-definition-file="/opt/charon/test/.charon/cluster-definition.json" \
+ --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \
+ --beacon-endpoints="http://lighthouse:5052/" \
+ --beacon-simulation-file-dir="/opt/charon/test" \
+ --beacon-load-test \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --mev-beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com" \
+ --mev-load-test \
+ --infra-disk-io-test-file-dir="/opt/charon/test"
+ ```
+
+
+
+ Pre-requisites
+
+ - Cluster lock file, supplied to the `--peers-lock-file` flag.
+ - Running beacon node(s) towards which tests will be executed, supplied to `--beacon-endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+ - Running validator client towards which tests will be executed.
+ - Running MEV relay(s) towards which tests will be executed, supplied to `--mev-endpoints` flag.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--mev-beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test all \
+ --peers-lock-file="/opt/charon/test/.charon/cluster-lock.json" \
+ --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \
+ --beacon-endpoints="http://lighthouse:5052/" \
+ --beacon-simulation-file-dir="/opt/charon/test" \
+ --beacon-load-test \
+ --mev-endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --mev-beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com" \
+ --mev-load-test \
+ --infra-disk-io-test-file-dir="/opt/charon/test"
+ ```
+
+
+
+
+
+
+
+
+Test connection to peers
+
+ Run tests towards other Charon peers to evaluate the effectiveness of a potential cluster setup. The command sets up a libp2p node, similarly to what Charon normally does. This test command **has to be run simultaneously with the other peers**. After the node is up it waits for other peers to get their nodes up and running, retrying the connection every 3 seconds. The libp2p node connects to relays (configurable with `p2p-relays` flag) and to other libp2p nodes via TCP. Other peer nodes are discoverable by using their ENRs. Note that for a peer to be successfully discovered, it needs to be connected to the same relay. After completion of the test suite the libp2p node stays alive (duration configurable with `keep-alive` flag) for other peers to continue testing against it. The node can be forcefully stopped as well.
+
+ To be able to establish direct connection, you have to ensure:
+
+ - Your machine is publicly accessible on the internet or at least a specific port is.
+ - You add flag `p2p-tcp-address` (i.e.: `127.0.0.1:9001`) flag and the port specified in it is free and publicly accessible.
+ - You add the flag `p2p-external-ip` (i.e.: `8.8.8.8`) and specify your public IP.
+
+ If all points are satisfied by you and the other peers, you should be able to establish a direct TCP connection between each other. Note that a relay is still required, as it is used for peer discovery.
+
+ Based on which stage you are with your cluster creation, some steps are easened.
+
+
+ Pre-requisites
+
+ - [ENR private key](../../learn/charon/charon-cli-reference#creating-an-enr-for-charon).
+ - Peers' ENRs, supplied to the `--enrs` flag.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test peers \
+ --enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY" \
+ --private-key-file="/opt/charon/test/.charon/charon-enr-private-key"
+ ```
+
+
+ Pre-requisites
+
+ - Cluster definition file, supplied to the `--definition-file` flag.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test peers \
+ --definition-file="/opt/charon/test/.charon/cluster-definition.json" \
+ --private-key-file="/opt/charon/test/.charon/charon-enr-private-key"
+ ```
+
+
+ Pre-requisites
+
+ - Cluster lock file, supplied to the `--lock-file` flag.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test peers \
+ --lock-file="/opt/charon/test/.charon/cluster-lock.json" \
+ --private-key-file="/opt/charon/test/.charon/charon-enr-private-key"
+ ```
+
+
+
+Test beacon node
+
+Run tests on beacon node(s), to evaluate their effectiveness for a Distributed Validator cluster. The beacon node is usually the client doing the most work in a validating stack, especially with a high number of validators being serviced by the validator client(s) and Charon(s) that depend on it.
+
+
+
+
+ Regular tests intended for relatively fast run, without putting any major load on any tested system.
+
+ Pre-requisites
+
+ - Running beacon node(s) towards which tests will be executed, supplied to `--endpoints` flag.
+
+ Example run
+
+ ```shell
+ docker run obolnetwork/charon:v1.1.2 alpha test beacon \
+ --endpoints="https://ethereum-holesky-beacon-api.publicnode.com,https://ethereum-sepolia-beacon-api.publicnode.com"
+ ```
+
+
+
+ Load tests intended for more time consuming run. Beacon nodes are put under heavy load.
+
+ These tests include simulated workloads for an increasing number of validators, and the process takes some time (approximately ~33 minutes). It is normal to observe some warnings during the simulations.
+
+ A file with detailed results about simulations done is saved at the current working directory (configurable by changing the volume mapped from `$(pwd)` to the desired directory).
+
+ Pre-requisites
+
+ - Running beacon node(s) towards which tests will be executed, supplied to `--endpoints` flag. It is important that the node is expecting to handle huge load and that it is **not** a publicly accessible one, which can block you.
+
+ Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test beacon \
+ --endpoints="http://lighthouse:5052/" \
+ --simulation-file-dir="/opt/charon/test" \
+ --load-test
+ ```
+
+
+
+
+Test validator client
+
+Run tests towards your validator client, to evaluate its effectiveness for a Distributed Validator cluster.
+
+Default endpoint for validator and port is used at `127.0.0.1:3600`. This can be changed by supplying different endpoint to the `--validator-api-address` flag.
+
+Pre-requisites
+
+- Running validator client towards which tests will be executed.
+
+Example run
+
+```shell
+docker run obolnetwork/charon:v1.1.2 alpha test validator
+```
+
+Test MEV relay
+
+Run tests towards MEV relays, to evaluate their effectiveness for a Distributed Validator cluster. If MEV-Boost clients are configured for the distributed validator nodes, it is of utmost importance that the relays they connect to are fast and reliable. If not, the chance of missing a block proposal increases significantly. Supplying `--beacon-node-endpoint` and `--load-test` flags allows the test to ask relays for real MEV headers, increasing the accuracy (and duration) of this test.
+
+At least 1 endpoint is required to be supplied to the `--endpoints` flag.
+
+
+
+ Pre-requisites
+
+ - Running MEV relay(s) towards which tests will be executed.
+
+ Example run
+
+ ```shell
+ docker run obolnetwork/charon:v1.1.2 alpha test mev \
+ --endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz"
+ ```
+
+
+
+
+ Pre-requisites
+
+ - Running MEV relay(s) towards which tests will be executed.
+ - Running beacon node which will be used for fetching data required by the MEV relay for block creation, supplied to `--beacon-node-endpoint`. There is no restrictions on the node and a public one can be used.
+
+ Example run
+
+ ```shell
+ docker run obolnetwork/charon:v1.1.2 alpha test mev \
+ --endpoints="https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live,https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net,https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com,https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net,https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@regional.titanrelay.xyz,https://0x8c4ed5e24fe5c6ae21018437bde147693f68cda427cd1122cf20819c30eda7ed74f72dece09bb313f2a1855595ab677d@global.titanrelay.xyz" \
+ --load-test \
+ --beacon-node-endpoint="https://ethereum-beacon-api.publicnode.com"
+ ```
+
+
+
+
+
+Test machine and network performance
+
+Run tests of your machine and network, to evaluate their effectiveness for a Distributed Validator cluster. Distributed Validators need stable, low-latency, internet, a reasonable amount of RAM, and a highly performant disk drive for storage. This test aims to analyse these requirements to give an overview of the systems suitability.
+
+Pre-requisites
+
+None.
+
+Example run
+
+ ```shell
+ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/charon:v1.1.2 alpha test infra \
+ --disk-io-test-file-dir=/opt/charon/test
+ ```
+
+
+
diff --git a/docs/run/start/quickstart-builder-api.mdx b/docs/run/start/quickstart-builder-api.mdx
index 860431b7cc..9b00a6dada 100644
--- a/docs/run/start/quickstart-builder-api.mdx
+++ b/docs/run/start/quickstart-builder-api.mdx
@@ -43,7 +43,7 @@ charon run --builder-api
### Consensus Clients
-The following flags need to be configured on your chosen consensus client. A Flashbots relay URL is provided for example purposes, you should use the [charon test mev command](../../run/prepare/test-command.md#test-mev-relay) and select the two or three relays with the lowest latency to your node that also conform to your block building preferences. A public list of MEV relays is available [here](https://github.com/eth-educators/ethstaker-guides/blob/main/MEV-relay-list.md#mev-relay-list-for-mainnet).
+The following flags need to be configured on your chosen consensus client. A Flashbots relay URL is provided for example purposes, you should use the [charon test mev command](../../run/prepare/test-command.mdx#test-mev-relay) and select the two or three relays with the lowest latency to your node that also conform to your block building preferences. A public list of MEV relays is available [here](https://github.com/eth-educators/ethstaker-guides/blob/main/MEV-relay-list.md#mev-relay-list-for-mainnet).