Update README and Plotting

README.md

@@ -126,6 +126,7 @@ Quadrupedal Walking__ follow the following instruction:
     ```
   > **Note**: If you want to run the simulation and the controller on two different machines, like in the paper, the installation has to be done on both.
 
+
 **2. _Simulation Machine:_ Start the simulation**
 * On the machine that should run the simulation run the container and start the simulation
     ```bash
@@ -136,36 +137,43 @@ Quadrupedal Walking__ follow the following instruction:
 
 * Visualization will be rendered (if configured) and can be seen in the browser (URL will be shown in the terminal, often [localhost:7000](http://localhost:7000/)).
 
-**3. _Target computer:_ Setup experiments**
-* The experiments are run from the script [ws/src/solver_experiments/solver_experiments/solver_experiments.py](ws/src/solver_experiments/solver_experiments/solver_experiments.py),
-* The experimemts can be configured the following way:
-  - [Line 609](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L609) sets which MPC solvers are included in the tests
-  - [Line 619](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L619) sets which condensed levels are tested for MPC solvers with a sparse interface
-  - [Line 616](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L616) sets which modes are included into the tests for the HPIPM MPC solver
-  - [Line 617](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L617) sets which WBC solvers are included in the tests
-  - [Line 618](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L618) sets which WBC scenes/formulations are included in the tests
-  - [Line 641](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L641) sets how often an experiment is repeated until it is considered as 'failed'
-
-**4. _Target computer:_ Run the experiments**
-* Make sure the [ws/src/solver_experiments/results](ws/src/solver_experiments/results) folder is empty and exists.
-* Start the container and run the experiment script. The script will do all experiments automatically 
+**3. _Target computer:_ Run the experiments**
+* Run the automatic script which runs the set of experiments from the simulation automatically for both prediction horizons:
     ```bash
-    sr # This sources the ros environment for the simulation
-    export ROS_DOMAIN_ID=123 # Make sure both machines run on the same ROS 2 Domain ID
-    ros2 run solver_experiments solver_experiments      
-    ```
+    # Outside of docker run:
+    ./automatic_solver_experiments.sh <pc_name> <test_preset> 123 main 20 N10_prediction_horizon 10
+    
+    # <pc_name> specfies the name of the target computer which is used to label the experiments (in the paper this was out of [arm_orin, desktop_pc, latte_panda])
 
-    > **Note:** The test script will run all experiments (i.e. combinations of MPC solvers, WBC solvers, condensing levels, etc.). For each experiment it first runs the standing and then the trot scenario.
+    # <test_preset> specifies which test preset should be run for the paper this is both [mpc, wbc]
+
+    # 123 is the ROS_DOMAIN_ID on which the simulation runs
+
+    # main 20 is the branch and the corresponding prediction horizon on that branch
+    # N10_prediction_horizon 10 is the branch and the corresponding prediction horizon on that branch
+    ```
+    > **Note:** The test script will spin a docker container, build the project and run all experiments (i.e. combinations of MPC solvers, WBC solvers, condensing levels, etc.). For each experiment it first runs the standing and then the trot scenario.
     If a single experiment fails (*the terminal might show errors etc.*), the script will after a while automatically restart the experiment and mark the old one as failed, it will repeat this as often as specified above.
     The full script might take depending on your system at least one hour. Please wait until the script tells you that it is finished.
-
-    > **Note:** If the test script is stopped manually or crashes a subprocess (leg driver or controller) might still be running. In that case it is important to manually kill that processes or just **restart the docker container**.
-* Move the [ws/src/solver_experiments/results](ws/src/solver_experiments/results) folder to a new location (another empty folder). The folder should be also renamed with the following pattern, such that it can be later imported into the plotting script: `<target_platform>-N<precition_horizon>-<experiment_name_tag>`
+
+* The automatic script saves the experiment sets to folders under [ws/src/solver_experiments/](ws/src/solver_experiments/)The folders are named automatically in the following way, such that it can be later imported into the plotting script: `<target_platform>-N<precition_horizon>-<experiment_name_tag>`
   * `<target_platform>` indicates on which target computer the controller was executed, e.g. `arm_orin`.
   * `<prediciton_horizon>` indicates whith which MPC prediction horizon the experiments were executed, e.g. `10`.
-  * `<experiment_name_tag>` is a free field to put some information about the experiments, e.g. if this field contains `wbc` the respective experiments in that folder will later be excluded from any MPC related plot and vice versa.
-* **Changing the prediction horizon**:
-  * The MPC prediction horizon is a compile constant and has to be changed manually in several places. The easiest is to checkout the branch `N10_prediction_horizon` and recompile the software stack. (The docker container doesn't need a rebuild.)
+  * `<experiment_name_tag>` contains the test preset and a timestamp
+
+* **Optional** If you with to run your own set of experiments you can change the underlying python script which is runned by the [automatic_solver_experiments.sh](automatic_solver_experiments.sh):
+    * The experiments are run from the script [ws/src/solver_experiments/solver_experiments/solver_experiments.py](ws/src/solver_experiments/solver_experiments/solver_experiments.py)
+    * The experimemts can be configured the following way:
+        - [Line 626](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L609) sets which MPC solvers are included in the tests
+        - [Line 637](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L637) sets which condensed levels are tested for MPC solvers with a sparse interface
+        - [Line 633](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L633) sets which modes are included into the tests for the HPIPM MPC solver
+        - [Line 638](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L638) sets which WBC solvers are included in the tests
+        - [Line 639](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L639) sets which WBC scenes/formulations are included in the tests
+        - [Line 645](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L645) sets how often an experiment is repeated until it is considered as 'failed'
+        - [Line 634](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L634) sets which OSQP backends are included in the tests
+        - [Line 640](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L640) sets which solver tolerances are included in the tests
+        - [Lines 649ff](ws/src/solver_experiments/solver_experiments/solver_experiments.py#L649) define the test presets as used by the automatic experiment script
+
 
 **5. Generate the plots**
 * Assuming all different experiment result folders (named in the scheme defined above) containing the single experiments have been moved to one empty root folder, the plots can be generated.