UNIT_Server_TEST - ServerRepeat/ServerFixture.ServerConfigSensorPlugin/0 test is flaky on Ubuntu Noble Jenkins CI

[iche033]

Environment

• OS Version: 24.04
• Source or binary build? Noble CI build gz-sim9 branch

Description

• Expected behavior: UNIT_Server_TEST's ServerRepeat/ServerFixture.ServerConfigSensorPlugin/0 test should pass on Ubuntu noble Jenkins CI
• Actual behavior: Test is flaky. It times out in some builds.

Steps to reproduce

Check gz-sim9 ubuntu noble CI build, e.g https://build.osrfoundation.org/job/gz_sim-ci-gz-sim9-noble-amd64/6

Output

37: [ RUN      ] ServerRepeat/ServerFixture.ServerConfigSensorPlugin/0
37: (2024-09-05 06:20:41.547) [debug] [Server_TEST.cc:318] Create server
37: (2024-09-05 06:20:41.553) [info] [Server.cc:145] Loading SDF world file[/home/jenkins/workspace/gz_sim-ci-gz-sim9-noble-amd64/gz-sim/test/worlds/air_pressure.sdf].
37: (2024-09-05 06:20:41.613) [info] [SystemManager.cc:54] Serving entity system service on [/entity/system/add]
37: (2024-09-05 06:20:41.616) [debug] [SystemManager.cc:80] Loaded system [gz::sim::systems::AirPressure] for entity [1]
37: (2024-09-05 06:20:41.617) [debug] [SystemManager.cc:80] Loaded system [gz::sim::systems::SceneBroadcaster] for entity [1]
37: (2024-09-05 06:20:41.617) [info] [LevelManager.cc:641] Loaded level [default]
37: (2024-09-05 06:20:41.618) [debug] [TestSensorSystem.hh:49] Constructing TestSensorSystem
37: (2024-09-05 06:20:41.618) [debug] [TestSensorSystem.hh:69] Configuring TestSensorSystem
37: (2024-09-05 06:20:41.619) [debug] [SystemManager.cc:80] Loaded system [gz::sim::TestSensorSystem] for entity [8]
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:260] Serving world controls on [/world/air_pressure_sensor/control], [/world/air_pressure_sensor/control/state] and [/world/air_pressure_sensor/playback/control]
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:286] Serving GUI information on [/world/air_pressure_sensor/gui/info]
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:289] World [air_pressure_sensor] initialized with [fast] physics profile.
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:296] Serving world SDF generation service on [/world/air_pressure_sensor/generate_world_sdf]
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:334] Serving world names on [/gazebo/worlds]
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:347] Resource path add service on [/gazebo/resource_paths/add].
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:360] Resource path get service on [/gazebo/resource_paths/get].
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:375] Resource path resolve service on [/gazebo/resource_paths/resolve].
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:389] Resource paths published on [/gazebo/resource_paths].
37: (2024-09-05 06:20:41.620) [info] [ServerPrivate.cc:402] Server control service on [/server_control].
37: (2024-09-05 06:20:41.620) [debug] [Server_TEST.cc:326] Run server
37: (2024-09-05 06:20:41.620) [debug] [Server_TEST.cc:332] Request service
37: (2024-09-05 06:20:41.620) [debug] [Server_TEST.cc:339] Requesting /test/service/sensor
37: (2024-09-05 06:20:41.620) [debug] [TestSensorSystem.hh:59] TestSensorSystem service called
37: (2024-09-05 06:20:41.620) [info] [SimulationRunner.cc:743] Found no publishers on /stats, adding root stats topic
37: (2024-09-05 06:20:41.620) [info] [SimulationRunner.cc:777] Found no publishers on /clock, adding root clock topic
37: (2024-09-05 06:20:41.620) [debug] [SimulationRunner.cc:542] Creating PostUpdate worker threads: 3
37: (2024-09-05 06:20:41.620) [debug] [SimulationRunner.cc:553] Creating postupdate worker thread (0)
37: (2024-09-05 06:20:41.620) [debug] [SimulationRunner.cc:553] Creating postupdate worker thread (1)
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:634] Serving scene information on [/world/air_pressure_sensor/scene/info]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:643] Serving graph information on [/world/air_pressure_sensor/scene/graph]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:654] Serving full state on [/world/air_pressure_sensor/state]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:663] Serving full state (async) on [/world/air_pressure_sensor/state_async]
37: (2024-09-05 06:20:41.621) [debug] [AirPressureSensor.cc:126] Air pressure for [air_pressure_model::link::air_pressure_sensor] advertised on [world/air_pressure_sensor/model/air_pressure_model/link/link/sensor/air_pressure_sensor/air_pressure]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:671] Publishing scene information on [/world/air_pressure_sensor/scene/info]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:680] Publishing entity deletions on [/world/air_pressure_sensor/scene/deletion]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:689] Publishing state changes on [/world/air_pressure_sensor/state]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:700] Publishing pose messages on [/world/air_pressure_sensor/pose/info]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:711] Publishing dynamic pose messages on [/world/air_pressure_sensor/dynamic_pose/info]
 37/324 Test  #37: UNIT_Server_TEST ........................................***Timeout 240.10 sec

[ntfshard]

Running this test with taskset -c 1 ./UNIT_Server_TEST to limit physical threads can help to reproduce this problem (on one of my set-ups it shoots 1:10)

I guess it's related to #1240
Due to it can't join the tread which was created to send notifications

I guess I'll try to deep dive in it, but code looks quite convoluted here

[azeey]

Unfortunately, I wasn't able to reproduce it with your instructions @ntfshard. Any help with debugging this would be highly appreciated.

[ntfshard]

For automation I'm using for i in {1..101}; do echo "Iteration: $i"; taskset -c 1 ./UNIT_Server_TEST || break ; done but once I had almost 100 runs in a row w/o deadlock.

But in generally it blocks inside ServerPrivate destructor, when it trying to join threads. Initially it was deadlock on an aux. tread which calling stop on service call. W/O it it hangs on joining worker threads.

I tried to make some fixes but seems logic inside is quite convoluted. Maybe you have some initial design description or something?

[scpeters]

I think I have a fix for the flakiness in #2740