Skip to content

Commit 3aa84b0

Browse files
technolojinpre-commit-ci[bot]miursh
authored
feat(multi_object_tracker): multi object input (#6820)
* refactor: frequently used types, namespace Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * test: multiple inputs Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: check latest measurement time Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: define input manager class Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: interval measures Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: store and sort inputs PoC Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: rename classes Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: object collector Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * impl input manager, no subscribe Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: subscribe and trigger callback Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: subscriber and callbacks are working Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: callback object is fixed, tracker is working Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: get object time argument revise Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: back to periodic publish, analyze input latency and timings Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: enable timing debugger Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: separate object interval function Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: prepare message triggered process Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: trigger tracker by main message arrive Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: clean-up, set namespace Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: object lists with detector index Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: define input channel struct Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: define type for object list Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add channel wise existence probability Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: relocate debugger Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: total existence logic change Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: publishing object debug info, need to fix marker id Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: indexing marker step 1 Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: uuid management Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: association line fix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: print channel names Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: association lines are color-coded Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: association debug marker bugfix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * style(pre-commit): autofix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add option for debug marker Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: skip time statistics update in case of outlier Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: auto-tune latency band Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: pre-defined channels, select on launcher Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add input channels Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove marker idx map Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: to do not miss the latest message of the target stream Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove priority, separate timing optimization Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: time interval bug fix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: refactoring timing state update Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: set parameters optionally Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: revise object time range logic Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: launcher to set input channels Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: exempt spell check 'pointpainting' Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: remove expected interval Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: implement spawn switch Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove debug messages Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: update readme Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: change tentative object topic Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * Revert "fix: remove debug messages" This reverts commit 725a49e. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: reset times when jumps to past Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: check if interval is negative Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: missing config, default value Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove debug messages Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: change no-object message level Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * Update perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: Update copyright to uppercase Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: fix readme links to config files Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: move and rename uuid functions Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: fix debug topic to use node name Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: express meaning of threshold Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: revise decay rate, update function Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: define constants with explanation Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * style(pre-commit): autofix --------- Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com>
1 parent 7877192 commit 3aa84b0

36 files changed

+1616
-205
lines changed

launch/tier4_perception_launch/launch/object_recognition/tracking/tracking.launch.xml

+2
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
<!--multi object tracking-->
1818
<include file="$(find-pkg-share multi_object_tracker)/launch/multi_object_tracker.launch.xml">
1919
<arg name="data_association_matrix_path" value="$(var object_recognition_tracking_multi_object_tracker_data_association_matrix_param_path)"/>
20+
<arg name="input_channels_path" value="$(var object_recognition_tracking_multi_object_tracker_input_channels_param_path)"/>
2021
<arg name="tracker_setting_path" value="$(var object_recognition_tracking_multi_object_tracker_node_param_path)"/>
2122
</include>
2223
</group>
@@ -26,6 +27,7 @@
2627
<!--multi object tracking for near objects-->
2728
<include file="$(find-pkg-share multi_object_tracker)/launch/multi_object_tracker.launch.xml">
2829
<arg name="data_association_matrix_path" value="$(var object_recognition_tracking_multi_object_tracker_data_association_matrix_param_path)"/>
30+
<arg name="input_channels_path" value="$(var object_recognition_tracking_multi_object_tracker_input_channels_param_path)"/>
2931
<arg name="tracker_setting_path" value="$(var object_recognition_tracking_multi_object_tracker_node_param_path)"/>
3032
<arg name="output" value="/perception/object_recognition/tracking/near_objects"/>
3133
</include>

launch/tier4_perception_launch/launch/perception.launch.xml

+1
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
<arg name="object_recognition_detection_object_range_splitter_radar_param_path"/>
2323
<arg name="object_recognition_detection_object_range_splitter_radar_fusion_param_path"/>
2424
<arg name="object_recognition_tracking_multi_object_tracker_data_association_matrix_param_path"/>
25+
<arg name="object_recognition_tracking_multi_object_tracker_input_channels_param_path"/>
2526
<arg name="object_recognition_tracking_multi_object_tracker_node_param_path"/>
2627
<arg name="object_recognition_tracking_radar_object_tracker_data_association_matrix_param_path"/>
2728
<arg name="object_recognition_tracking_radar_object_tracker_tracking_setting_param_path"/>

perception/multi_object_tracker/CMakeLists.txt

+3-1
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,10 @@ include_directories(
2222
# Generate exe file
2323
set(MULTI_OBJECT_TRACKER_SRC
2424
src/multi_object_tracker_core.cpp
25-
src/debugger.cpp
25+
src/debugger/debugger.cpp
26+
src/debugger/debug_object.cpp
2627
src/processor/processor.cpp
28+
src/processor/input_manager.cpp
2729
src/data_association/data_association.cpp
2830
src/data_association/mu_successive_shortest_path/mu_successive_shortest_path_wrapper.cpp
2931
src/tracker/motion_model/motion_model_base.cpp

perception/multi_object_tracker/README.md

+24-21
Original file line numberDiff line numberDiff line change
@@ -46,31 +46,38 @@ Example:
4646

4747
### Input
4848

49-
| Name | Type | Description |
50-
| --------- | ----------------------------------------------------- | ----------- |
51-
| `~/input` | `autoware_auto_perception_msgs::msg::DetectedObjects` | obstacles |
49+
Multiple inputs are pre-defined in the input channel parameters (described below) and the inputs can be configured
50+
51+
| Name | Type | Description |
52+
| ------------------------- | -------------------------- | ---------------------- |
53+
| `selected_input_channels` | `std::vector<std::string>` | array of channel names |
54+
55+
- default value: `selected_input_channels:="['detected_objects']"`, merged DetectedObject message
56+
- multi-input example: `selected_input_channels:="['lidar_centerpoint','camera_lidar_fusion','detection_by_tracker','radar_far']"`
5257

5358
### Output
5459

55-
| Name | Type | Description |
56-
| ---------- | ---------------------------------------------------- | ------------------ |
57-
| `~/output` | `autoware_auto_perception_msgs::msg::TrackedObjects` | modified obstacles |
60+
| Name | Type | Description |
61+
| ---------- | ---------------------------------------------------- | --------------- |
62+
| `~/output` | `autoware_auto_perception_msgs::msg::TrackedObjects` | tracked objects |
5863

5964
## Parameters
6065

61-
<!-- Write parameters of this package.
66+
### Input Channel parameters
6267

63-
Example:
64-
### Node Parameters
68+
Available input channels are defined in [input_channels.param.yaml](config/input_channels.param.yaml).
6569

66-
| Name | Type | Description |
67-
| ---------------------- | ---- | ------------------------------- |
68-
| `output_debug_markers` | bool | whether to output debug markers |
69-
-->
70+
| Name | Type | Description |
71+
| --------------------------------- | ----------------------------------------------------- | ------------------------------------- |
72+
| `<channel>` | | the name of channel |
73+
| `<channel>.topic` | `autoware_auto_perception_msgs::msg::DetectedObjects` | detected objects |
74+
| `<channel>.can_spawn_new_tracker` | `bool` | a switch allow to spawn a new tracker |
75+
| `<channel>.optional.name` | `std::string` | channel name for analysis |
76+
| `<channel>.optional.short_name` | `std::string` | short name for visualization |
7077

7178
### Core Parameters
7279

73-
Node parameters are defined in [multi_object_tracker.param.yaml](config/multi_object_tracker.param.yaml) and association parameters are defined in [data_association.param.yaml](config/data_association.param.yaml).
80+
Node parameters are defined in [multi_object_tracker_node.param.yaml](config/multi_object_tracker_node.param.yaml) and association parameters are defined in [data_association_matrix.param.yaml](config/data_association_matrix.param.yaml).
7481

7582
#### Node parameters
7683

@@ -80,6 +87,9 @@ Node parameters are defined in [multi_object_tracker.param.yaml](config/multi_ob
8087
| `world_frame_id` | double | object kinematics definition frame |
8188
| `enable_delay_compensation` | bool | if True, tracker use timers to schedule publishers and use prediction step to extrapolate object state at desired timestamp |
8289
| `publish_rate` | double | Timer frequency to output with delay compensation |
90+
| `publish_processing_time` | bool | enable to publish debug message of process time information |
91+
| `publish_tentative_objects` | bool | enable to publish tentative tracked objects, which have lower confidence |
92+
| `publish_debug_markers` | bool | enable to publish debug markers, which indicates association of multi-inputs, existence probability of each detection |
8393

8494
#### Association parameters
8595

@@ -93,13 +103,6 @@ Node parameters are defined in [multi_object_tracker.param.yaml](config/multi_ob
93103

94104
## Assumptions / Known limits
95105

96-
<!-- Write assumptions and limitations of your implementation.
97-
98-
Example:
99-
This algorithm assumes obstacles are not moving, so if they rapidly move after the vehicle started to avoid them, it might collide with them.
100-
Also, this algorithm doesn't care about blind spots. In general, since too close obstacles aren't visible due to the sensing performance limit, please take enough margin to obstacles.
101-
-->
102-
103106
See the [model explanations](models.md).
104107

105108
## (Optional) Error detection and handling
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
/**:
2+
ros__parameters:
3+
input_channels:
4+
detected_objects:
5+
topic: "/perception/object_recognition/detection/objects"
6+
can_spawn_new_tracker: true
7+
optional:
8+
name: "detected_objects"
9+
short_name: "all"
10+
# LIDAR - rule-based
11+
lidar_clustering:
12+
topic: "/perception/object_recognition/detection/clustering/objects"
13+
can_spawn_new_tracker: true
14+
optional:
15+
name: "clustering"
16+
short_name: "Lcl"
17+
# LIDAR - DNN
18+
lidar_centerpoint:
19+
topic: "/perception/object_recognition/detection/centerpoint/objects"
20+
can_spawn_new_tracker: true
21+
optional:
22+
name: "centerpoint"
23+
short_name: "Lcp"
24+
lidar_centerpoint_validated:
25+
topic: "/perception/object_recognition/detection/centerpoint/validation/objects"
26+
can_spawn_new_tracker: true
27+
optional:
28+
name: "centerpoint"
29+
short_name: "Lcp"
30+
lidar_apollo:
31+
topic: "/perception/object_recognition/detection/apollo/objects"
32+
can_spawn_new_tracker: true
33+
optional:
34+
name: "apollo"
35+
short_name: "Lap"
36+
lidar_apollo_validated:
37+
topic: "/perception/object_recognition/detection/apollo/validation/objects"
38+
can_spawn_new_tracker: true
39+
optional:
40+
name: "apollo"
41+
short_name: "Lap"
42+
# LIDAR-CAMERA - DNN
43+
# cspell:ignore lidar_pointpainitng pointpainting
44+
lidar_pointpainitng:
45+
topic: "/perception/object_recognition/detection/pointpainting/objects"
46+
can_spawn_new_tracker: true
47+
optional:
48+
name: "pointpainting"
49+
short_name: "Lpp"
50+
lidar_pointpainting_validated:
51+
topic: "/perception/object_recognition/detection/pointpainting/validation/objects"
52+
can_spawn_new_tracker: true
53+
optional:
54+
name: "pointpainting"
55+
short_name: "Lpp"
56+
# CAMERA-LIDAR
57+
camera_lidar_fusion:
58+
topic: "/perception/object_recognition/detection/clustering/camera_lidar_fusion/objects"
59+
can_spawn_new_tracker: true
60+
optional:
61+
name: "camera_lidar_fusion"
62+
short_name: "CLf"
63+
# CAMERA-LIDAR+TRACKER
64+
detection_by_tracker:
65+
topic: "/perception/object_recognition/detection/detection_by_tracker/objects"
66+
can_spawn_new_tracker: false
67+
optional:
68+
name: "detection_by_tracker"
69+
short_name: "dbT"
70+
# RADAR
71+
radar:
72+
topic: "/sensing/radar/detected_objects"
73+
can_spawn_new_tracker: true
74+
optional:
75+
name: "radar"
76+
short_name: "R"
77+
radar_far:
78+
topic: "/perception/object_recognition/detection/radar/far_objects"
79+
can_spawn_new_tracker: true
80+
optional:
81+
name: "radar_far"
82+
short_name: "Rf"

perception/multi_object_tracker/config/multi_object_tracker_node.param.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -17,5 +17,6 @@
1717
# debug parameters
1818
publish_processing_time: false
1919
publish_tentative_objects: false
20+
publish_debug_markers: false
2021
diagnostics_warn_delay: 0.5 # [sec]
2122
diagnostics_error_delay: 1.0 # [sec]
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
// Copyright 2024 TIER IV, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
#ifndef MULTI_OBJECT_TRACKER__DEBUGGER__DEBUG_OBJECT_HPP_
16+
#define MULTI_OBJECT_TRACKER__DEBUGGER__DEBUG_OBJECT_HPP_
17+
18+
#include "multi_object_tracker/tracker/model/tracker_base.hpp"
19+
20+
#include <rclcpp/rclcpp.hpp>
21+
#include <tier4_autoware_utils/ros/uuid_helper.hpp>
22+
23+
#include "unique_identifier_msgs/msg/uuid.hpp"
24+
#include <autoware_auto_perception_msgs/msg/detected_objects.hpp>
25+
#include <autoware_auto_perception_msgs/msg/tracked_objects.hpp>
26+
#include <geometry_msgs/msg/point.hpp>
27+
#include <visualization_msgs/msg/marker_array.hpp>
28+
29+
#include <boost/functional/hash.hpp>
30+
#include <boost/uuid/uuid.hpp>
31+
#include <boost/uuid/uuid_generators.hpp>
32+
33+
#include <list>
34+
#include <memory>
35+
#include <string>
36+
#include <unordered_map>
37+
#include <unordered_set>
38+
#include <vector>
39+
40+
struct ObjectData
41+
{
42+
rclcpp::Time time;
43+
44+
// object uuid
45+
boost::uuids::uuid uuid;
46+
int uuid_int;
47+
std::string uuid_str;
48+
49+
// association link, pair of coordinates
50+
// tracker to detection
51+
geometry_msgs::msg::Point tracker_point;
52+
geometry_msgs::msg::Point detection_point;
53+
bool is_associated{false};
54+
55+
// existence probabilities
56+
std::vector<float> existence_vector;
57+
58+
// detection channel id
59+
uint channel_id;
60+
};
61+
62+
class TrackerObjectDebugger
63+
{
64+
public:
65+
explicit TrackerObjectDebugger(std::string frame_id);
66+
67+
private:
68+
bool is_initialized_{false};
69+
std::string frame_id_;
70+
visualization_msgs::msg::MarkerArray markers_;
71+
std::unordered_set<int> current_ids_;
72+
std::unordered_set<int> previous_ids_;
73+
rclcpp::Time message_time_;
74+
75+
std::vector<ObjectData> object_data_list_;
76+
std::list<int32_t> unused_marker_ids_;
77+
int32_t marker_id_ = 0;
78+
std::vector<std::vector<ObjectData>> object_data_groups_;
79+
80+
std::vector<std::string> channel_names_;
81+
82+
public:
83+
void setChannelNames(const std::vector<std::string> & channel_names)
84+
{
85+
channel_names_ = channel_names;
86+
}
87+
void collect(
88+
const rclcpp::Time & message_time, const std::list<std::shared_ptr<Tracker>> & list_tracker,
89+
const uint & channel_index,
90+
const autoware_auto_perception_msgs::msg::DetectedObjects & detected_objects,
91+
const std::unordered_map<int, int> & direct_assignment,
92+
const std::unordered_map<int, int> & reverse_assignment);
93+
94+
void reset();
95+
void draw(
96+
const std::vector<std::vector<ObjectData>> object_data_groups,
97+
visualization_msgs::msg::MarkerArray & marker_array) const;
98+
void process();
99+
void getMessage(visualization_msgs::msg::MarkerArray & marker_array) const;
100+
};
101+
102+
#endif // MULTI_OBJECT_TRACKER__DEBUGGER__DEBUG_OBJECT_HPP_

0 commit comments

Comments
 (0)