feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

badai-nguyen · 2023-11-13T08:28:27Z

Description

This PR to add a fusion node of pointcloud with image segmentation mask which is output from some kind of 2D detection model such as model to be added by #5553.
The fusion result could be used to filter the unnecessary pointcloud for autonomous driving which is discussed at #5594

One of the idea for node use

Tests performed

Testing filter using front camera semantic segmentation:

Green pointcloud (input): /perception/object_recognition/detection/pointcloud_map_filtered/pointcloud
Pink pointcloud (output): /perception/object_recognition/detection/image_segmentation_filtered/pointcloud (note that it also covers the input green pointcloud)

semantic_segmentation_based_filter_01d060d9-8d25-45e0-b45e-9fd7201ac27b_2023-05-26-11-18-34_p0900_0_.webm

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.
The PR has been properly tested.
The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.
The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

github-actions · 2023-11-21T00:43:49Z

Documentation URL: https://autowarefoundation.github.io/autoware.universe/pr-5562/
Modified URLs:

https://autowarefoundation.github.io/autoware.universe/pr-5562/perception/image_projection_based_fusion/segmentation-pointcloud-fusion/

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

yukkysaito · 2023-11-21T05:08:30Z

Can you share the video and debugging screen with us?

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

badai-nguyen · 2023-11-21T15:53:21Z

Can you share the video and debugging screen with us?

@yukkysaito I have added the debug video in test performed. could you please check it.

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

yukkysaito · 2023-11-22T01:06:00Z

Good 👍
Does this launch by default?

badai-nguyen · 2023-11-22T02:24:19Z

Does this launch by default?

@yukkysaito Not yet.

Since the yolox-s semseg header was trained only on public dataset and need more time for the further improvement so I would to keep this new feature out of default. I hope can merge this feature so we can test autoware for some specific enviroments or weather condition and see how good it works first.

This reverts commit 5dfa160.

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

badai-nguyen · 2023-12-26T08:20:13Z

@yukkysaito As I dicussed with @miursh, since the semseg yolox model is still being improved by @dan-dnn, so I would like to merge the image_pointcloud fusion node first as preparation. The pointcloud filter and perception pipeline change will be considered and updated later when the semseg yolox is ready.
I removed the change of tier4_perception_launch from this PR so could you review this again?

codecov · 2023-12-26T11:08:06Z

Codecov Report

Attention: 67 lines in your changes are missing coverage. Please review.

Comparison is base (1d6e7e8) 15.28% compared to head (ef36ead) 15.28%.
Report is 66 commits behind head on main.

Files	Patch %	Lines
...fusion/src/segmentation_pointcloud_fusion/node.cpp	0.00%	52 Missing ⚠️
.../image_projection_based_fusion/src/fusion_node.cpp	0.00%	12 Missing ⚠️
...ction_based_fusion/src/roi_cluster_fusion/node.cpp	0.00%	1 Missing ⚠️
...sed_fusion/src/roi_detected_object_fusion/node.cpp	0.00%	1 Missing ⚠️
...on_based_fusion/src/roi_pointcloud_fusion/node.cpp	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5562      +/-   ##
==========================================
- Coverage   15.28%   15.28%   -0.01%     
==========================================
  Files        1750     1751       +1     
  Lines      120568   120620      +52     
  Branches    36781    36781              
==========================================
  Hits        18434    18434              
- Misses      81469    81521      +52     
  Partials    20665    20665

Flag	Coverage Δ		*Carryforward flag
differential	`0.00% <0.00%> (?)`
total	`15.29% <ø> (+<0.01%)`	⬆️	Carriedforward from 4c0f353

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

YoshiRi

LGTM in code grammar, but I did not check if it works in real data.

perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp

…tcloud_fusion/node.cpp Co-authored-by: Yoshi Ri <yoshiyoshidetteiu@gmail.com>

miursh · 2024-01-09T16:34:50Z

perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp

+    }
+  }
+
+  pcl::toROSMsg(output_cloud, output_pointcloud_msg);


[imo]
Just some comments; not required to change now.
Since we don't use the PCL method, is it possible not to use PCL?
Converting ROS messages to PCL (and vice versa) incurs a significant computational cost, so we should try to avoid it.

We can access each point in directly ros message like this.

miursh · 2024-01-09T16:45:31Z

perception/image_projection_based_fusion/docs/segmentation-pointcloud-fusion.md

+- The pointclouds are projected onto the semantic/instance segmentation mask image which is the output from 2D segmentation neural network.
+- The pointclouds are belong to segment such as road, sidewalk, building, vegetation, sky ... are considered as less important points for autonomous driving and could be removed.
+
+![segmentation_pointcloud_fusion_image](./images/segmentation_pointcloud_fusion.png)


[Q]
The image file has not been committed?

miursh

LGTM with a few comments

…d_fusion for pointcloud filter (autowarefoundation#5562) * fix: roiCallback for Image Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: pointcloud filter Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: pointcloud filter bugfix Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: segmentation poincloud filter Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: add option in launch file Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: refactor Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * docs: update docs Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: roi_pointcloud_fusion with new template Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: add param of selectable semantic id Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: typo Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: launch file Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * Revert "fix: launch file" This reverts commit 5dfa160. * fix: reverse perception launcher Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * Update perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp Co-authored-by: Yoshi Ri <yoshiyoshidetteiu@gmail.com> * style(pre-commit): autofix --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Yoshi Ri <yoshiyoshidetteiu@gmail.com>

badai-nguyen added 8 commits November 10, 2023 13:30

fix: roiCallback for Image

ef93592

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

fix: pointcloud filter

fcb8169

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

fix: pointcloud filter bugfix

94ba048

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

fix: segmentation poincloud filter

cfa1d63

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

fix: add option in launch file

73e4fd8

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

chore: refactor

949ac3c

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

docs: update docs

0702bf6

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

fix: roi_pointcloud_fusion with new template

55da066

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

github-actions bot added type:documentation Creating or refining documentation. (auto-assigned) component:perception Advanced sensor data processing and environment understanding. (auto-assigned) component:launch Launch files, scripts and initialization tools. (auto-assigned) labels Nov 13, 2023

badai-nguyen mentioned this pull request Nov 13, 2023

feat(tensorrt yolox): inference and publish mask image from yolox model with semantic segmentation header #5553

Merged

7 tasks

badai-nguyen added the run:deploy-docs Mark for deploy-docs action generation. (used-by-ci) label Nov 21, 2023

fix: add param of selectable semantic id

d4ebbb8

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

badai-nguyen marked this pull request as ready for review November 21, 2023 02:36

badai-nguyen requested review from yukkysaito, yukke42, miursh and YoshiRi as code owners November 21, 2023 02:36

chore: typo

2ff0ff3

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

fix: launch file

5dfa160

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

badai-nguyen added 2 commits December 26, 2023 15:55

Revert "fix: launch file"

b999c78

This reverts commit 5dfa160.

fix: reverse perception launcher

6a9331a

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

badai-nguyen requested review from ktro2828 and tzhong518 as code owners December 26, 2023 07:01

Merge branch 'main' into feat/add_image_segmentation_fusion

bf32121

github-actions bot removed the component:launch Launch files, scripts and initialization tools. (auto-assigned) label Dec 26, 2023

style(pre-commit): autofix

4c0f353

badai-nguyen added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Dec 26, 2023

YoshiRi approved these changes Jan 9, 2024

View reviewed changes

perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp Outdated Show resolved Hide resolved

badai-nguyen and others added 2 commits January 9, 2024 23:49

Update perception/image_projection_based_fusion/src/segmentation_poin…

1499f30

…tcloud_fusion/node.cpp Co-authored-by: Yoshi Ri <yoshiyoshidetteiu@gmail.com>

style(pre-commit): autofix

ef36ead

miursh reviewed Jan 9, 2024

View reviewed changes

miursh approved these changes Jan 9, 2024

View reviewed changes

miursh enabled auto-merge (squash) January 9, 2024 16:46

miursh merged commit 39e9acc into autowarefoundation:main Jan 9, 2024

tier4-autoware-public-bot bot mentioned this pull request Jan 10, 2024

chore: sync upstream tier4/autoware_universe#1084

Merged

badai-nguyen mentioned this pull request Jan 10, 2025

Add a new pointcloud filter based on image semantic segmentation #5594

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

badai-nguyen commented Nov 13, 2023 •

edited

Loading

github-actions bot commented Nov 21, 2023

yukkysaito commented Nov 21, 2023

badai-nguyen commented Nov 21, 2023

yukkysaito commented Nov 22, 2023 •

edited

Loading

badai-nguyen commented Nov 22, 2023 •

edited

Loading

badai-nguyen commented Dec 26, 2023

codecov bot commented Dec 26, 2023 •

edited

Loading

YoshiRi left a comment

miursh Jan 9, 2024

miursh Jan 9, 2024

miursh Jan 9, 2024

miursh left a comment

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

Conversation

badai-nguyen commented Nov 13, 2023 • edited Loading

Description

Related links

Tests performed

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

github-actions bot commented Nov 21, 2023

yukkysaito commented Nov 21, 2023

badai-nguyen commented Nov 21, 2023

yukkysaito commented Nov 22, 2023 • edited Loading

badai-nguyen commented Nov 22, 2023 • edited Loading

badai-nguyen commented Dec 26, 2023

codecov bot commented Dec 26, 2023 • edited Loading

Codecov Report

YoshiRi left a comment

Choose a reason for hiding this comment

miursh Jan 9, 2024

Choose a reason for hiding this comment

miursh Jan 9, 2024

Choose a reason for hiding this comment

miursh Jan 9, 2024

Choose a reason for hiding this comment

miursh left a comment

Choose a reason for hiding this comment

badai-nguyen commented Nov 13, 2023 •

edited

Loading

yukkysaito commented Nov 22, 2023 •

edited

Loading

badai-nguyen commented Nov 22, 2023 •

edited

Loading

codecov bot commented Dec 26, 2023 •

edited

Loading