fix(image_projection_based_fusion): add run length decoding for segmentation_pointcloud_fusion #7909

badai-nguyen · 2024-07-09T04:58:32Z

Description

Since the segmentation mask will be encoded before publish as PR: fix(tensorrt_yolox): add run length encoding for sematic segmentation mask #7905, so this PR is to add corresponding decoder for that.
This PR need merge after fix(tensorrt_yolox): add run length encoding for sematic segmentation mask #7905 merged.
Related issue: Add a new pointcloud filter based on image semantic segmentation #5594

How was this PR tested?

Run ros2 launch autoware_tensorrt_yolox yolox.launch.xml with is_publish_color_mask: true
Run ros2 launch autoware_image_projection_based_fusion segmentation_pointcloud_fusion.launch.xml with is_publish_debug_mask: true
Play rosbag (input_bag of an internal dataset) which contains /sensing/camera/camera0/image_rect_color/compressed and Pointcoud2 input topic for /segmentation_pointcloud_fusion node
Confirmed the debug images by rqt that have the same segmentation result as below (Note: the colors are different because of different color map was used)

Notes for reviewers

None.

Interface changes

None.

Effects on system behavior

None.

github-actions · 2024-07-09T04:58:50Z

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

You've checked our contribution guidelines.
Your PR follows our pull request guidelines.
All required CI checks pass before marking the PR ready for review.

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

…lter

kminoda · 2024-08-08T13:30:30Z

perception/autoware_image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp

+  Eigen::Matrix4d projection;
+  projection << camera_info.p.at(0), camera_info.p.at(1), camera_info.p.at(2), camera_info.p.at(3),
+    camera_info.p.at(4), camera_info.p.at(5), camera_info.p.at(6), camera_info.p.at(7),
+    camera_info.p.at(8), camera_info.p.at(9), camera_info.p.at(10), camera_info.p.at(11), 0.0, 0.0,
+    0.0, 1.0;


Is this used somewhere?

@kminoda thank you for you comment. I my mistake when rebased branch. I fixed t b8abe57

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

…lter

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

…lter

codecov · 2024-08-19T02:05:55Z

Codecov Report

Attention: Patch coverage is 0% with 8 lines in your changes missing coverage. Please review.

Project coverage is 23.88%. Comparing base (1d87bdb) to head (84c90cf).
Report is 32 commits behind head on main.

Files	Patch %	Lines
...fusion/src/segmentation_pointcloud_fusion/node.cpp	0.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7909      +/-   ##
==========================================
- Coverage   23.90%   23.88%   -0.02%     
==========================================
  Files        1381     1382       +1     
  Lines      101921   101907      -14     
  Branches    38808    38805       -3     
==========================================
- Hits        24362    24342      -20     
+ Misses      75115    75065      -50     
- Partials     2444     2500      +56

Flag	Coverage Δ		*Carryforward flag
differential	`2.91% <0.00%> (?)`
total	`23.89% <ø> (-0.01%)`	⬇️	Carriedforward from cc8e1b8

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…lter

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

…lter

kminoda

I added one small comment 🙏

Also, please update How was this PR tested? section.

perception/autoware_image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp

…tion_pointcloud_fusion/node.cpp Co-authored-by: kminoda <44218668+kminoda@users.noreply.github.com>

YoshiRi

LGTM

This reverts commit 30fa9ae.

badai-nguyen · 2024-08-20T06:54:16Z

I added one small comment 🙏

Also, please update How was this PR tested? section.

@kminoda thanks for your comment.
I updated How was PR tested. Could you check again.
By the way, I also open PR for unit test to confirm the output after decoder the same with the input image here :#8414

…lter

badai-nguyen · 2024-08-20T23:29:02Z

@kminoda Do you have other comments? Otherwise, I will merge this PR.

…ntation_pointcloud_fusion (autowarefoundation#7909) * fix: add rle decompress Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix: use rld in tensorrt utils Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: rebase error Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: dependency Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: skip publish debug mask Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * Update perception/autoware_image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp Co-authored-by: kminoda <44218668+kminoda@users.noreply.github.com> * style(pre-commit): autofix * Revert "fix: skip publish debug mask" This reverts commit 30fa9ae. --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: kminoda <44218668+kminoda@users.noreply.github.com>

….29.0 (#1526) * feat(tensorrt yolox): inference and publish mask image from yolox model with semantic segmentation header (autowarefoundation#5553) * add segmentation model Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> fix: add multitask for segment Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: publish mask Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * feat: publish colorized mask Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: resize yolox mask Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: add memory allocate operations Signed-off-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> * refactor: remove underscore for a local variable Signed-off-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> * chore: add condition to check the number of subscriber for newly added topics Signed-off-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> * chore: pre-commit Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: add roi overlapping segment Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: roi overlay segment Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: refactor Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * docs: update readme Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: update model name Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: add utils into tensorrt_yolox Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: launch file Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: remove unnecessary depend Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: fix yaml file Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: remove duplicated param in launch Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: semantic class Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * docs: update readme Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: update default param Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: add processing time topic Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: typo Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * chore: fix cspell error Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: yolox default param Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: rename debug topics Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: rename debug topics Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * docs: update model description Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix: launch Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * refactor: unpublish mask for single task Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * Update perception/tensorrt_yolox/src/tensorrt_yolox.cpp Co-authored-by: Manato Hirabayashi <3022416+manato@users.noreply.github.com> * Update perception/tensorrt_yolox/src/tensorrt_yolox.cpp Co-authored-by: Manato Hirabayashi <3022416+manato@users.noreply.github.com> * Update perception/tensorrt_yolox/src/tensorrt_yolox.cpp Co-authored-by: Manato Hirabayashi <3022416+manato@users.noreply.github.com> * style(pre-commit): autofix * docs: update reamde Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix: skip mask size as yolox output Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Signed-off-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> Co-authored-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Manato Hirabayashi <3022416+manato@users.noreply.github.com> * fix(tensorrt_yolox): add run length encoding for sematic segmentation mask (autowarefoundation#7905) * fix: add rle compress Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: rle compress Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: move rle into utils Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: pre-commit Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * Update perception/autoware_tensorrt_yolox/src/utils.cpp Co-authored-by: Yukihiro Saito <yukky.saito@gmail.com> * fix: remove unused variable Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * Update perception/autoware_tensorrt_yolox/src/utils.cpp Co-authored-by: Manato Hirabayashi <3022416+manato@users.noreply.github.com> * style(pre-commit): autofix * feat: add unit test for utils Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix: unit test Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: change to explicit index Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix: cuda cmake Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: separate unit test into different PR Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Co-authored-by: Yukihiro Saito <yukky.saito@gmail.com> Co-authored-by: Manato Hirabayashi <3022416+manato@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor(tensorrt_yolox): move utils into perception_utils (autowarefoundation#8435) * chore(tensorrt_yolo): refactor utils Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix: tensorrt_yolox Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix(perception_utils): install library after build (autowarefoundation#8501) Signed-off-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> * fix(image_projection_based_fusion): resize sematic segmentation mask as input image size (autowarefoundation#7635) Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix(image_projection_based_fusion): segmentation pointcloud fusion param update (autowarefoundation#7858) * fix(image_projection_based_fusion): add run length decoding for segmentation_pointcloud_fusion (autowarefoundation#7909) * fix: add rle decompress Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix: use rld in tensorrt utils Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: rebase error Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: dependency Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: skip publish debug mask Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * Update perception/autoware_image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp Co-authored-by: kminoda <44218668+kminoda@users.noreply.github.com> * style(pre-commit): autofix * Revert "fix: skip publish debug mask" This reverts commit 30fa9ae. --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: kminoda <44218668+kminoda@users.noreply.github.com> * feat(tier4_perception_launch): add image segmentation based pointcloud filter (autowarefoundation#7225) * feat(tier4_perception_launch): add image segmentation based pointcloud filter Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: typo Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * fix: detection launch Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * chore: add maintainer Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * Revert "chore: add maintainer" This reverts commit 5adfef6. --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> * style(pre-commit): autofix * fix(autoware_image_projection_based_fusion): resolve issue with segmentation pointcloud fusion node failing with multiple mask inputs (autowarefoundation#8769) --------- Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp> Signed-off-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> Co-authored-by: Manato HIRABAYASHI <manato.hirabayashi@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Manato Hirabayashi <3022416+manato@users.noreply.github.com> Co-authored-by: Yukihiro Saito <yukky.saito@gmail.com> Co-authored-by: kminoda <44218668+kminoda@users.noreply.github.com> Co-authored-by: Yi-Hsiang Fang (Vivid) <146902905+vividf@users.noreply.github.com>

github-actions bot added the component:perception Advanced sensor data processing and environment understanding. (auto-assigned) label Jul 9, 2024

badai-nguyen mentioned this pull request Jul 10, 2024

fix(tensorrt_yolox): add run length encoding for sematic segmentation mask #7905

Merged

badai-nguyen changed the title ~~fix(image_projection_based_fusion): add rle decompress for segmentation_pointcloud_fusion~~ fix(image_projection_based_fusion): add run length decoding for segmentation_pointcloud_fusion Jul 10, 2024

badai-nguyen and others added 3 commits July 26, 2024 16:39

fix: add rle decompress

4d1ce1c

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

style(pre-commit): autofix

12f0b2b

fix: use rld in tensorrt utils

9ea3cdf

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

badai-nguyen force-pushed the fix/segmenation_pointcloud_filter_add_rle_filter branch from 9dcc246 to 9ea3cdf Compare July 26, 2024 07:55

badai-nguyen added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Jul 26, 2024

Merge branch 'main' into fix/segmenation_pointcloud_filter_add_rle_fi…

725bd85

…lter

badai-nguyen marked this pull request as ready for review August 8, 2024 13:26

badai-nguyen requested review from kminoda, ktro2828, miursh, tzhong518, YoshiRi and yukkysaito as code owners August 8, 2024 13:26

kminoda reviewed Aug 8, 2024

View reviewed changes

badai-nguyen and others added 2 commits August 9, 2024 09:46

fix: rebase error

b8abe57

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

Merge branch 'main' into fix/segmenation_pointcloud_filter_add_rle_fi…

8d23163

…lter

badai-nguyen marked this pull request as draft August 10, 2024 07:00

Shin-kyoto assigned badai-nguyen Aug 13, 2024

badai-nguyen and others added 3 commits August 15, 2024 17:16

Merge branch 'main' into fix/segmenation_pointcloud_filter_add_rle_fi…

26a99fa

…lter

fix: dependency

a812eb2

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

Merge branch 'main' into fix/segmenation_pointcloud_filter_add_rle_fi…

4eded7e

…lter

badai-nguyen and others added 3 commits August 19, 2024 13:56

Merge branch 'main' into fix/segmenation_pointcloud_filter_add_rle_fi…

7dc5af1

…lter

fix: skip publish debug mask

30fa9ae

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

Merge branch 'main' into fix/segmenation_pointcloud_filter_add_rle_fi…

99e1fe1

…lter

badai-nguyen marked this pull request as ready for review August 19, 2024 08:14

kminoda reviewed Aug 20, 2024

View reviewed changes

perception/autoware_image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp Outdated Show resolved Hide resolved

YoshiRi mentioned this pull request Aug 20, 2024

feat(detected_object_validation): copy parameter files update from universe autowarefoundation/autoware_launch#1126

Merged

7 tasks

badai-nguyen and others added 2 commits August 20, 2024 15:07

Update perception/autoware_image_projection_based_fusion/src/segmenta…

75f0eb0

…tion_pointcloud_fusion/node.cpp Co-authored-by: kminoda <44218668+kminoda@users.noreply.github.com>

style(pre-commit): autofix

fd6ce2e

YoshiRi approved these changes Aug 20, 2024

View reviewed changes

Revert "fix: skip publish debug mask"

cc8e1b8

This reverts commit 30fa9ae.

Merge branch 'main' into fix/segmenation_pointcloud_filter_add_rle_fi…

84c90cf

…lter

badai-nguyen merged commit 556816d into autowarefoundation:main Aug 21, 2024
31 of 32 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(image_projection_based_fusion): add run length decoding for segmentation_pointcloud_fusion #7909

fix(image_projection_based_fusion): add run length decoding for segmentation_pointcloud_fusion #7909

badai-nguyen commented Jul 9, 2024 •

edited

Loading

github-actions bot commented Jul 9, 2024 •

edited

Loading

kminoda Aug 8, 2024

badai-nguyen Aug 9, 2024

codecov bot commented Aug 19, 2024 •

edited

Loading

kminoda left a comment

YoshiRi left a comment

badai-nguyen commented Aug 20, 2024

badai-nguyen commented Aug 20, 2024

fix(image_projection_based_fusion): add run length decoding for segmentation_pointcloud_fusion #7909

fix(image_projection_based_fusion): add run length decoding for segmentation_pointcloud_fusion #7909

Conversation

badai-nguyen commented Jul 9, 2024 • edited Loading

Description

Related links

How was this PR tested?

Notes for reviewers

Interface changes

Effects on system behavior

github-actions bot commented Jul 9, 2024 • edited Loading

kminoda Aug 8, 2024

Choose a reason for hiding this comment

badai-nguyen Aug 9, 2024

Choose a reason for hiding this comment

codecov bot commented Aug 19, 2024 • edited Loading

Codecov Report

kminoda left a comment

Choose a reason for hiding this comment

YoshiRi left a comment

Choose a reason for hiding this comment

badai-nguyen commented Aug 20, 2024

badai-nguyen commented Aug 20, 2024

badai-nguyen commented Jul 9, 2024 •

edited

Loading

github-actions bot commented Jul 9, 2024 •

edited

Loading

codecov bot commented Aug 19, 2024 •

edited

Loading