Evaluate prediction accuracy with nuScenes

[xmfcx]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I've agreed with the maintainers that I can plan this task.

Description

Autoware currently uses rule-based prediction modules with simple linear models. We aim to assess the performance of these existing prediction modules using the nuScenes dataset's Prediction Challenge to better understand their accuracy and reliability before integrating learning-based prediction algorithms.

Similar previous work:

• Evaluate performance of object detection pipeline #565

Purpose

The purpose of this evaluation is to benchmark Autoware's current prediction capabilities against the nuScenes dataset. This will provide a clear baseline of performance and highlight areas for improvement as we transition to more sophisticated learning-based prediction algorithms.

Possible approaches

• Utilize the nuscenes prediction dataset and follow the guidelines provided in the nuscenes prediction tutorial.
• Analyze the performance of existing linear prediction models in Autoware against the benchmark data.
• Document the findings and identify potential areas where machine learning models could offer significant improvements.

Definition of done

• Download the nuScenes prediction dataset from the nuScenes download page.
• Setup the environment and necessary dependencies to run the prediction tutorial notebook.
• Create a new repository for an adapter tool to facilitate the integration of Autoware with the nuScenes dataset.
• Run the existing Autoware prediction models on the nuScenes dataset using the adapter tool.
• Collect and document the performance metrics from the evaluations.
• Prepare a step-by-step document detailing the process to reproduce the evaluations, including any configurations and tools used.
• Prepare a final report summarizing the findings and outlining recommendations for the integration of learning-based models.

[sezan92]

I have started working on it. I will try to update in this issue regularly. Right now I am trying to understand the notebook.
my next plan is to generate some sample predictions and run the evaluation pipeline. then i would like to use autoware for prediction

[sezan92]

Update 2024/04/21

• I have kind of read the overall idea to nuscense datase and the Evaluate performance of object detection pipeline #565

[meliketanrikulu]

Hello @sezan92 . Thanks for your efforts. Is there any update or open PR related this issue ?

[sezan92]

@meliketanrikulu yes, I am continuously working on this. I am in the middle of setting up the nuscense dev kit in my local machine , then I plan to use Autoware perception stack . My plan is to feed the data from nuscenes to the perception node (sorry if my naming is wrong) and get the prediction. then i will evaluate using the prediction. but I am progressing slowly as I am also busy in my full time job. is that fine?

[meliketanrikulu]

@meliketanrikulu yes, I am continuously working on this. I am in the middle of setting up the nuscense dev kit in my local machine , then I plan to use Autoware perception stack . My plan is to feed the data from nuscenes to the perception node (sorry if my naming is wrong) and get the prediction. then i will evaluate using the prediction. but I am progressing slowly as I am also busy in my full time job. is that fine?

@sezan92 Thanks for your effort. No problem . I just asked to check the assignee part. I wish you good work :)

[sezan92]

Update 2024/06/06

• sorry for late update. I have successfully run the prediction notebook provided by nuscense. I am going through a similar PR (perception_benchmark_tool): add package perception_benchmark_tool#1 to get idea how to create a benchmarking tool for this

TODO

• start working on a basic benchmarking tool using autoware.

[mitsudome-r]

@sezan92
Are you working on this at the moment?
If you cannot work on this, I would like to assign someone else.

[sezan92]

@mitsudome-r i am working on this but the progress is slow, if it is urgent for autoware, then you are free to re-assign.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[sezan92]

dummy, still working on it .

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.