Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(autoware_tenssort_common): validate TensorRT engine version for cached engine #10320

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+47 −1
Open

feat(autoware_tenssort_common): validate TensorRT engine version for cached engine #10320

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions perception/autoware_tensorrt_common/include/autoware/tensorrt_common/tensorrt_common.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ using ProfileDimsPtr = std::unique_ptr<std::vector<ProfileDims>>;
using TensorsVec = std::vector<std::pair<void *, nvinfer1::Dims>>;
using TensorsMap = std::unordered_map<const char *, std::pair<void *, nvinfer1::Dims>>;

constexpr int TRT_MAJOR_IDX = 24;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that these magic values come from an official source (answer from NVIDIA). Could you leave a evidence here so that people know where that came from (and hopefully can track and fix it if/when in the future NVIDIA changes it?)

constexpr int TRT_MINOR_IDX = 25;
constexpr int TRT_PATCH_IDX = 26;

/**
* @class TrtCommon
* @brief TensorRT common library.
Expand Down Expand Up @@ -317,6 +321,13 @@ class TrtCommon // NOLINT
*/
bool buildEngineFromOnnx();

/**
* @brief Validate TensorRT engine.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* @brief Validate TensorRT engine.
* @brief Validate the TensorRT engine.

*
* @return Whether TensorRT version used for building engine is compatible.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* @return Whether TensorRT version used for building engine is compatible.
* @return Whether the TensorRT version used for building engine is compatible.

*/
bool validateEngine();

/**
* @brief Load TensorRT engine.
*
Expand Down
37 changes: 36 additions & 1 deletion perception/autoware_tensorrt_common/src/tensorrt_common.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,15 @@
// Load engine file if it exists
if (fs::exists(trt_config_->engine_path)) {
logger_->log(nvinfer1::ILogger::Severity::kINFO, "Loading engine");
if (!loadEngine()) {
if (!validateEngine()) {
logger_->log(

Check warning on line 119 in perception/autoware_tensorrt_common/src/tensorrt_common.cpp

View check run for this annotation

Codecov / codecov/patch

perception/autoware_tensorrt_common/src/tensorrt_common.cpp#L119 

Added line #L119 was not covered by tests
nvinfer1::ILogger::Severity::kWARNING,
"Engine validation failed for loaded engine from file. Rebuilding engine");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Engine validation failed for loaded engine from file. Rebuilding engine");
"Validation failed for the existing engine file. Rebuilding");

// Rebuild engine if version mismatch occurred
if (!build_engine_with_log()) {
return false;
}
} else if (!loadEngine()) {
return false;
}
logger_->log(nvinfer1::ILogger::Severity::kINFO, "Network validation");
Expand Down Expand Up @@ -543,6 +551,33 @@
return true;
}

bool TrtCommon::validateEngine()

Check warning on line 554 in perception/autoware_tensorrt_common/src/tensorrt_common.cpp

View check run for this annotation

Codecov / codecov/patch

perception/autoware_tensorrt_common/src/tensorrt_common.cpp#L554 

Added line #L554 was not covered by tests
{
#if (NV_TENSORRT_MAJOR * 1000) + (NV_TENSORRT_MINOR * 100) + NV_TENSOR_PATCH >= 8600
std::ifstream engine_file(trt_config_->engine_path);
std::stringstream engine_buffer;
engine_buffer << engine_file.rdbuf();
std::string engine_str = engine_buffer.str();

auto const blob = reinterpret_cast<uint8_t *>(engine_str.data());
logger_->log(

Check warning on line 563 in perception/autoware_tensorrt_common/src/tensorrt_common.cpp

View check run for this annotation

Codecov / codecov/patch

perception/autoware_tensorrt_common/src/tensorrt_common.cpp#L563 

Added line #L563 was not covered by tests
nvinfer1::ILogger::Severity::kINFO, "Plan was created with TensorRT %d.%d.%d",
static_cast<int32_t>(blob[TRT_MAJOR_IDX]), static_cast<int32_t>(blob[TRT_MINOR_IDX]),
static_cast<int32_t>(blob[TRT_PATCH_IDX]));
auto plan_ver = static_cast<int32_t>(blob[TRT_MAJOR_IDX]) * 1000 +
static_cast<int32_t>(blob[TRT_MINOR_IDX]) * 100 +
static_cast<int32_t>(blob[TRT_PATCH_IDX]);

Check warning on line 569 in perception/autoware_tensorrt_common/src/tensorrt_common.cpp

View check run for this annotation

Codecov / codecov/patch

perception/autoware_tensorrt_common/src/tensorrt_common.cpp#L565-L569 

Added lines #L565 - L569 were not covered by tests
if (plan_ver != (NV_TENSORRT_MAJOR * 1000) + (NV_TENSORRT_MINOR * 100) + NV_TENSORRT_PATCH) {
logger_->log(

Check warning on line 571 in perception/autoware_tensorrt_common/src/tensorrt_common.cpp

View check run for this annotation

Codecov / codecov/patch

perception/autoware_tensorrt_common/src/tensorrt_common.cpp#L571 

Added line #L571 was not covered by tests
nvinfer1::ILogger::Severity::kWARNING,
"Plan was created with a different version of TensorRT! Current version: %d.%d.%d",
NV_TENSORRT_MAJOR, NV_TENSORRT_MINOR, NV_TENSORRT_PATCH);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
NV_TENSORRT_MAJOR, NV_TENSORRT_MINOR, NV_TENSORRT_PATCH);
"Plan was created with a different version of TensorRT: %d, whereas current version is %d.%d.%d"
plan_ver, NV_TENSORRT_MAJOR, NV_TENSORRT_MINOR, NV_TENSORRT_PATCH);

return false;

Check warning on line 575 in perception/autoware_tensorrt_common/src/tensorrt_common.cpp

View check run for this annotation

Codecov / codecov/patch

perception/autoware_tensorrt_common/src/tensorrt_common.cpp#L575 

Added line #L575 was not covered by tests
}
#endif
return true;
}

Check warning on line 579 in perception/autoware_tensorrt_common/src/tensorrt_common.cpp

View check run for this annotation

Codecov / codecov/patch

perception/autoware_tensorrt_common/src/tensorrt_common.cpp#L579 

Added line #L579 was not covered by tests

bool TrtCommon::loadEngine()
{
std::ifstream engine_file(trt_config_->engine_path);
Expand Down
Loading