Triton Model Navigator v0.6.0

• new: Zero-copy runners for Torch, ONNX and TensorRT - omit H2D and D2H memory copy between runners execution

• new: nav.pacakge.profile API method to profile generated models on provided dataloader

• change: ProfilerConfig replaced with OptimizationProfile:

  • new: OptimizationProfile impact the conversion for TensorRT
  • new: batch_sizes and max_batch_size limit the max profile in TensorRT conversion
  • new: Allow to provide separate dataloader for profiling - first sample used only

• new: allow to run nav.package.optimize on empty package - status generation only

• new: use torch.inference_mode for inference runner when PyTorch 2.x is available

• fix: Missing model in config when passing package generated during nav.{framework}.optimize directly to nav.package.optimize command

• Other minor fixes and improvements

• Version of external components used during testing:

  • PyTorch 2.1.0a0+4136153
  • TensorFlow 2.12.0
  • TensorRT 8.6.1
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.47.1
  • GraphSurgeon: 0.3.26
  • tf2onnx v1.14.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.5.6

• fix: Load samples as sorted to keep valid order

• fix: Execute conversion when model already exists in path

• Other minor fixes and improvements

• Version of external components used during testing:

  • PyTorch 2.1.0a0+fe05266f
  • TensorFlow 2.12.0
  • TensorRT 8.6.1
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.47.1
  • GraphSurgeon: 0.3.26
  • tf2onnx v1.14.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.5.5

• new: Public nav.utilities module with UnpackedDataloader wrapper
• new: Added support for strict flag in Torch custom config
• new: Extended TensorRT custom config to support builder optimization level and hardware compatibility flags
• fix: Invalid optimal shape calculation for odd values in max batch size

• Version of external components used during testing:
  • PyTorch 2.1.0a0+fe05266f
  • TensorFlow 2.12.0
  • TensorRT 8.6.1
  • ONNX Runtime 1.14.1
  • Polygraphy: 0.47.1
  • GraphSurgeon: 0.3.26
  • tf2onnx v1.14.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.5.4

• new: Custom implementation for ONNX and TensorRT runners

• new: Use CUDA 12 for JAX in unit tests and functional tests

• new: Step-by-step examples

• new: Updated documentation

• new: TensorRTCUDAGraph runner introduced with support for CUDA graphs

• fix: Optimal shape not set correctly during adaptive conversion

• fix: Find max batch size command for JAX

• fix: Save stdout to logfiles in debug mode

• Version of external components used during testing:

  • PyTorch 2.1.0a0+fe05266f
  • TensorFlow 2.12.0
  • TensorRT 8.6.1
  • ONNX Runtime 1.14.1
  • Polygraphy: 0.47.1
  • GraphSurgeon: 0.3.26
  • tf2onnx v1.14.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.5.3

• fix: filter outputs using output_metadata in ONNX runners

• Version of external components used during testing:
  • PyTorch 2.0.0a0+1767026
  • TensorFlow 2.11.0
  • TensorRT 8.5.3.1
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.44.2
  • GraphSurgeon: 0.3.26
  • tf2onnx v1.14.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.5.2

• new: Added Contributor License Agreement (CLA)

• fix: Added missing --extra-index-url to installation instruction for pypi

• fix: Updated wheel readme

• fix: Do not run TorchScript export when only ONNX in target formats and ONNX extended export is disabled

• fix: Log full traceback for ModelNavigatorUserInputError

• Version of external components used during testing:

  • PyTorch 2.0.0a0+1767026
  • TensorFlow 2.11.0
  • TensorRT 8.5.3.1
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.44.2
  • GraphSurgeon: 0.3.26
  • tf2onnx v1.14.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.5.1

• fix: Using relative workspace cause error during Onnx to TensorRT conversion

• fix: Added external weight in package for ONNX format

• fix: bugfixes for functional tests

• Version of external components used during testing:

  • PyTorch 1.14.0a0+410ce96
  • TensorFlow 2.11.0
  • TensorRT 8.5.3
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.44.2
  • GraphSurgeon: 0.4.6
  • tf2onnx v1.13.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.5.0

• new: Support for PyTriton deployemnt

• new: Support for Python models with python.optimize API

• new: PyTorch 2 compile CPU and CUDA runners

• new: Collect conversion max batch size in status

• new: PyTorch runners with compile support

• change: Improved handling CUDA and CPU runners

• change: Reduced finding device max batch size time by running it once as separate pipeline

• change: Stored find max batch size result in separate filed in status

• Version of external components used during testing:

  • PyTorch 1.14.0a0+410ce96
  • TensorFlow 2.11.0
  • TensorRT 8.5.3
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.44.2
  • GraphSurgeon: 0.4.6
  • tf2onnx v1.13.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.4.4

• fix: when exporting single input model to saved model, unwrap one element list with inputs

• Version of external components used during testing:
  • PyTorch 1.14.0a0+410ce96
  • TensorFlow 2.11.0
  • TensorRT 8.5.3
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.44.2
  • GraphSurgeon: 0.4.6
  • tf2onnx v1.13.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.

Triton Model Navigator v0.4.3

• fix: in Keras inference use model.predict(tensor) for single input models

• Version of external components used during testing:
  • PyTorch 1.14.0a0+410ce96
  • TensorFlow 2.11.0
  • TensorRT 8.5.3
  • ONNX Runtime 1.13.1
  • Polygraphy: 0.44.2
  • GraphSurgeon: 0.4.6
  • tf2onnx v1.13.0
  • Other component versions depend on the used framework containers versions.
    See its support matrix
    for a detailed summary.