Skip to content

Latest commit

 

History

History
130 lines (112 loc) · 5.1 KB

README.md

File metadata and controls

130 lines (112 loc) · 5.1 KB

Image Classification Demo (C++) {#ovms_demo_image_classification_cpp}

This demo provides 2 clients:

  • classification_client_sync - simple client using synchronous gRPC API, testing accuracy of classification models
  • classification_client_async_benchmark - client using asynchronous gRPC API, testing accuracy and performance with real image data

To build the clients, run make command in this directory. It will build docker image named ovms_cpp_image_classification with all dependencies. The example clients image also contains test images required for accuracy measurements. It is also possible to use custom images.

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/image_classification/cpp
make

Note: In this directory you can only see the code specific to the benchmark client. The code shared with other C++ demos as well as all building utilities are placed in the common C++ directory.

Prepare classification model

Start OVMS with resnet50-binary model:

curl -L --create-dirs https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.bin -o resnet50-binary/1/model.bin https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.xml -o resnet50-binary/1/model.xml

Client requesting prediction synchronously

The client sends requests synchronously and displays latency for each request. You can specify number of iterations and layout: nchw, nhwc or binary. Each request contains image in selected format. The client also tests server responses for accuracy.

Prepare the server

chmod -R 755 resnet50-binary
docker run -d -u $(id -u):$(id -g) -v $(pwd)/resnet50-binary:/model -p 9001:9001 openvino/model_server:latest \
--model_path /model --model_name resnet --port 9001 --layout NHWC:NCHW

Start the client:

docker run --rm --network host -e "no_proxy=localhost" -v ${PWD}/input_images.txt:/clients/input_images.txt ovms_cpp_image_classification ./classification_client_sync --grpc_port=9001 --iterations=10 --layout="binary"

Address: localhost
Images list path: input_images.txt
Layout: binary
call predict ok
call predict time: 6ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 4ms
outputs size is 1
call predict ok
call predict time: 5ms
outputs size is 1
Overall accuracy: 100%
Total time divided by number of requests: 5ms

Clients requesting predictions asynchronously

The client sends requests asynchronously to mimic parallel clients scenario. There are plenty of parameters to configure those clients.

name description default available with synthetic data
grpc_address url to grpc service localhost yes
grpc_port port to grpc service 9000 yes
model_name model name to request resnet yes
input_name input tensor name with image 0 no, deduced automatically
output_name output tensor name with classification result 1463 no
iterations number of requests to be send by each producer thread 10 yes
batch_size batch size of each iteration 1 no, deduced automatically
images_list path to a file with a list of labeled images input_images.txt no
layout binary, nhwc or nchw nchw no, deduced automatically
producers number of threads asynchronously scheduling prediction 1 yes
consumers number of threads receiving responses 8 yes
max_parallel_requests maximum number of parallel inference requests; 0=no limit 100 yes
benchmark_mode 1 removes pre/post-processing and logging; 0 enables accuracy measurement 0 no

Async client with real image data

Prepare the server

docker run -d -u $(id -u):$(id -g) -v $(pwd)/resnet50-binary:/model -p 9001:9001 openvino/model_server:latest \
--model_path /model --model_name resnet --port 9001 --layout NCHW

Start the client:

docker run --rm --network host -e "no_proxy=localhost"  -v ${PWD}/input_images.txt:/clients/input_images.txt ovms_cpp_image_classification ./classification_client_async_benchmark --grpc_port=9001 --layout="nchw" --iterations=2000 --batch_size=1 --max_parallel_requests=100 --consumers=8 --producers=1 --benchmark_mode=1

Address: localhost:9001
Model name: resnet
Images list path: input_images.txt

Running the workload...
========================
        Summary
========================
Benchmark mode: True
Accuracy: N/A
Total time: 1976ms
Total iterations: 2000
Layout: nchw
Batch size: 1
Producer threads: 1
Consumer threads: 8
Max parallel requests: 100
Avg FPS: 1012.15