Skip to content

benchmarking.md

Latest commit

 

History

History
60 lines (35 loc) · 1.51 KB

benchmarking.md

File metadata and controls

60 lines (35 loc) · 1.51 KB

Seldon-core Benchmarking and Load Testing

This page is a work in progress to provide benchmarking and load testing.

This work is ongoing and we welcome feedback

Tools

  • For REST tests we use vegeta
  • For gRPC tests we use ghz

Service Orchestrator

These benchmark tests are to evaluate the extra latency added by including the service orchestrator.

Results

On A 3 node DigitalOcean cluster 24vCPUs 96 GB, running Tensorflow Flowers image classfier.

Test Additional latency
REST 9ms
gRPC 4ms

Further work:

  • Statistical confidence test

Tensorflow

Test the max throughput and HPA usage.

Results

On A 3 node DigitalOcean cluster 24vCPUs 96 GB, running Tensorflow Flowers image classfier with HPA and running at max throughput for a single model. No ramp up, as vegeta does not support this. See notebook for details.

Latencies:

mean: 259.990239 ms
50th: 131.917169 ms
90th: 310.053255 ms
95th: 916.684759 ms
99th: 2775.05271 ms

Throughput: 23.997572337989126/s
Errors: False

Flexible Benchmarking with Argo Workflows

We have also an example that shows how to leverage the batch processing workflow that we showcase in the examples, but to perform benchmarking with Seldon Core models.