Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/analytics.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/azure.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/azuredashboard.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/batch-processor.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/car.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/cat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/cat_explanation.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/cicd-seldon.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/cifar10-od_34_0.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/cifar10-od_34_2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/dashboard_image.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/demo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/deploy-graph.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/e2e-model-serving.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/git-user.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions docs-gb/.gitbook/assets/gpt2-graph.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs-gb/.gitbook/assets/graf.jpg
Binary file added docs-gb/.gitbook/assets/grafana-mlflow.jpg
Binary file added docs-gb/.gitbook/assets/inf-graph.png
Binary file added docs-gb/.gitbook/assets/k-pipeline-dashboard.jpg
Binary file added docs-gb/.gitbook/assets/k8s-dashboard-1.jpg
Binary file added docs-gb/.gitbook/assets/k8s-dashboard-2.jpg
Binary file added docs-gb/.gitbook/assets/kubeflow-pipeline.jpg
1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/language-wrappers-1.svg

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/language-wrappers-2.svg

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/metadata.svg

Large diffs are not rendered by default.

Binary file added docs-gb/.gitbook/assets/metrics.png
Binary file added docs-gb/.gitbook/assets/mlflow-dashboard.png
1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/model-servers.svg

Large diffs are not rendered by default.

Binary file added docs-gb/.gitbook/assets/new-item.png
1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/operator.svg

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/orchestrator.svg

Large diffs are not rendered by default.

Binary file added docs-gb/.gitbook/assets/outliers-combiner-1.jpg
Binary file added docs-gb/.gitbook/assets/outliers-combiner-2.jpg
Binary file added docs-gb/.gitbook/assets/outliers-combiner-3.jpg
Binary file added docs-gb/.gitbook/assets/pipeline-config.png
Binary file added docs-gb/.gitbook/assets/pipeline-stages.png
Binary file added docs-gb/.gitbook/assets/prometheus-graph.png
Binary file added docs-gb/.gitbook/assets/prometheus-targets.png
Binary file added docs-gb/.gitbook/assets/running-pipeline.jpg
Binary file added docs-gb/.gitbook/assets/scm-config.png
Binary file added docs-gb/.gitbook/assets/seldon-analytics.jpg
Binary file added docs-gb/.gitbook/assets/seldon-iter8-single.png
1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/seldondeployment-crd.svg

Large diffs are not rendered by default.

Binary file added docs-gb/.gitbook/assets/tensorflow-grpc-kafka.png
Binary file added docs-gb/.gitbook/assets/tensorflow-rest-kafka.png
32 changes: 32 additions & 0 deletions docs-gb/.gitbook/assets/tfserving-mnist_20_0.svg
Binary file added docs-gb/.gitbook/assets/tfserving-mnist_23_0.png
1 change: 1 addition & 0 deletions docs-gb/.gitbook/assets/tracing.svg

Large diffs are not rendered by default.

Binary file added docs-gb/.gitbook/assets/triton_examples_11_1.png
Binary file added docs-gb/.gitbook/assets/triton_examples_17_1.png
Binary file added docs-gb/.gitbook/assets/triton_examples_24_1.png
Binary file added docs-gb/.gitbook/assets/triton_examples_30_1.png
8 changes: 4 additions & 4 deletions docs-gb/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
* [Model Metadata](deployments/metadata.md)
* [Budgeting Disruptions](deployments/disruption-budgets.md)
* [Graph Deployment Options](deployments/graph-modes.md)
* [AB Tests and Progerssive Rollouts](deployments/abtests.md)
* [AB Tests and Progressive Rollouts](deployments/abtests.md)
* [Troubleshooting Deployments](deployments/troubleshooting.md)
* [Servers](configuration/servers/README.md)
* [Custom Inference Servers](servers/custom.md)
Expand Down Expand Up @@ -173,9 +173,9 @@
* [Open Inference Protocol](reference/prediction-apis/v2-protocol/README.md)
* [REST](reference/prediction-apis/v2-protocol/rest/README.md)
* ```yaml
type: builtin:openapi
props:
models: true
type: builtin:openapi
dependencies:
spec:
ref:
Expand All @@ -190,9 +190,9 @@
* [Prediction Open API Spec](reference/prediction-apis/prediction-open-api-spec/README.md)
* [Seldon Core External via Ambassador](reference/prediction-apis/prediction-open-api-spec/seldon-core-external-via-ambassador/README.md)
* ```yaml
type: builtin:openapi
props:
models: true
type: builtin:openapi
dependencies:
spec:
ref:
Expand All @@ -201,9 +201,9 @@
```
* [Seldon Core Internal microservice API](reference/prediction-apis/prediction-open-api-spec/seldon-core-internal-microservice-api/README.md)
* ```yaml
type: builtin:openapi
props:
models: true
type: builtin:openapi
dependencies:
spec:
ref:
Expand Down
26 changes: 11 additions & 15 deletions docs-gb/deployments/abtests.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,28 @@
# AB Testing and Progressive Rollouts
# AB Tests and Progressive Rollouts

## Simple AB Tests

Seldon Core provides the ability to easily create AB Tests and Shadows using Istio and Ambassador to split traffic as required.

* [Istio AB Test/Canary Example](../examples/istio_canary.html)
* [Ambassador AB Tests/Canary Example](../examples/ambassador_canary.html)
* [Istio AB Test/Canary Example](../examples/istio_canary.html)
* [Ambassador AB Tests/Canary Example](../examples/ambassador_canary.html)

Metrics can be evaluated in prometheus for the different predictors in the AB Test using the [Seldon Analytics dashboard](../analytics/analytics.html).


## Advanced AB Test Experiments and Progressive Rollouts

For more advanced use cases we recommend our integration with [Iter8](https://iter8.tools) to provide clear experimentation utilizing clear objectives and rewards for candidate model selection. Iter8 also provides progressive rollout capabilities to automatically allow testing of candidate models and promoting them to the production model if they perform better than the incumbant model.

In Seldon we provide two current examples on how to run Iter8 experiments.

1. Seldon/Iter8 Experiment over single Seldon Deployment.
1. Seldon/Iter8 experiment over separate Seldon Deployments.

1. Seldon/Iter8 Experiment over single Seldon Deployment.
2. Seldon/Iter8 experiment over separate Seldon Deployments.

## Seldon - Iter8 Experiment over single Seldon Deployment

The first option is to create an AB Test for the candidate model with an updated Seldon Deployment and run an Iter8 experiment to progressively rollout the candidate based on a set of metrics. The architecture is show below:

![seldonIter8Single](../images/seldon-iter8-single.png)

![seldonIter8Single](../.gitbook/assets/seldon-iter8-single.png)

We begin by updating our default model to start an AB test as shown below:

Expand Down Expand Up @@ -259,10 +256,10 @@ spec:

This has several key sections:

* Strategy: The type of experiment to run and actions to take on completion.
* Criteria: Key metrics for rewards and service objectives.
* Duration: How long to run the experiment.
* VersionInfo: Details of the various candidate models to compare.
* Strategy: The type of experiment to run and actions to take on completion.
* Criteria: Key metrics for rewards and service objectives.
* Duration: How long to run the experiment.
* VersionInfo: Details of the various candidate models to compare.

Once the experiment is launched traffic will be moved to the various candidates based on the defined rewards and objectives.

Expand Down Expand Up @@ -340,7 +337,7 @@ We can also run experiments over separate Seldon Deployments. This though would

The architecture for this type of experiment is shown below:

![seldonIter8Separate](../images/seldon-iter8-separate.png)
![seldonIter8Separate](../.gitbook/assets/seldon-iter8-separate.png)

The difference here is we have two Seldon Deployments. A baseline:

Expand Down Expand Up @@ -388,7 +385,6 @@ spec:

Then, for Istio we need a new routing-rule to split traffic between the two:


```yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
Expand Down
2 changes: 1 addition & 1 deletion docs-gb/deployments/deploying.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Deployment
# Deployment Techniques

Once you have created your inference graph as a JSON or YAML Seldon Deployment resource you can deploy to Kubernetes with `kubectl`. For example, if your deployment is packaged in `my_ml_deployment.yaml`:

Expand Down
Loading
Loading