From 0d55fc57700daf93c90e2671d48b67e41b45ed02 Mon Sep 17 00:00:00 2001 From: Leo Romanovsky Date: Wed, 24 Jul 2024 11:11:30 -0700 Subject: [PATCH 1/2] feat: [docs for ruby v3 ufc + bandits] (FF-2859) --- docs/sdks/server-sdks/python.md | 3 +- docs/sdks/server-sdks/ruby.md | 243 +++++++++++++++++++++++++------- 2 files changed, 190 insertions(+), 56 deletions(-) diff --git a/docs/sdks/server-sdks/python.md b/docs/sdks/server-sdks/python.md index 96f69aab..66ad868e 100644 --- a/docs/sdks/server-sdks/python.md +++ b/docs/sdks/server-sdks/python.md @@ -33,7 +33,7 @@ client = eppo_client.get_instance() … ``` -This generates a singleton client instance that can be reused throughout the application lifecycle +This generates a singleton client instance that can be reused throughout the application lifecycle. ### C. Assign variations @@ -57,7 +57,6 @@ Here's how this configuration looks in the [flag page](https://eppo.cloud/featur ![Test checkout configuration](/img/feature-flagging/test-checkout-configuration.png) - That’s it: You can already start changing the feature flag on the page and see how it controls your code! However, if you want to run experiments, there’s a little extra work to configure it properly. diff --git a/docs/sdks/server-sdks/ruby.md b/docs/sdks/server-sdks/ruby.md index 64650fd7..fca8d960 100644 --- a/docs/sdks/server-sdks/ruby.md +++ b/docs/sdks/server-sdks/ruby.md @@ -8,7 +8,9 @@ Eppo's Ruby SDK can be used for both feature flagging and experiment assignment: - [GitHub repository](https://github.com/Eppo-exp/ruby-sdk) - [RubyGems gem](https://rubygems.org/gems/eppo-server-sdk/) -## 1. Install the SDK +## 1. Getting started + +### A. Install the SDK Install the SDK with gem: @@ -19,33 +21,75 @@ gem install eppo-server-sdk or add to you `Gemfile`: ``` -gem 'eppo-server-sdk', '~> 0.3.0' +gem 'eppo-server-sdk', '~> 3.0.0' ``` -## 2. Initialize the SDK +### B. Initialize the SDK -Initialize the SDK with a SDK key, which can be generated in the Eppo interface. Initialization the SDK when your application starts up to generate a singleton client instance, once per application lifecycle: +To initialize the SDK, you will need an SDK key. You can generate one [in the flag interface](https://eppo.cloud/feature-flags/keys). ```ruby require 'eppo_client' config = EppoClient::Config.new('') -client = EppoClient::init(config) +EppoClient::init(config) +client = EppoClient::Client.instance ``` -After initialization, the SDK begins polling Eppo’s API at regular intervals to retrieve the most recent experiment configurations such as variation values and traffic allocation. The SDK stores these configurations in memory so that assignments are effectively instant. For more information, see the [architecture overview](/sdks/overview) page. +This generates a singleton client instance that can be reused throughout the application lifecycle. -If you are using the SDK for experiment assignments, make sure to pass in an assignment logging callback (see [section](#define-an-assignment-logger-experiment-assignment-only) below). +After initialization, the SDK begins polling Eppo's API at regular intervals to retrieve the most recent experiment configurations such as variation values and traffic allocation. The SDK stores these configurations in memory so that assignments are effectively instant. For more information, see the [architecture overview](/sdks/overview) page. -:::info +### C. Assign variations -By default the Eppo client initialization is asynchronous to ensure no critical code paths are blocked. For more information on handling non-blocking initialization, see our [documentation here](/sdks/common-issues#3-not-handling-non-blocking-initialization). +Assign users to flags or experiments using `get__assignment`, depending on the type of the flag. +For example, for a String-valued flag, use `get_string_assignment`: -::: +```ruby +require 'eppo_client' + +client = EppoClient::Client.instance +variation = client.get_string_assignment( + '', + '', + { + # Optional map of subject metadata for targeting. + }, + '' +) +``` + +The `get_string_assignment` function takes four required inputs to assign a variation: + +* `` is the key that you chose when creating a flag; you can find it on the [flag page](https://eppo.cloud/feature-flags). For the rest of this presentation, we'll use `"test-checkout"`. To follow along, we recommend that you create a test flag in your account, and split users between `"fast_checkout"` and `"standard_checkout"`. +* `` is the value that identifies each entity in your experiment, typically `user_id`. +* `` is a dictionary of metadata about the subject used for targeting. If you create targeting rules based on attributes, those attributes must be passed in on every assignment call. If no attributes are needed, pass in an empty dictionary. +* `` is the value that will be returned if no allocation matches the subject, if the flag is not enabled, if `get_string_assignment` is invoked before the SDK has finished initializing, or if the SDK was not able to retrieve the flag configuration. + +### Typed assignments + +Additional functions are available: + +``` +get_boolean_assignment(...) +get_numeric_assignment(...) +get_json_string_assignment(...) +get_parsed_json_assignment(...) +``` + +Here's how this configuration looks in the [flag page](https://eppo.cloud/feature-flags): + +![Test checkout configuration](/img/feature-flagging/test-checkout-configuration.png) + +That's it: You can already start changing the feature flag on the page and see how it controls your code! + +However, if you want to run experiments, there's a little extra work to configure it properly. + +## 2. Assignment Logging for Experiment -### Define an assignment logger (experiment assignment only) +If you are using the Eppo SDK for **experiment** assignment (i.e., randomization), we will need to know which entity, typically which user, passed through an entry point and was exposed to the experiment. For that, we need to log that information. -If you are using the Eppo SDK for experiment assignment (i.e randomization), include a logger instance in the config that is passed to the `init` function on SDK initialization. The SDK invokes the `log_assignment` method in the instance to capture assignment data whenever a variation is assigned. +Include a logger instance in the config that is passed to the `init` function on SDK initialization. The SDK invokes the `log_assignment` method in the instance to capture assignment data whenever a variation is assigned. The code below illustrates an example implementation of logging with Segment, but you could also use other event-tracking systems. The only requirement is that the SDK can call a `log_assignment` method. Here we override Eppo's `AssignmentLogger` class with a function named `log_assignment`, then instantiate a config using an instance of the custom logger class, and finally instantiate the client: @@ -65,7 +109,17 @@ config = EppoClient::Config.new( '', assignment_logger: CustomAssignmentLogger.new ) -client = EppoClient::init(config) +EppoClient::init(config) + +# Allow the configuration to be loaded from the CDN. +sleep(1) + +variation = EppoClient::Client.instance.get_string_assignment( + flag_key, + subject_key, + {}, + 'default' +) ``` The SDK will invoke the `log_assignment` function with an `assignment` object that contains the following fields: @@ -84,71 +138,152 @@ The SDK will invoke the `log_assignment` function with an `assignment` object th More details about logging and examples (with Segment, Rudderstack, mParticle, and Snowplow) can be found in the [event logging](/sdks/event-logging/) page. ::: -## 3. Assign variations +## 3. Running the SDK -Assigning users to flags or experiments with a single `get_string_assignment` function: +How is this SDK, hosted on your servers, actually getting the relevant information from Eppo? -```ruby -require 'eppo_client' +### A. Loading Configuration -client = EppoClient::Client.instance -variation = client.get_string_assignment( - '', - '', - { - # Optional map of subject metadata for targeting. - } -) -``` +At initialization, the SDK polls Eppo's API to retrieve the most recent experiment configuration. The SDK stores that configuration in memory. This is why assignments are effectively instant, as you can see yourself by profiling the code above. -The `get_string_assignment` function takes two required and one optional input to assign a variation: +:::note -- `subject_key` - The entity ID that is being experimented on, typically represented by a uuid. -- `flag_or_experiment_key` - This key is available on the detail page for both flags and experiments. -- `subject_attributes` - An optional map of metadata about the subject used for targeting. If you create rules based on attributes on a flag/experiment, those attributes should be passed in on every assignment call. +Your users' private information doesn't leave your servers. Eppo only stores your flag and experiment configurations. -### Typed assignments +::: -Additional functions are available: +For more information on the performance of Eppo's SDKs, see the [latency](/sdks/faqs/latency) and [risk](/sdks/faqs/risk) pages. -``` -get_boolean_assignment(...) -get_numeric_assignment(...) -get_json_string_assignment(...) -get_parsed_json_assignment(...) -``` +### B. Automatically Updating the SDK Configuration + +After initialization, the SDK continues polling Eppo's API at 30-second intervals. This retrieves the most recent flag and experiment configurations such as variation values, targeting rules, and traffic allocation. This happens independently of assignment calls. -### Handling `nil` +:::note -We recommend always handling the `nil` case in your code. Here are some examples of when the SDK returns `nil`: +Changes made to experiments on Eppo's web interface are almost instantly propagated through our Content-delivery network (CDN) Fastly. Because of the refresh rate, it may take up to 30 seconds (± 5 seconds fuzzing) for those to be reflected by the SDK assignments. -1. The **Traffic Exposure** setting on experiments/allocations determines the percentage of subjects the SDK will assign to that experiment/allocation. For example, if Traffic Exposure is 25%, the SDK will assign a variation for 25% of subjects and `nil` for the remaining 75% (unless the subject is part of an allow list). +::: -2. Assignments occur within the environments of feature flags. You must enable the environment corresponding to the feature flag's allocation in the user interface before `getStringAssignment` returns variations. It will return `nil` if the environment is not enabled. -![Toggle to enable environment](/img/feature-flagging/enable-environment.png) +:::info -3. If `get_string_assignment` is invoked before the SDK has finished initializing, the SDK may not have access to the most recent experiment configurations. In this case, the SDK will assign a variation based on any previously downloaded experiment configurations stored in local storage, or return `nil` if no configurations have been downloaded. +By default, the Eppo client initialization is asynchronous to ensure no critical code paths are blocked. For more information on handling non-blocking initialization, see our [documentation here](/sdks/common-issues#3-not-handling-non-blocking-initialization). -### Debugging `nil` +::: -If you need more visibility into why `get_string_assignment` is returning `nil`, you can change the logging level to `Logger::DEBUG` to see more details in the standard output. +## 4. Contextual Bandits + +To leverage Eppo's contextual bandits using the Ruby SDK, there are two additional steps over regular feature flags: +1. Add a bandit action logger to the assignment logger +2. Querying the bandit for an action + +### A. Add a bandit action logger to the assignment logger + +In order for the bandit to learn an optimized policy, we need to capture and log the bandit actions. +This requires adding a bandit action logging callback to the AssignmentLogger class +```ruby +class MyLogger < EppoClient::AssignmentLogger + def log_assignment(assignment): + ... + + def log_bandit_action(self, bandit_action): + # implement me +``` + +We automatically log the following data: + +| Field | Description | Example | +|------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|-------------------------------------| +| `timestamp` (Date) | The time when the action is taken in UTC | 2024-03-22T14:26:55.000Z | +| `flagKey` (String) | The key of the feature flag corresponding to the bandit | "bandit-test-allocation-4" | +| `banditKey` (String) | The key (unique identifier) of the bandit | "ad-bandit-1" | +| `subject` (String) | An identifier of the subject or user assigned to the experiment variation | "ed6f85019080" | +| `action` (String) | The action assigned by the bandit | "promo-20%-off" | +| `subjectNumericAttributes` (Hash{String => Float}) | Metadata about numeric attributes of the subject. Hash of the name of attributes their numeric values | `{"age": 30}` | +| `subjectCategoricalAttributes` (Hash{String => String}) | Metadata about non-numeric attributes of the subject. Hash of the name of attributes their string values | `{"loyalty_tier": "gold"}` | +| `actionNumericAttributes` (Hash{String => Float}) | Metadata about numeric attributes of the assigned action. Hash of the name of attributes their numeric values | `{"discount": 0.1}` | +| `actionCategoricalAttributes` (Hash{String => String}) | Metadata about non-numeric attributes of the assigned action. Hash of the name of attributes their string values | `{"promoTextColor": "white"}` | +| `actionProbability` (Float) | The weight between 0 and 1 the bandit valued the assigned action | 0.25 | +| `modelVersion` (String) | Unique identifier for the version (iteration) of the bandit parameters used to determine the action probability | "v123" | + +### B. Querying the bandit for an action + +To query the bandit for an action, you can use the `get_bandit_action` function. This function takes the following parameters: +- `flag_key` (str): The key of the feature flag corresponding to the bandit +- `subject_key` (str): The key of the subject or user assigned to the experiment variation +- `subject_attributes` (Attributes): The context of the subject +- `actions` (Hash{String => Attributes}): A hash that maps available actions to their attributes +- `default` (str): The default *variation* to return if the bandit cannot be queried + +The following code queries the bandit for an action: ```ruby require 'eppo_client' -require 'logger' client = EppoClient::Client.instance -variation = client.get_string_assignment( - '', - '', - {}, - Logger::DEBUG +bandit_result = client.get_bandit_action( + "shoe-bandit", + name, + EppoClient::Attributes.new( + numeric_attributes: { "age" => age }, categorical_attributes: { "country" => country } + ), + { + "nike" => EppoClient::Attributes.new( + numeric_attributes: { "brand_affinity" => 2.3 }, + categorical_attributes: { "image_aspect_ratio" => "16:9" } + ), + "adidas" => EppoClient::Attributes.new( + numeric_attributes: { "brand_affinity" => 0.2 }, + categorical_attributes: { "image_aspect_ratio" => "16:9" } + ) + }, + "control" ) ``` -
+#### Subject Context + +The subject context contains contextual information about the subject that is independent of bandit actions. +For example, the subject's age or country. + +The subject context has type `Attributes` which has two fields: + +- `numeric_attributes` (Hash{String => Float}): A hash of numeric attributes (such as "age") +- `categorical_attributes` (Hash{String => String}): A hash of categorical attributes (such as "country") :::note -It may take up to 10 seconds for changes to Eppo experiments to be reflected by the SDK assignments. +The `categerical_attributes` are also used for targeting rules for the feature flag similar to how `subject_attributes` are used for that with regular feature flags. ::: + +#### Action Contexts + +Next, supply a hash with actions and their attributes: `actions: Hash{String => Attributes}`. +If the user is assigned to the bandit, the bandit selects one of the actions supplied here, +and all actions supplied are considered to be valid; if an action should not be shown to a user, do not include it in this hash. + +The action attributes are similar to the `subject_attributes` but hold action-specific information. +Note that we can use `Attributes.empty` to create an empty attribute context. + +Note that action contexts can contain two kinds of information: +- Action-specific context: e.g., the image aspect ratio of the image corresponding to this action +- User-action interaction context: e.g., there could be a "brand-affinity" model that computes brand affinities of users to brands, and scores of this model can be added to the action context to provide additional context for the bandit. + +#### Result + +The `bandit_result` is an instance of `BanditResult`, which has two fields: + +- `variation` (String): The variation that was assigned to the subject +- `action` (Optional[String]): The action that was assigned to the subject + +The variation returns the feature flag variation; this can be the bandit itself, or the "status quo" variation if the user is not assigned to the bandit. +If we are unable to generate a variation, for example when the flag is turned off, then the `default` variation is returned. +In both of those cases, the `action` is `nil`, and you should use the status-quo algorithm to select an action. + +When `action` is not `nil`, the bandit has selected that action to be shown to the user. + +#### Status quo algorithm + +In order to accurately measure the performance of the bandit, we need to compare it to the status quo algorithm using an experiment. +This status quo algorithm could be a complicated algorithm that selects an action according to a different model, or a simple baseline such as selecting a fixed or random action. +When you create an analysis allocation for the bandit and the `action` in `BanditResult` is `nil`, implement the desired status quo algorithm based on the `variation` value. + From 2f81ca69b12f1e083e1f87516a4dbc1fbaca9caa Mon Sep 17 00:00:00 2001 From: Leo Romanovsky Date: Mon, 29 Jul 2024 15:14:13 -0700 Subject: [PATCH 2/2] PR feedback --- docs/sdks/client-sdks/javascript.md | 2 +- docs/sdks/client-sdks/react-native.md | 2 +- docs/sdks/server-sdks/dotnet.md | 2 +- docs/sdks/server-sdks/go.md | 2 +- docs/sdks/server-sdks/php.md | 2 +- docs/sdks/server-sdks/ruby.md | 20 ++++++++++---------- 6 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/sdks/client-sdks/javascript.md b/docs/sdks/client-sdks/javascript.md index 50bdf006..6e60d18c 100644 --- a/docs/sdks/client-sdks/javascript.md +++ b/docs/sdks/client-sdks/javascript.md @@ -53,7 +53,7 @@ If you install via a `