Skip to content

feat: [docs for ruby v3 ufc + bandits] (FF-2859) #432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: [docs for ruby v3 ufc + bandits] (FF-2859) #432

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0d55fc5
feat: [docs for ruby v3 ufc + bandits] (FF-2859)
leoromanovsky Jul 24, 2024
7fe61f8
Merge branch 'main' into lr/ff-2859
leoromanovsky Jul 29, 2024
2f81ca6
PR feedback
leoromanovsky Jul 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions docs/sdks/server-sdks/python.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ client = eppo_client.get_instance()
…
```

This generates a singleton client instance that can be reused throughout the application lifecycle
This generates a singleton client instance that can be reused throughout the application lifecycle.

### C. Assign variations

Expand All @@ -57,7 +57,6 @@ Here's how this configuration looks in the [flag page](https://eppo.cloud/featur

![Test checkout configuration](/img/feature-flagging/test-checkout-configuration.png)


That’s it: You can already start changing the feature flag on the page and see how it controls your code!

However, if you want to run experiments, there’s a little extra work to configure it properly.
Expand Down
243 changes: 189 additions & 54 deletions docs/sdks/server-sdks/ruby.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ Eppo's Ruby SDK can be used for both feature flagging and experiment assignment:
- [GitHub repository](https://github.com/Eppo-exp/ruby-sdk)
- [RubyGems gem](https://rubygems.org/gems/eppo-server-sdk/)

## 1. Install the SDK
## 1. Getting started

### A. Install the SDK

Install the SDK with gem:

Expand All @@ -19,33 +21,75 @@ gem install eppo-server-sdk
or add to you `Gemfile`:

```
gem 'eppo-server-sdk', '~> 0.3.0'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I honestly find it really weird we're jumping from 0.3.0 to 3.0.0. I understand the motivation but would find it jarring as a user

gem 'eppo-server-sdk', '~> 3.0.0'
```

## 2. Initialize the SDK
### B. Initialize the SDK

Initialize the SDK with a SDK key, which can be generated in the Eppo interface. Initialization the SDK when your application starts up to generate a singleton client instance, once per application lifecycle:
To initialize the SDK, you will need an SDK key. You can generate one [in the flag interface](https://eppo.cloud/feature-flags/keys).

```ruby
require 'eppo_client'

config = EppoClient::Config.new('<YOUR_API_KEY>')
client = EppoClient::init(config)
EppoClient::init(config)
client = EppoClient::Client.instance
```

After initialization, the SDK begins polling Eppo’s API at regular intervals to retrieve the most recent experiment configurations such as variation values and traffic allocation. The SDK stores these configurations in memory so that assignments are effectively instant. For more information, see the [architecture overview](/sdks/overview) page.
This generates a singleton client instance that can be reused throughout the application lifecycle.

If you are using the SDK for experiment assignments, make sure to pass in an assignment logging callback (see [section](#define-an-assignment-logger-experiment-assignment-only) below).
After initialization, the SDK begins polling Eppo's API at regular intervals to retrieve the most recent experiment configurations such as variation values and traffic allocation. The SDK stores these configurations in memory so that assignments are effectively instant. For more information, see the [architecture overview](/sdks/overview) page.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm not sure about "effectively instant" - I'd reword as "synchronous" to be more precise


:::info
### C. Assign variations

By default the Eppo client initialization is asynchronous to ensure no critical code paths are blocked. For more information on handling non-blocking initialization, see our [documentation here](/sdks/common-issues#3-not-handling-non-blocking-initialization).
Assign users to flags or experiments using `get_<type>_assignment`, depending on the type of the flag.
For example, for a String-valued flag, use `get_string_assignment`:

:::
```ruby
require 'eppo_client'

client = EppoClient::Client.instance
variation = client.get_string_assignment(
'<FLAG-KEY>',
'<SUBJECT-KEY>',
{
# Optional map of subject metadata for targeting.
},
'<DEFAULT-VALUE>'
)
```

The `get_string_assignment` function takes four required inputs to assign a variation:

* `<FLAG-KEY>` is the key that you chose when creating a flag; you can find it on the [flag page](https://eppo.cloud/feature-flags). For the rest of this presentation, we'll use `"test-checkout"`. To follow along, we recommend that you create a test flag in your account, and split users between `"fast_checkout"` and `"standard_checkout"`.
* `<SUBJECT-KEY>` is the value that identifies each entity in your experiment, typically `user_id`.
* `<SUBJECT-ATTRIBUTES>` is a dictionary of metadata about the subject used for targeting. If you create targeting rules based on attributes, those attributes must be passed in on every assignment call. If no attributes are needed, pass in an empty dictionary.
* `<DEFAULT-VALUE>` is the value that will be returned if no allocation matches the subject, if the flag is not enabled, if `get_string_assignment` is invoked before the SDK has finished initializing, or if the SDK was not able to retrieve the flag configuration.

### Typed assignments

Additional functions are available:

```
get_boolean_assignment(...)
get_numeric_assignment(...)
get_json_string_assignment(...)
get_parsed_json_assignment(...)
```

Here's how this configuration looks in the [flag page](https://eppo.cloud/feature-flags):

![Test checkout configuration](/img/feature-flagging/test-checkout-configuration.png)

That's it: You can already start changing the feature flag on the page and see how it controls your code!

However, if you want to run experiments, there's a little extra work to configure it properly.

## 2. Assignment Logging for Experiment

### Define an assignment logger (experiment assignment only)
If you are using the Eppo SDK for **experiment** assignment (i.e., randomization), we will need to know which entity, typically which user, passed through an entry point and was exposed to the experiment. For that, we need to log that information.

If you are using the Eppo SDK for experiment assignment (i.e randomization), include a logger instance in the config that is passed to the `init` function on SDK initialization. The SDK invokes the `log_assignment` method in the instance to capture assignment data whenever a variation is assigned.
Include a logger instance in the config that is passed to the `init` function on SDK initialization. The SDK invokes the `log_assignment` method in the instance to capture assignment data whenever a variation is assigned.

The code below illustrates an example implementation of logging with Segment, but you could also use other event-tracking systems. The only requirement is that the SDK can call a `log_assignment` method. Here we override Eppo's `AssignmentLogger` class with a function named `log_assignment`, then instantiate a config using an instance of the custom logger class, and finally instantiate the client:

Expand All @@ -65,7 +109,17 @@ config = EppoClient::Config.new(
'<YOUR_API_KEY>',
assignment_logger: CustomAssignmentLogger.new
)
client = EppoClient::init(config)
EppoClient::init(config)

# Allow the configuration to be loaded from the CDN.
sleep(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤨 no awaits or callbacks in ruby? or can we make a synchronous version?

Don't love seeing sleep() in our example 😞

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, ruby is single threaded by default though and doesnt have async primitives like async/await. we could have a callback instead, but that would require changing the interface


variation = EppoClient::Client.instance.get_string_assignment(
flag_key,
subject_key,
{},
'default'
)
```

The SDK will invoke the `log_assignment` function with an `assignment` object that contains the following fields:
Expand All @@ -84,71 +138,152 @@ The SDK will invoke the `log_assignment` function with an `assignment` object th
More details about logging and examples (with Segment, Rudderstack, mParticle, and Snowplow) can be found in the [event logging](/sdks/event-logging/) page.
:::

## 3. Assign variations
## 3. Running the SDK

Assigning users to flags or experiments with a single `get_string_assignment` function:
How is this SDK, hosted on your servers, actually getting the relevant information from Eppo?

```ruby
require 'eppo_client'
### A. Loading Configuration

client = EppoClient::Client.instance
variation = client.get_string_assignment(
'<SUBJECT-KEY>',
'<FLAG-KEY>',
{
# Optional map of subject metadata for targeting.
}
)
```
At initialization, the SDK polls Eppo's API to retrieve the most recent experiment configuration. The SDK stores that configuration in memory. This is why assignments are effectively instant, as you can see yourself by profiling the code above.

The `get_string_assignment` function takes two required and one optional input to assign a variation:
:::note

- `subject_key` - The entity ID that is being experimented on, typically represented by a uuid.
- `flag_or_experiment_key` - This key is available on the detail page for both flags and experiments.
- `subject_attributes` - An optional map of metadata about the subject used for targeting. If you create rules based on attributes on a flag/experiment, those attributes should be passed in on every assignment call.
Your users' private information doesn't leave your servers. Eppo only stores your flag and experiment configurations.

### Typed assignments
:::

Additional functions are available:
For more information on the performance of Eppo's SDKs, see the [latency](/sdks/faqs/latency) and [risk](/sdks/faqs/risk) pages.

```
get_boolean_assignment(...)
get_numeric_assignment(...)
get_json_string_assignment(...)
get_parsed_json_assignment(...)
```
### B. Automatically Updating the SDK Configuration

After initialization, the SDK continues polling Eppo's API at 30-second intervals. This retrieves the most recent flag and experiment configurations such as variation values, targeting rules, and traffic allocation. This happens independently of assignment calls.

### Handling `nil`
:::note

We recommend always handling the `nil` case in your code. Here are some examples of when the SDK returns `nil`:
Changes made to experiments on Eppo's web interface are almost instantly propagated through our Content-delivery network (CDN) Fastly. Because of the refresh rate, it may take up to 30 seconds (± 5 seconds fuzzing) for those to be reflected by the SDK assignments.

1. The **Traffic Exposure** setting on experiments/allocations determines the percentage of subjects the SDK will assign to that experiment/allocation. For example, if Traffic Exposure is 25%, the SDK will assign a variation for 25% of subjects and `nil` for the remaining 75% (unless the subject is part of an allow list).
:::

2. Assignments occur within the environments of feature flags. You must enable the environment corresponding to the feature flag's allocation in the user interface before `getStringAssignment` returns variations. It will return `nil` if the environment is not enabled.

![Toggle to enable environment](/img/feature-flagging/enable-environment.png)
:::info

3. If `get_string_assignment` is invoked before the SDK has finished initializing, the SDK may not have access to the most recent experiment configurations. In this case, the SDK will assign a variation based on any previously downloaded experiment configurations stored in local storage, or return `nil` if no configurations have been downloaded.
By default, the Eppo client initialization is asynchronous to ensure no critical code paths are blocked. For more information on handling non-blocking initialization, see our [documentation here](/sdks/common-issues#3-not-handling-non-blocking-initialization).

### Debugging `nil`
:::

If you need more visibility into why `get_string_assignment` is returning `nil`, you can change the logging level to `Logger::DEBUG` to see more details in the standard output.
## 4. Contextual Bandits

To leverage Eppo's contextual bandits using the Ruby SDK, there are two additional steps over regular feature flags:
1. Add a bandit action logger to the assignment logger
2. Querying the bandit for an action

### A. Add a bandit action logger to the assignment logger

In order for the bandit to learn an optimized policy, we need to capture and log the bandit actions.
This requires adding a bandit action logging callback to the AssignmentLogger class
```ruby
class MyLogger < EppoClient::AssignmentLogger
def log_assignment(assignment):
...

def log_bandit_action(self, bandit_action):
# implement me
```

We automatically log the following data:

| Field | Description | Example |
|------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|-------------------------------------|
| `timestamp` (Date) | The time when the action is taken in UTC | 2024-03-22T14:26:55.000Z |
| `flagKey` (String) | The key of the feature flag corresponding to the bandit | "bandit-test-allocation-4" |
| `banditKey` (String) | The key (unique identifier) of the bandit | "ad-bandit-1" |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does our Ruby event use cammel case property names? Should we make it snake for Ruby land? If we don't have a translation layer, given the difference in language conventions, I wonder if event case is some setting we pass through to rust.

| `subject` (String) | An identifier of the subject or user assigned to the experiment variation | "ed6f85019080" |
| `action` (String) | The action assigned by the bandit | "promo-20%-off" |
| `subjectNumericAttributes` (Hash{String => Float}) | Metadata about numeric attributes of the subject. Hash of the name of attributes their numeric values | `{"age": 30}` |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we're missing a word here. I think we want something like "hash of the name of the attributes to their numeric values" or "hash of the name of the attributes and their numeric values"

| `subjectCategoricalAttributes` (Hash{String => String}) | Metadata about non-numeric attributes of the subject. Hash of the name of attributes their string values | `{"loyalty_tier": "gold"}` |
| `actionNumericAttributes` (Hash{String => Float}) | Metadata about numeric attributes of the assigned action. Hash of the name of attributes their numeric values | `{"discount": 0.1}` |
| `actionCategoricalAttributes` (Hash{String => String}) | Metadata about non-numeric attributes of the assigned action. Hash of the name of attributes their string values | `{"promoTextColor": "white"}` |
| `actionProbability` (Float) | The weight between 0 and 1 the bandit valued the assigned action | 0.25 |
| `modelVersion` (String) | Unique identifier for the version (iteration) of the bandit parameters used to determine the action probability | "v123" |

### B. Querying the bandit for an action

To query the bandit for an action, you can use the `get_bandit_action` function. This function takes the following parameters:
- `flag_key` (str): The key of the feature flag corresponding to the bandit
- `subject_key` (str): The key of the subject or user assigned to the experiment variation
- `subject_attributes` (Attributes): The context of the subject
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other SDKs we allow either freeform attributes ([key] => bool | number | string) or contextual attributes (numeric_attributes => [key] => number, categorical_attributes => [key] => bool | number | string).

Assuming we do here (and if not, we should adjust) we should call this out in the documentation. It may also help to right after this explain the difference, as we do in our JavaScript documentation (link).

- `actions` (Hash{String => Attributes}): A hash that maps available actions to their attributes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for actions, except we also allow just a set of Strings too if we don't have any action context.

- `default` (str): The default *variation* to return if the bandit cannot be queried

The following code queries the bandit for an action:

```ruby
require 'eppo_client'
require 'logger'

client = EppoClient::Client.instance
variation = client.get_string_assignment(
'<SUBJECT-KEY>',
'<FLAG-KEY>',
{},
Logger::DEBUG
bandit_result = client.get_bandit_action(
"shoe-bandit",
name,
EppoClient::Attributes.new(
numeric_attributes: { "age" => age }, categorical_attributes: { "country" => country }
),
{
"nike" => EppoClient::Attributes.new(
numeric_attributes: { "brand_affinity" => 2.3 },
categorical_attributes: { "image_aspect_ratio" => "16:9" }
),
"adidas" => EppoClient::Attributes.new(
numeric_attributes: { "brand_affinity" => 0.2 },
categorical_attributes: { "image_aspect_ratio" => "16:9" }
)
},
"control"
)
```

<br />
#### Subject Context

The subject context contains contextual information about the subject that is independent of bandit actions.
For example, the subject's age or country.

The subject context has type `Attributes` which has two fields:

- `numeric_attributes` (Hash{String => Float}): A hash of numeric attributes (such as "age")
- `categorical_attributes` (Hash{String => String}): A hash of categorical attributes (such as "country")

:::note
It may take up to 10 seconds for changes to Eppo experiments to be reflected by the SDK assignments.
The `categerical_attributes` are also used for targeting rules for the feature flag similar to how `subject_attributes` are used for that with regular feature flags.
:::

#### Action Contexts

Next, supply a hash with actions and their attributes: `actions: Hash{String => Attributes}`.
If the user is assigned to the bandit, the bandit selects one of the actions supplied here,
and all actions supplied are considered to be valid; if an action should not be shown to a user, do not include it in this hash.

The action attributes are similar to the `subject_attributes` but hold action-specific information.
Note that we can use `Attributes.empty` to create an empty attribute context.

Note that action contexts can contain two kinds of information:
- Action-specific context: e.g., the image aspect ratio of the image corresponding to this action
- User-action interaction context: e.g., there could be a "brand-affinity" model that computes brand affinities of users to brands, and scores of this model can be added to the action context to provide additional context for the bandit.

#### Result

The `bandit_result` is an instance of `BanditResult`, which has two fields:

- `variation` (String): The variation that was assigned to the subject
- `action` (Optional[String]): The action that was assigned to the subject
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is Optional[String] Ruby-speek?


The variation returns the feature flag variation; this can be the bandit itself, or the "status quo" variation if the user is not assigned to the bandit.
If we are unable to generate a variation, for example when the flag is turned off, then the `default` variation is returned.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is another case, where the bandit encounters an error selecting an action (or there are no actions provided) in which case it will return the bandit variation, but a nil action. I wonder if that's worth explicitly calling out--I think it is.

In both of those cases, the `action` is `nil`, and you should use the status-quo algorithm to select an action.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this may not make sense because one of the above cases is the bandit itself, in which case action won't be nil.

Perhaps reword to "In both of those cases, when action is nil, you should use the status-quo algorithm to select an action.


When `action` is not `nil`, the bandit has selected that action to be shown to the user.

#### Status quo algorithm

In order to accurately measure the performance of the bandit, we need to compare it to the status quo algorithm using an experiment.
This status quo algorithm could be a complicated algorithm that selects an action according to a different model, or a simple baseline such as selecting a fixed or random action.
When you create an analysis allocation for the bandit and the `action` in `BanditResult` is `nil`, implement the desired status quo algorithm based on the `variation` value.