Skip to content

Implement stale while revalidate (SWR) cache strategy #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kxzk merged 24 commits into from
Jan 22, 2026
Merged

Implement stale while revalidate (SWR) cache strategy #35

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
593f0e8
feat(deps): Add concurrent-ruby gem
drborges Nov 28, 2025
d2a9e13
feat(config): add stale-while-revalidate configuration options
drborges Nov 28, 2025
098c59b
refactor(cache): update CacheEntry to support fresh/stale/expired states
drborges Nov 28, 2025
f761e8f
feat(cache): implement stale-while-revalidate in RailsCacheAdapter
drborges Nov 28, 2025
1c184ef
feat(client): wire SWR configuration to RailsCacheAdapter
drborges Nov 28, 2025
d0d4772
refactor(api_client): add intelligent caching strategy selection
drborges Nov 28, 2025
26ce6d6
fix(rubocop): disable RSpec/MultipleMemoizedHelpers for a few specs
drborges Nov 28, 2025
d5b915d
feat(prompt_cache): Add SWR to PromptCache
drborges Dec 2, 2025
e8ae59d
feat(config): Allow configuring SWR caches that never expire
drborges Dec 3, 2025
687e1e0
refactor(config): Apply better defaults for stale_ttl
drborges Dec 3, 2025
ab7628d
docs(swr): Add documentation on stale while revalidate behavior
drborges Dec 3, 2025
b9abbd8
fix(swr): Consider SWR enabled as long as stale_ttl is positive
drborges Dec 4, 2025
dd24ee8
style(lint): Disable ClassLength for ApiClient
drborges Dec 4, 2025
316706a
refactor(swr): raises error when attempting to use SWR and invalid st…
drborges Dec 5, 2025
16e7618
style(comments): Remove emoji from comments
drborges Dec 9, 2025
e18b701
refactor(cache): Unify locking API across cache implementations
drborges Dec 10, 2025
30b72d3
refactor(cache): Move fetch_with_lock to RailsCacheAdapter
drborges Dec 10, 2025
45425f2
refactor(cache): Remove expires_in parameter from set/cache_set methods
drborges Dec 10, 2025
737539f
feat(shutdown): Add cache shutdown to client cleanup
drborges Dec 10, 2025
f7e9fdc
refactor(config): Use :indefinite symbol instead of Float::INFINITY f…
drborges Dec 10, 2025
c2b2b4b
feat(config): Add helpful validation for SWR with nil cache_stale_ttl
drborges Dec 10, 2025
df3ace7
refactor(swr): Set min_threads to 0 for on-demand thread creation
drborges Dec 12, 2025
f76f579
docs(swr): Document memory leak risk with in-memory locks in PromptCache
drborges Dec 12, 2025
9f26069
Merge branch 'main' of github.com:simplepractice/langfuse-rb into fea…
drborges Jan 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ PATH
specs:
langfuse-rb (0.1.0)
base64 (~> 0.2)
concurrent-ruby (~> 1.2)
faraday (~> 2.0)
faraday-retry (~> 2.0)
mustache (~> 1.1)
Expand All @@ -19,6 +20,7 @@ GEM
ast (2.4.3)
base64 (0.3.0)
bigdecimal (3.3.1)
concurrent-ruby (1.3.5)
crack (1.0.0)
bigdecimal
rexml
Expand Down Expand Up @@ -137,6 +139,7 @@ GEM

PLATFORMS
arm64-darwin-22
arm64-darwin-24
arm64-darwin-25
x86_64-linux

Expand Down
8 changes: 6 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

- 🎯 **Prompt Management** - Centralized prompt versioning with Mustache templating
- 📊 **LLM Tracing** - Zero-boilerplate observability built on OpenTelemetry
- ⚡ **Performance** - In-memory or Redis-backed caching with stampede protection
- ⚡ **Performance** - In-memory or Redis-backed caching with stampede protection, both supporting stale-while-revalidate cache strategy
- 💬 **Chat & Text Prompts** - First-class support for both formats
- 🔄 **Automatic Retries** - Built-in exponential backoff for resilient API calls
- 🛡️ **Fallback Support** - Graceful degradation when API unavailable
Expand Down Expand Up @@ -41,6 +41,10 @@ Langfuse.configure do |config|
config.secret_key = ENV['LANGFUSE_SECRET_KEY']
# Optional: for self-hosted instances
config.base_url = ENV.fetch('LANGFUSE_BASE_URL', 'https://cloud.langfuse.com')

# Optional: Enable stale-while-revalidate for best performance
config.cache_backend = :rails # or :memory
config.cache_stale_while_revalidate = true
end
```

Expand Down Expand Up @@ -103,4 +107,4 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.

## License

[MIT](LICENSE)
[MIT](LICENSE)
155 changes: 83 additions & 72 deletions docs/API_REFERENCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,24 @@ Langfuse.configure { |config| ... }

Block receives a configuration object with these properties:

| Property | Type | Required | Default | Description |
|----------|------|----------|---------|-------------|
| `public_key` | String | Yes | - | Langfuse public API key |
| `secret_key` | String | Yes | - | Langfuse secret API key |
| `base_url` | String | No | `"https://cloud.langfuse.com"` | API endpoint |
| `timeout` | Integer | No | `5` | HTTP timeout (seconds) |
| `cache_ttl` | Integer | No | `60` | Prompt cache TTL (seconds) |
| `cache_max_size` | Integer | No | `1000` | Max cached prompts |
| `cache_backend` | Symbol | No | `:memory` | `:memory` or `:rails` |
| `cache_lock_timeout` | Integer | No | `10` | Lock timeout (seconds) |
| `batch_size` | Integer | No | `50` | Score batch size |
| `flush_interval` | Integer | No | `10` | Score flush interval (seconds) |
| `logger` | Logger | No | Auto-detected | Logger instance |
| `tracing_async` | Boolean | No | `true` | ⚠️ Experimental (not implemented) |
| `job_queue` | Symbol | No | `:default` | ⚠️ Experimental (not implemented) |
| Property | Type | Required | Default | Description |
| ------------------------------ | ------- | -------- | ------------------------------ | --------------------------------- |
| `public_key` | String | Yes | - | Langfuse public API key |
| `secret_key` | String | Yes | - | Langfuse secret API key |
| `base_url` | String | No | `"https://cloud.langfuse.com"` | API endpoint |
| `timeout` | Integer | No | `5` | HTTP timeout (seconds) |
| `cache_ttl` | Integer | No | `60` | Prompt cache TTL (seconds) |
| `cache_max_size` | Integer | No | `1000` | Max cached prompts |
| `cache_backend` | Symbol | No | `:memory` | `:memory` or `:rails` |
| `cache_lock_timeout` | Integer | No | `10` | Lock timeout (seconds) |
| `cache_stale_while_revalidate` | Boolean | No | `false` | Enable stale-while-revalidate |
| `cache_stale_ttl` | Integer | No | `60` when SWR is enabled | Stale TTL (seconds) |
| `cache_refresh_threads` | Integer | No | `5` | Background refresh threads |
Comment on lines +43 to +45
Copy link
Member

@NoahFisher NoahFisher Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Configuration Complexity

Concern: This PR adds 3 new configuration options (cache_stale_while_revalidate, cache_stale_ttl, cache_refresh_threads). I'd push back on exposing all of these to users.

Principle: Every configuration option is a decision the user has to make. Good defaults mean fewer decisions.

Suggestion: Consider collapsing to a single option:

Instead of:

config.cache_stale_while_revalidate = true
config.cache_stale_ttl = 300
config.cache_refresh_threads = 5

Just:

config.cache_stale_while_revalidate = true  # Uses sensible defaults internally

Reasoning:

  • cache_stale_ttl defaulting to cache_ttl is already the right choice 99% of the time - why expose it?
  • cache_refresh_threads at 5 is fine for virtually all deployments - this is an implementation detail, not a user concern
  • Users who truly need to tune these can always subclass or we can add them later

The bar for adding config options should be: "Will a significant number of users need to change this from the default?" If not, hardcode it.

Copy link

@benlangfeld benlangfeld Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We absolutely need to set the TTL to infinite. Not exposing the TTL setting won't work for us.

Copy link
Member

@NoahFisher NoahFisher Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what the use case for a non-infinite TTL?

Copy link
Contributor Author

@drborges drborges Dec 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every configuration option is a decision the user has to make. Good defaults mean fewer decisions.

I agree, it certainly makes sense to hide cache_refresh_threads. Exposing cache_stale_ttl would be more of a flexibility in my opinion in case clients have very specific needs.

With that said, if I choose to enable SWR, I would more often than not want an infinite grace period, e.g. cache_stale_ttl = :indefinite for high availability reasons, then use cache_ttl to control how often cache is refreshed.

Based on that, only exposing config.cache_stale_while_revalidate would make sense to me if that defaults cache_stale_ttl to :indefinite as per the proposed implementation.

Does that make sense?

| `batch_size` | Integer | No | `50` | Score batch size |
| `flush_interval` | Integer | No | `10` | Score flush interval (seconds) |
| `logger` | Logger | No | Auto-detected | Logger instance |
| `tracing_async` | Boolean | No | `true` | ⚠️ Experimental (not implemented) |
| `job_queue` | Symbol | No | `:default` | ⚠️ Experimental (not implemented) |

**Example:**

Expand All @@ -54,6 +57,7 @@ Langfuse.configure do |config|
config.secret_key = ENV['LANGFUSE_SECRET_KEY']
config.cache_ttl = 300
config.cache_backend = :rails
config.cache_stale_while_revalidate = true # Serve stale data while refreshing
end
```

Expand Down Expand Up @@ -133,17 +137,18 @@ get_prompt(name, version: nil, label: nil, fallback: nil, type: nil)

**Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `name` | String | Yes | Prompt name |
| `version` | Integer | No | Specific version (mutually exclusive with `label`) |
| `label` | String | No | Version label (e.g., "production") |
| `fallback` | String | No | Fallback template if not found |
| `type` | Symbol | Conditional | `:text` or `:chat` (required if `fallback` provided) |
| Parameter | Type | Required | Description |
| ---------- | ------- | ----------- | ---------------------------------------------------- |
| `name` | String | Yes | Prompt name |
| `version` | Integer | No | Specific version (mutually exclusive with `label`) |
| `label` | String | No | Version label (e.g., "production") |
| `fallback` | String | No | Fallback template if not found |
| `type` | Symbol | Conditional | `:text` or `:chat` (required if `fallback` provided) |

**Returns:** `TextPromptClient` or `ChatPromptClient`

**Raises:**

- `NotFoundError` if prompt doesn't exist (unless `fallback` provided)
- `UnauthorizedError` if credentials invalid
- `ApiError` on network/server errors
Expand Down Expand Up @@ -183,9 +188,9 @@ compile_prompt(name, variables: {}, version: nil, label: nil, fallback: nil, typ

Same as `get_prompt`, plus:

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `variables` | Hash | No | Template variables (symbol or string keys) |
| Parameter | Type | Required | Description |
| ----------- | ---- | -------- | ------------------------------------------ |
| `variables` | Hash | No | Template variables (symbol or string keys) |

**Returns:** String (text prompts) or Array<Hash> (chat prompts)

Expand Down Expand Up @@ -218,10 +223,10 @@ list_prompts(page: 1, limit: 50)

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `page` | Integer | No | `1` | Page number |
| `limit` | Integer | No | `50` | Results per page |
| Parameter | Type | Required | Default | Description |
| --------- | ------- | -------- | ------- | ---------------- |
| `page` | Integer | No | `1` | Page number |
| `limit` | Integer | No | `50` | Results per page |

**Returns:** Array of prompt hashes

Expand Down Expand Up @@ -255,14 +260,14 @@ Returned by `get_prompt` for text prompts.

**Properties:**

| Property | Type | Description |
|----------|------|-------------|
| `name` | String | Prompt name |
| `version` | Integer | Version number |
| `labels` | Array<String> | Version labels |
| `tags` | Array<String> | Tags |
| `config` | Hash | Configuration metadata |
| `prompt` | String | Raw template |
| Property | Type | Description |
| --------- | ------------- | ---------------------- |
| `name` | String | Prompt name |
| `version` | Integer | Version number |
| `labels` | Array<String> | Version labels |
| `tags` | Array<String> | Tags |
| `config` | Hash | Configuration metadata |
| `prompt` | String | Raw template |

**Methods:**

Expand Down Expand Up @@ -326,17 +331,18 @@ observe(name, attributes = {}, as_type: :span) # => BaseObservation

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `name` | String | Yes | - | Operation name |
| `attributes` | Hash | No | `{}` | Initial attributes |
| `as_type` | Symbol | No | `:span` | Observation type |
| Parameter | Type | Required | Default | Description |
| ------------ | ------ | -------- | ------- | ------------------ |
| `name` | String | Yes | - | Operation name |
| `attributes` | Hash | No | `{}` | Initial attributes |
| `as_type` | Symbol | No | `:span` | Observation type |

**Observation Types:**

`:span`, `:generation`, `:event`, `:embedding`, `:agent`, `:tool`, `:chain`, `:retriever`, `:evaluator`, `:guardrail`

**Returns:**

- Block mode: block return value
- Stateful mode: observation instance

Expand Down Expand Up @@ -365,13 +371,13 @@ Returned by `observe` in stateful mode or passed to block.

**Properties:**

| Property | Type | Description |
|----------|------|-------------|
| `id` | String | Observation ID (hex span ID) |
| `trace_id` | String | Trace ID (hex trace ID) |
| `trace_url` | String | URL to Langfuse UI |
| `otel_span` | OpenTelemetry::SDK::Trace::Span | Underlying OTel span |
| `type` | String | Observation type |
| Property | Type | Description |
| ----------- | ------------------------------- | ---------------------------- |
| `id` | String | Observation ID (hex span ID) |
| `trace_id` | String | Trace ID (hex trace ID) |
| `trace_url` | String | URL to Langfuse UI |
| `otel_span` | OpenTelemetry::SDK::Trace::Span | Underlying OTel span |
| `type` | String | Observation type |

**Methods:**

Expand Down Expand Up @@ -519,15 +525,15 @@ create_score(name:, value:, trace_id: nil, observation_id: nil, comment: nil, me

**Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `name` | String | Yes | Score name |
| `value` | Numeric/String/Boolean | Yes | Score value |
| `trace_id` | String | No | Trace ID to score |
| `observation_id` | String | No | Observation ID to score |
| `comment` | String | No | Score comment |
| `metadata` | Hash | No | Additional metadata |
| `data_type` | Symbol | No | `:numeric`, `:boolean`, or `:categorical` |
| Parameter | Type | Required | Description |
| ---------------- | ---------------------- | -------- | ----------------------------------------- |
| `name` | String | Yes | Score name |
| `value` | Numeric/String/Boolean | Yes | Score value |
| `trace_id` | String | No | Trace ID to score |
| `observation_id` | String | No | Observation ID to score |
| `comment` | String | No | Score comment |
| `metadata` | Hash | No | Additional metadata |
| `data_type` | Symbol | No | `:numeric`, `:boolean`, or `:categorical` |

**Note:** Must provide at least one of `trace_id` or `observation_id`.

Expand Down Expand Up @@ -585,9 +591,9 @@ flush_scores(timeout: 30)

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `timeout` | Integer | No | `30` | Flush timeout (seconds) |
| Parameter | Type | Required | Default | Description |
| --------- | ------- | -------- | ------- | ----------------------- |
| `timeout` | Integer | No | `30` | Flush timeout (seconds) |

**Example:**

Expand Down Expand Up @@ -623,14 +629,14 @@ propagate_attributes(user_id: nil, session_id: nil, metadata: nil, version: nil,

**Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `user_id` | String | No | User identifier (≤200 chars) |
| `session_id` | String | No | Session identifier (≤200 chars) |
| `metadata` | Hash<String, String> | No | Metadata hash |
| `version` | String | No | Version (≤200 chars) |
| `tags` | Array<String> | No | Tags array |
| `as_baggage` | Boolean | No | Propagate across services via OTel baggage |
| Parameter | Type | Required | Description |
| ------------ | -------------------- | -------- | ------------------------------------------ |
| `user_id` | String | No | User identifier (≤200 chars) |
| `session_id` | String | No | Session identifier (≤200 chars) |
| `metadata` | Hash<String, String> | No | Metadata hash |
| `version` | String | No | Version (≤200 chars) |
| `tags` | Array<String> | No | Tags array |
| `as_baggage` | Boolean | No | Propagate across services via OTel baggage |

**Example:**

Expand Down Expand Up @@ -717,6 +723,7 @@ All exceptions inherit from `Langfuse::Error < StandardError`.
Configuration validation errors.

**Raised when:**

- Missing `public_key` or `secret_key`
- Invalid configuration values

Expand All @@ -725,6 +732,7 @@ Configuration validation errors.
Base class for API HTTP errors.

**Raised when:**

- Network issues
- Server errors (500, 503)
- Timeouts
Expand All @@ -734,6 +742,7 @@ Base class for API HTTP errors.
Extends `ApiError`. Authentication failures (401).

**Raised when:**

- Invalid API credentials
- Expired keys

Expand All @@ -742,6 +751,7 @@ Extends `ApiError`. Authentication failures (401).
Extends `ApiError`. Resource not found (404).

**Raised when:**

- Prompt doesn't exist
- Invalid version/label

Expand All @@ -750,6 +760,7 @@ Extends `ApiError`. Resource not found (404).
Cache warming operation failed.

**Raised when:**

- One or more prompts failed to warm

See [ERROR_HANDLING.md](ERROR_HANDLING.md) for complete guide.
Expand Down Expand Up @@ -785,9 +796,9 @@ Langfuse.shutdown(timeout: 30)

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `timeout` | Integer | No | `30` | Shutdown timeout (seconds) |
| Parameter | Type | Required | Default | Description |
| --------- | ------- | -------- | ------- | -------------------------- |
| `timeout` | Integer | No | `30` | Shutdown timeout (seconds) |

**Example:**

Expand Down
Loading