Implement stale while revalidate (SWR) cache strategy

[drborges]

Summary

Add support to Stale-While-Revalidate (SWR) prompt cache strategy based off the Design Document: docs/future-enhancements/STALE_WHILE_REVALIDATE_DESIGN.md

The GIF below shows the new feature in action against a self-hosted production instance of Langfuse using the following config:

Langfuse.configure do |config|
  config.public_key = ENV["LANGFUSE_PUB_KEY"]
  config.secret_key = ENV["LANGFUSE_SECRET_KEY"]
  config.base_url = ENV["LANGFUSE_BASE_URL"]

  config.cache_backend = :rails
  config.cache_ttl = 2 # cache entries are fresh for 2 seconds, after that, they become stale

  # Enable SWR
  config.cache_stale_while_revalidate = true
  config.cache_stale_ttl = 2 # cache stale entries are served for 2 seconds until the expire completely (grace period).
  config.cache_refresh_threads = 5
end

📊 Impact

• Performance: Enables instant responses for 99% of requests by serving stale data while refreshing in background
• Backward Compatibility: Fully backward compatible - SWR is opt-in via configuration (requires :rails backend)
• Dependencies: Requires concurrent-ruby gem for thread pool management

Usage

Configure the client:

Langfuse.configure do |config|
  config.public_key = ENV['LANGFUSE_PUBLIC_KEY']
  config.secret_key = ENV['LANGFUSE_SECRET_KEY']
  
  config.cache_backend = :rails
  config.cache_ttl = 300 # Fresh for 5 minutes
  
  # Enable SWR
  config.cache_stale_while_revalidate = true
  config.cache_stale_ttl = 300 # Grace period: 5 more minutes
  config.cache_refresh_threads = 10 # Background thread pool size
end

Once configured, SWR works transparently:

client = Langfuse.client

# First request - populates cache
prompt = client.get_prompt("greeting") # ~100ms (API call)

# Subsequent requests while fresh
prompt = client.get_prompt("greeting") # ~1ms (cache hit)

# After cache_ttl expires but within grace period
prompt = client.get_prompt("greeting") # ~1ms (stale data + background refresh)

# Background refresh completes, next request gets fresh data
prompt = client.get_prompt("greeting") # ~1ms (fresh cache)

More details on the proposed documentation.

[drborges]

@NoahFisher I still have to deal with some rubocop offenses and so one final review of the test coverage but I could use some feedback on this draft, I also have one open question to double check in the PR description.

[drborges]

@NoahFisher, @kxzk I think this is in a good spot for some review. Let me know if there are any concerns or change suggestions you'd like me to address. Thanks.

[drborges]

My last commit is disabling Rubocop's Metrics/ClassLength for the ApiClient class but it may be interesting to perhaps move parts of the class into mixin modules to keep the file under the desired length. Thoughts @NoahFisher?

[kxzk]

@NoahFisher, @kxzk I think this is in a good spot for some review. Let me know if there are any concerns or change suggestions you'd like me to address. Thanks.

Will try and review today.

[kxzk]

Few comments but overall looks solid

[drborges]

@kxzk thanks for all the feedback. I'll address them next.

[drborges]

@kxzk I think I was able to address all your feedback. This is ready for another round of reviews. Thanks again.

[drborges]

@kxzk, @NoahFisher let me know if this look good and what the next steps would be for us getting this merged to main.

[NoahFisher]

Additional Considerations:

1. Memory Leak Potential with In-Memory Locks

In PromptCache, locks are stored in a hash but only deleted on release_lock.

If a refresh thread crashes between acquire_lock and release_lock (despite the ensure block), the lock is never released. The ensure should handle this, but if the thread is killed externally (e.g., Thread#kill), the lock persists forever. Unlike Redis locks which have TTL expiration, these in-memory locks have no timeout.

Suggestion: Accept this as a known limitation of the in-memory backend, but document it.

2. No Backpressure Signal to Callers

When the thread pool queue fills up and tasks are discarded, callers have no idea. This is fine for SWR (they get stale data), but it means:

• Monitoring is blind to refresh failures
• If the API is down for extended periods, stale data keeps being served with no signal

Question: Should there be a hook or callback for observability? e.g., on_refresh_dropped, on_refresh_failed?

3. Cold Start Behavior

On application boot with an empty cache, the first request for each prompt will be a synchronous API call (cache miss). With SWR enabled, you've documented cache warming, which is good. But worth calling out:

SWR doesn't help cold starts - it only helps after the cache has been populated at least once.

4. Ruby's GVL (Global VM Lock)

Worth noting that Ruby's GVL means these threads aren't truly parallel for CPU work. However, for I/O-bound work (HTTP calls to Langfuse API), threads do release the GVL during I/O waits, so this is fine. Just worth being aware that max_threads: 5 doesn't mean 5x throughput for CPU-bound work.

5. Should SWR be Default-On for Rails Backend?

The PR makes SWR opt-in (cache_stale_while_revalidate = false by default). Given the latency benefits and that the distributed lock prevents thundering herd, is there an argument for making
it default-on for the :rails backend specifically? Users with Redis already have the
infrastructure for distributed locks.

Counter-argument: Surprising behavior changes are bad. Opt-in is safer.

Good work so far, I wouldn't consider the above blocking comments (I feel more strongly about the inline ones), but curious if we should document our thinking for next steps.

[NoahFisher]

On Configuration Complexity

Concern: This PR adds 3 new configuration options (cache_stale_while_revalidate, cache_stale_ttl, cache_refresh_threads). I'd push back on exposing all of these to users.

Principle: Every configuration option is a decision the user has to make. Good defaults mean fewer decisions.

Suggestion: Consider collapsing to a single option:

Instead of:

config.cache_stale_while_revalidate = true
config.cache_stale_ttl = 300
config.cache_refresh_threads = 5

Just:

config.cache_stale_while_revalidate = true  # Uses sensible defaults internally

Reasoning:

• cache_stale_ttl defaulting to cache_ttl is already the right choice 99% of the time - why expose it?
• cache_refresh_threads at 5 is fine for virtually all deployments - this is an implementation detail, not a user concern
• Users who truly need to tune these can always subclass or we can add them later

The bar for adding config options should be: "Will a significant number of users need to change this from the default?" If not, hardcode it.

[benlangfeld]

We absolutely need to set the TTL to infinite. Not exposing the TTL setting won't work for us.

[NoahFisher]

Can you explain what the use case for a non-infinite TTL?

[drborges]

Every configuration option is a decision the user has to make. Good defaults mean fewer decisions.

I agree, it certainly makes sense to hide cache_refresh_threads. Exposing cache_stale_ttl would be more of a flexibility in my opinion in case clients have very specific needs.

With that said, if I choose to enable SWR, I would more often than not want an infinite grace period, e.g. cache_stale_ttl = :indefinite for high availability reasons, then use cache_ttl to control how often cache is refreshed.

Based on that, only exposing config.cache_stale_while_revalidate would make sense to me if that defaults cache_stale_ttl to :indefinite as per the proposed implementation.

Does that make sense?

[drborges]

Thanks for all of the great feedback so far @NoahFisher! I've addressed some of the points you raised and wanted to follow up on a few of the points you brought up above:

Point 2

No Backpressure Signal to Callers
When the thread pool queue fills up and tasks are discarded, callers have no idea. This is fine for SWR (they get stale data), but it means:

Monitoring is blind to refresh failures
If the API is down for extended periods, stale data keeps being served with no signal
Question: Should there be a hook or callback for observability? e.g., on_refresh_dropped, on_refresh_failed?

This is a great point and I think it would be interesting to provide such hooks. Is the following somewhat aligned with what you had in mind?

Langfuse.configure do |config|
  # Hook called whenever a refresh attempt fails for whatever reason
  config.on_cache_refresh_failed do |error, cache_key|
    # handle refresh error accordingly... e.g. publish to Sentry, log something out, etc...
  end
  
  # Hook called whenever a refresh attempt is dropped due to not being able to acquire the lock
  config.on_cache_refresh_dropped do |cache_key|
    # handle refresh drops accordingly...
  end
end

Can you think of any other scenario we'd like to cover?

Also, would it make sense to ship that as another enhancement PR?

Point 3

With SWR enabled, you've documented cache warming, which is good.

Could you point me to that doc entry so I can double check? Cache warming should not be impacted by SWR as you pointed out, so that would likely be a mishap on my part.

Point 5

Should SWR be Default-On for Rails Backend?
The PR makes SWR opt-in (cache_stale_while_revalidate = false by default). Given the latency benefits and that the distributed lock prevents thundering herd, is there an argument for making
it default-on for the :rails backend specifically? Users with Redis already have the infrastructure for distributed locks.

Counter-argument: Surprising behavior changes are bad. Opt-in is safer.

My initial thinking was exactly that "Opt-in seemed safer" and intentional. Now, you do raise a good point, if 99% of the time Rails clients end up turning SWR on, then perhaps it should be the default? I could go either way on this one to be honest.

[drborges]

@NoahFisher, @kxzk do you think we may be able to cut a new release including the work in this PR at some point this week?

Also, one small snag I just ran into is that the project we plan on using langfuse-rb already has the faraday gem pinned to 1.0 due to some old dependency limitations we yet have to deal with.

Now the simplest solution to our problem seems to be having langfuse-rb depend on faraday >= 1.0 as it does not appear to require any specific feature from faraday 2.0. Would this be something we could consider for this PR or a follow-up?

[drborges]

Moving the faraday convo to a new issue: #38

[drborges]

@NoahFisher, @kxzk any thoughts on the last update? Do you think we could move on with the solution as is and address anything that is non-blocking in a follow-up PR perhaps?

[benlangfeld]

@kxzk @NoahFisher Happy new year. Could we please get this merged? It's been going quite a long time.

[kxzk]

@NoahFisher if you want to do one final review.

I created one-off test scripts to validate against live Langfuse instance (all 13 tests pass).

Process

• Create scripts/ directory

• Place both of these scripts in scripts/ dir

• You'll need to create test Langfuse account (can sign up via GitHub)

• Copy env creds to .env (LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST)

• Create Prompts

set -a && source .env && set +a && uv run scripts/setup_swr_prompts.py myrun

• Run Tests

set -a && source .env && set +a && bundle exec ruby scripts/validate_swr.rb myrun

Create Prompts

Python Script

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11,<3.14"
# dependencies = ["langfuse>=3.0.0"]
# ///
"""Create test prompts for SWR validation suite."""

import os
import sys
from langfuse import Langfuse

TEST_RUN_ID = sys.argv[1] if len(sys.argv) > 1 else "default"

PROMPT_SUFFIXES = [
    "fresh-test",
    "stale-test",
    "refresh-test",
    "expired-test",
    "dedup-test",
    "indefinite-test",
    "overflow-test",
    "shutdown-test",
    "concurrent-test",
    "evict-test",
    "isolation-test",
    "simple-test",
]

def main():
    langfuse = Langfuse()

    print(f"Creating test prompts for run: {TEST_RUN_ID}")

    for suffix in PROMPT_SUFFIXES:
        name = f"swr-{TEST_RUN_ID}-{suffix}"
        try:
            langfuse.create_prompt(
                name=name,
                type="text",
                prompt=f"Hello {{{{name}}}}! Prompt: {name}",
                labels=["production", "swr-test"],
            )
            print(f"  Created: {name}")
        except Exception as e:
            if "already exists" in str(e).lower():
                print(f"  Exists:  {name}")
            else:
                print(f"  Error:   {name} - {e}")

    langfuse.flush()
    print(f"\nDone. Created {len(PROMPT_SUFFIXES)} prompts.")
    print(f"Run ID: {TEST_RUN_ID}")

if __name__ == "__main__":
    main()

Run Tests

Ruby Script

#!/usr/bin/env ruby
# frozen_string_literal: true

require "bundler/setup"
require "langfuse"
require "concurrent"
require "securerandom"

class SWRValidator
  attr_reader :results, :test_prompts

  def initialize(test_run_id = nil)
    @results = []
    @test_prompts = []
    @test_run_id = test_run_id || SecureRandom.hex(4)
    @base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
    @public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
    @secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
  end

  def run_all
    puts "\n" + "=" * 70
    puts "STALE-WHILE-REVALIDATE VALIDATION SUITE (LIVE API)"
    puts "=" * 70
    puts "Test run ID: #{@test_run_id}"

    setup_test_prompts

    validate_fresh_state
    validate_stale_state_serves_immediately
    validate_stale_triggers_background_refresh
    validate_expired_state_blocks
    validate_refresh_deduplication
    validate_indefinite_stale_ttl
    validate_thread_pool_queue_overflow
    validate_graceful_shutdown
    validate_config_validation_errors
    validate_concurrent_access_safety
    validate_cache_eviction_under_swr
    validate_version_label_cache_key_isolation
    validate_swr_disabled_falls_back_to_simple_cache

    print_summary
  end

  private

  def setup_test_prompts
    @test_prompts = %w[
      fresh-test stale-test refresh-test expired-test dedup-test
      indefinite-test overflow-test shutdown-test concurrent-test
      evict-test isolation-test simple-test
    ].map { |n| "swr-#{@test_run_id}-#{n}" }

    puts "\nUsing #{@test_prompts.size} prompts with prefix: swr-#{@test_run_id}-*"
    puts "Verifying prompts exist..."

    verify_prompts_exist
  end

  def verify_prompts_exist
    Langfuse.reset!
    Langfuse.configure do |c|
      c.public_key = @public_key
      c.secret_key = @secret_key
      c.base_url = @base_url
    end

    client = Langfuse.client
    first_prompt = @test_prompts.first

    retries = 0
    loop do
      begin
        client.get_prompt(first_prompt)
        puts "Prompts verified."
        break
      rescue Langfuse::NotFoundError
        retries += 1
        if retries > 5
          puts "\nERROR: Prompts not found."
          puts "Create prompts first: uv run scripts/setup_swr_prompts.py #{@test_run_id}"
          exit 1
        end
        puts "Waiting for prompts... (#{retries}/5)"
        sleep 2
      end
    end

    client.shutdown
  end

  def prompt_name(suffix)
    "swr-#{@test_run_id}-#{suffix}"
  end

  def validate_fresh_state
    test("Fresh state serves immediately without API call") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 60
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 120
      end

      client = Langfuse.client
      name = prompt_name("fresh-test")

      start1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      client.get_prompt(name)
      first_elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start1

      start2 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      client.get_prompt(name)
      second_elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start2

      client.shutdown

      assert(second_elapsed < first_elapsed / 2, "Cached call should be much faster (#{ms(second_elapsed)}ms vs #{ms(first_elapsed)}ms)")
      assert(second_elapsed < 0.01, "Cached call should be < 10ms, got #{ms(second_elapsed)}ms")
    end
  end

  def validate_stale_state_serves_immediately
    test("Stale state returns data immediately (not blocked by refresh)") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.1
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 60
      end

      client = Langfuse.client
      name = prompt_name("stale-test")

      first_result = client.get_prompt(name)
      sleep 0.15

      start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      result = client.get_prompt(name)
      elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start

      client.shutdown

      assert(result, "Should return cached prompt")
      assert(result.prompt == first_result.prompt, "Should return same prompt content")
      assert(elapsed < 0.05, "Stale response should be < 50ms, got #{ms(elapsed)}ms")
    end
  end

  def validate_stale_triggers_background_refresh
    test("Stale state triggers background refresh that updates cache") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.1
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 60
      end

      client = Langfuse.client
      cache = client.instance_variable_get(:@api_client).instance_variable_get(:@cache)
      name = prompt_name("refresh-test")
      cache_key = Langfuse::PromptCache.build_key(name)

      client.get_prompt(name)
      sleep 0.15

      internal_cache = cache.instance_variable_get(:@cache)
      entry_before = internal_cache[cache_key]
      fresh_until_before = entry_before&.fresh_until

      client.get_prompt(name)
      sleep 0.5

      entry_after = internal_cache[cache_key]
      fresh_until_after = entry_after&.fresh_until

      client.shutdown

      assert(fresh_until_before, "Should have initial cache entry")
      assert(fresh_until_after, "Should have updated cache entry")
      assert(fresh_until_after > fresh_until_before, "Background refresh should update fresh_until")
    end
  end

  def validate_expired_state_blocks
    test("Expired state blocks until fresh data fetched") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.05
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 0.05
      end

      client = Langfuse.client
      name = prompt_name("expired-test")

      client.get_prompt(name)
      sleep 0.15

      start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      result = client.get_prompt(name)
      elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start

      client.shutdown

      assert(result, "Should return prompt after blocking fetch")
      assert(elapsed >= 0.05, "Should block for API call, got #{ms(elapsed)}ms")
    end
  end

  def validate_refresh_deduplication
    test("Multiple stale requests all return quickly (deduplication working)") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.05
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 60
      end

      client = Langfuse.client
      name = prompt_name("dedup-test")

      client.get_prompt(name)
      sleep 0.1

      response_times = Concurrent::Array.new
      threads = 5.times.map do
        Thread.new do
          start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
          client.get_prompt(name)
          response_times << (Process.clock_gettime(Process::CLOCK_MONOTONIC) - start)
        end
      end
      threads.each(&:join)

      client.shutdown

      max_response = response_times.max
      assert(max_response < 0.2, "All stale requests should return quickly (<200ms), slowest was #{ms(max_response)}ms")
    end
  end

  def validate_indefinite_stale_ttl
    test(":indefinite stale_ttl serves stale data without expiring") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.01
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = :indefinite
      end

      stale_ttl = Langfuse.configuration.normalized_stale_ttl
      assert(stale_ttl >= 31_536_000_000, ":indefinite should normalize to ~1000 years")

      client = Langfuse.client
      name = prompt_name("indefinite-test")

      client.get_prompt(name)
      sleep 0.05

      start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      result = client.get_prompt(name)
      elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start

      client.shutdown

      assert(result, "Should return cached prompt")
      assert(elapsed < 0.02, "Should serve stale immediately, got #{ms(elapsed)}ms")
    end
  end

  def validate_thread_pool_queue_overflow
    test("Thread pool handles queue overflow gracefully") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.01
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 60
        c.cache_refresh_threads = 1
      end

      client = Langfuse.client
      cache = client.instance_variable_get(:@api_client).instance_variable_get(:@cache)
      name = prompt_name("overflow-test")

      client.get_prompt(name)
      sleep 0.02

      error_raised = false
      begin
        30.times do
          cache.fetch_with_stale_while_revalidate("overflow-key-#{SecureRandom.hex(4)}") do
            sleep 0.1
            { data: "test" }
          end
        end
      rescue StandardError => e
        error_raised = true
        puts "    Unexpected error: #{e.message}"
      end

      client.shutdown

      assert(!error_raised, "Should not raise error on queue overflow")
    end
  end

  def validate_graceful_shutdown
    test("Shutdown completes gracefully with pending refreshes") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.01
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 60
      end

      client = Langfuse.client
      name = prompt_name("shutdown-test")

      client.get_prompt(name)
      sleep 0.02
      client.get_prompt(name)

      start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      client.shutdown
      elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start

      assert(elapsed < 6, "Shutdown should complete within timeout (got #{ms(elapsed)}ms)")
    end
  end

  def validate_config_validation_errors
    test("Configuration validation catches invalid SWR configs") do
      errors_caught = 0

      begin
        Langfuse.reset!
        Langfuse.configure do |c|
          c.public_key = "pk-test"
          c.secret_key = "sk-test"
          c.cache_stale_while_revalidate = true
          c.cache_stale_ttl = nil
        end
        Langfuse.configuration.validate!
      rescue Langfuse::ConfigurationError => e
        errors_caught += 1 if e.message.include?("cache_stale_ttl")
      end

      begin
        Langfuse.reset!
        Langfuse.configure do |c|
          c.public_key = "pk-test"
          c.secret_key = "sk-test"
          c.cache_stale_while_revalidate = true
          c.cache_stale_ttl = -5
        end
        Langfuse.configuration.validate!
      rescue Langfuse::ConfigurationError
        errors_caught += 1
      end

      begin
        Langfuse.reset!
        Langfuse.configure do |c|
          c.public_key = "pk-test"
          c.secret_key = "sk-test"
          c.cache_refresh_threads = 0
        end
        Langfuse.configuration.validate!
      rescue Langfuse::ConfigurationError
        errors_caught += 1
      end

      Langfuse.reset!

      assert(errors_caught == 3, "Should catch 3 validation errors, got #{errors_caught}")
    end
  end

  def validate_concurrent_access_safety
    test("Concurrent access is thread-safe") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 0.05
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 60
        c.cache_max_size = 100
      end

      client = Langfuse.client
      name = prompt_name("concurrent-test")
      errors = Concurrent::AtomicFixnum.new(0)
      success = Concurrent::AtomicFixnum.new(0)

      threads = 10.times.map do
        Thread.new do
          5.times do
            begin
              result = client.get_prompt(name)
              result ? success.increment : errors.increment
            rescue StandardError => e
              errors.increment
              puts "    Thread error: #{e.message}"
            end
          end
        end
      end

      threads.each(&:join)
      client.shutdown

      total = success.value + errors.value
      assert(errors.value == 0, "Should have 0 errors, got #{errors.value}/#{total}")
    end
  end

  def validate_cache_eviction_under_swr
    test("Cache eviction works correctly under SWR") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 60
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 120
        c.cache_max_size = 3
      end

      client = Langfuse.client
      cache = client.instance_variable_get(:@api_client).instance_variable_get(:@cache)

      cache.set("evict-1", { data: 1 })
      sleep 0.01
      cache.set("evict-2", { data: 2 })
      sleep 0.01
      cache.set("evict-3", { data: 3 })

      cache.get("evict-1")
      cache.set("evict-4", { data: 4 })

      cache_size = cache.size

      client.shutdown

      assert(cache_size <= 3, "Cache should respect max_size (got #{cache_size})")
    end
  end

  def validate_version_label_cache_key_isolation
    test("Version and label create isolated cache keys") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 60
        c.cache_stale_while_revalidate = true
        c.cache_stale_ttl = 120
      end

      client = Langfuse.client
      cache = client.instance_variable_get(:@api_client).instance_variable_get(:@cache)
      name = prompt_name("isolation-test")

      client.get_prompt(name)
      client.get_prompt(name, version: 1)
      client.get_prompt(name, label: "swr-test")

      internal_cache = cache.instance_variable_get(:@cache)

      latest_key = Langfuse::PromptCache.build_key(name)
      version_key = Langfuse::PromptCache.build_key(name, version: 1)
      label_key = Langfuse::PromptCache.build_key(name, label: "swr-test")

      has_latest = internal_cache.key?(latest_key)
      has_version = internal_cache.key?(version_key)
      has_label = internal_cache.key?(label_key)

      error_raised = false
      begin
        client.get_prompt(name, version: 1, label: "swr-test")
      rescue ArgumentError => e
        error_raised = e.message.include?("Cannot specify both")
      end

      client.shutdown

      assert(has_latest, "Should have cache key for latest (#{latest_key})")
      assert(has_version, "Should have cache key for version (#{version_key})")
      assert(has_label, "Should have cache key for label (#{label_key})")
      assert(error_raised, "Should raise ArgumentError when both version and label specified")
    end
  end

  def validate_swr_disabled_falls_back_to_simple_cache
    test("SWR disabled uses simple cache (no thread pool)") do
      Langfuse.reset!
      Langfuse.configure do |c|
        c.public_key = ENV.fetch("LANGFUSE_PUBLIC_KEY")
        c.secret_key = ENV.fetch("LANGFUSE_SECRET_KEY")
        c.base_url = ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
        c.cache_backend = :memory
        c.cache_ttl = 60
        c.cache_stale_while_revalidate = false
      end

      client = Langfuse.client
      cache = client.instance_variable_get(:@api_client).instance_variable_get(:@cache)
      name = prompt_name("simple-test")

      has_thread_pool = cache.instance_variable_get(:@thread_pool)&.running?

      start1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      client.get_prompt(name)
      first_elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start1

      start2 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
      client.get_prompt(name)
      second_elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start2

      client.shutdown

      assert(!has_thread_pool, "Thread pool should not be running when SWR disabled")
      assert(second_elapsed < first_elapsed / 2, "Simple cache should still work")
    end
  end

  def ms(seconds)
    (seconds * 1000).round(1)
  end

  def test(name)
    print "\n#{name}... "
    begin
      yield
      @results << { name: name, passed: true }
      puts "✓ PASS"
    rescue AssertionError => e
      @results << { name: name, passed: false, error: e.message }
      puts "✗ FAIL: #{e.message}"
    rescue StandardError => e
      @results << { name: name, passed: false, error: "#{e.class}: #{e.message}" }
      puts "✗ ERROR: #{e.class}: #{e.message}"
      puts e.backtrace.first(3).map { |l| "    #{l}" }.join("\n")
    end
  end

  def assert(condition, message = "Assertion failed")
    raise AssertionError, message unless condition
  end

  def print_summary
    puts "\n" + "=" * 70
    puts "SUMMARY"
    puts "=" * 70

    passed = @results.count { |r| r[:passed] }
    failed = @results.count { |r| !r[:passed] }

    puts "Passed: #{passed}"
    puts "Failed: #{failed}"

    if failed > 0
      puts "\nFailures:"
      @results.reject { |r| r[:passed] }.each do |r|
        puts "  - #{r[:name]}: #{r[:error]}"
      end
    end

    puts "\n" + (failed.zero? ? "ALL TESTS PASSED" : "SOME TESTS FAILED")
    puts "=" * 70 + "\n"

    exit(failed.zero? ? 0 : 1)
  end

  class AssertionError < StandardError; end
end

test_run_id = ARGV[0]
if test_run_id.nil? || test_run_id.empty?
  puts "Usage: ruby scripts/validate_swr.rb <test_run_id>"
  puts "First create prompts: uv run scripts/setup_swr_prompts.py <test_run_id>"
  exit 1
end

SWRValidator.new(test_run_id).run_all

[drborges]

@NoahFisher any ETA on when we might expect this to be merged/released?