Upgrade-to-lls-0.4.z

[dmaniloff]

Summary

• Use llama-stack 0.4.x and the corresponding client APIs.
• Continue to use deprecated registration endpoints for now given that pre-registration via run.yaml seems broken
• Add new demo using HF datasets
• Update tests

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Upgrade project to llama-stack 0.4.x and the new llama-stack-client API, updating dataset/benchmark client namespaces, demos, tests, CI matrix, and dependencies accordingly.

Sequence diagram for updated dataset and benchmark registration via llama-stack-client 0.4.x

sequenceDiagram
    actor User
    participant BasicDemoNotebook as BasicDemo_notebook
    participant LlamaStackClient as LlamaStackClient
    participant BetaNamespace as beta
    participant AlphaNamespace as alpha
    participant DatasetAPI as datasets
    participant BenchmarkAPI as benchmarks
    participant LlamaStackService as LlamaStack_server

    User->>BasicDemoNotebook: Run demo cells
    BasicDemoNotebook->>LlamaStackClient: init(base_url)
    BasicDemoNotebook->>LlamaStackClient: models.list()
    LlamaStackClient-->>BasicDemoNotebook: available_models

    Note over BasicDemoNotebook,LlamaStackClient: Dataset registration now uses client.beta.datasets

    BasicDemoNotebook->>BetaNamespace: client.beta
    BetaNamespace->>DatasetAPI: register(dataset_id, purpose, source, metadata)
    DatasetAPI->>LlamaStackService: POST /beta/datasets/register
    LlamaStackService-->>DatasetAPI: dataset_response
    DatasetAPI-->>BasicDemoNotebook: dataset_response

    Note over BasicDemoNotebook,LlamaStackClient: Benchmark registration now uses client.alpha.benchmarks

    BasicDemoNotebook->>AlphaNamespace: client.alpha
    AlphaNamespace->>BenchmarkAPI: register(benchmark_id, dataset_id, scoring_functions, provider_id)
    BenchmarkAPI->>LlamaStackService: POST /alpha/benchmarks/register
    LlamaStackService-->>BenchmarkAPI: benchmark_response
    BenchmarkAPI-->>BasicDemoNotebook: benchmark_response

    Note over BasicDemoNotebook,LlamaStackClient: Dataset cleanup now uses beta namespace

    BasicDemoNotebook->>BetaNamespace: datasets.unregister(dataset_id)
    BetaNamespace->>DatasetAPI: unregister(dataset_id)
    DatasetAPI->>LlamaStackService: DELETE /beta/datasets/{dataset_id}
    LlamaStackService-->>DatasetAPI: success
    DatasetAPI-->>BasicDemoNotebook: success

Loading

Sequence diagram for Kubeflow component using beta.datasets.retrieve

sequenceDiagram
    participant KubeflowPipeline as Kubeflow_pipeline
    participant RetrieveComponent as retrieve_data_from_llama_stack
    participant LlamaStackClient as LlamaStackClient
    participant BetaNamespace as beta
    participant DatasetAPI as datasets
    participant LlamaStackService as LlamaStack_server
    participant OutputArtifact as output_dataset

    KubeflowPipeline->>RetrieveComponent: invoke(llama_stack_base_url, dataset_id, output_dataset)
    RetrieveComponent->>LlamaStackClient: init(base_url=llama_stack_base_url)

    Note over RetrieveComponent,LlamaStackClient: Dataset retrieval now uses client.beta.datasets

    RetrieveComponent->>BetaNamespace: client.beta
    BetaNamespace->>DatasetAPI: retrieve(dataset_id)
    DatasetAPI->>LlamaStackService: GET /beta/datasets/{dataset_id}
    LlamaStackService-->>DatasetAPI: dataset(source.rows)
    DatasetAPI-->>RetrieveComponent: dataset

    RetrieveComponent->>RetrieveComponent: df = DataFrame(dataset.source.rows)
    RetrieveComponent->>OutputArtifact: df.to_json(path, records, lines=True)
    OutputArtifact-->>KubeflowPipeline: JSON dataset artifact

Loading

Class diagram for updated llama-stack-client 0.4.x namespace usage

classDiagram
    class LlamaStackClient {
        +str base_url
        +ModelsNamespace models
        +BetaNamespace beta
        +AlphaNamespace alpha
        +__init__(base_url)
    }

    class ModelsNamespace {
        +list list()
    }

    class BetaNamespace {
        +DatasetsNamespace datasets
    }

    class AlphaNamespace {
        +BenchmarksNamespace benchmarks
    }

    class DatasetsNamespace {
        +register(dataset_id, purpose, source, metadata)
        +unregister(dataset_id)
        +retrieve(dataset_id)
    }

    class BenchmarksNamespace {
        +register(benchmark_id, dataset_id, scoring_functions, provider_id)
    }

    LlamaStackClient --> ModelsNamespace : has
    LlamaStackClient --> BetaNamespace : has
    LlamaStackClient --> AlphaNamespace : has
    BetaNamespace --> DatasetsNamespace : has
    AlphaNamespace --> BenchmarksNamespace : has

Loading

File-Level Changes

Change	Details	Files
Update client usage to new beta/alpha namespaces for datasets and benchmarks in demos, tests, and Kubeflow integration code.	Switch dataset registration/unregistration/list/retrieve calls to use client.beta.datasets. Switch benchmark registration/list calls to use client.alpha.benchmarks. Adjust inline evaluation and Kubeflow tests to use the new client namespaces Update config comments to reference the new dataset registration API	demos/basic_demo.ipynb tests/test_inline_evaluation.py src/llama_stack_provider_ragas/config.py src/llama_stack_provider_ragas/remote/kubeflow/components.py tests/test_kubeflow_integration.py
Align dependencies and CI matrix with llama-stack 0.4.x and the separate llama-stack-client package.	Bump llama-stack dependency to >=0.4.0 Add explicit llama-stack-client>=0.4.0 dependency Simplify CI matrix to test only main and 0.4.0 versions of llama-stack	pyproject.toml .github/workflows/ci.yml uv.lock
Make the basic demo less strict about available model types and more diagnostic.	Replace assertions about available model types with pprint of the available models list	demos/basic_demo.ipynb

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've left some high level feedback:

• In basic_demo.ipynb, replacing the simple model-type assertions with just pprint(available_models) removes a quick sanity check; consider keeping a minimal validation (e.g., assert at least one LLM model exists) alongside the debug print to preserve early failure behavior for users following the demo.
• Now that datasets and benchmarks are accessed under beta/alpha namespaces in several places, it would be good to double-check that all usages across the repository (including any less-visible scripts or utilities) are consistently updated to avoid mixing legacy and new client APIs.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In `basic_demo.ipynb`, replacing the simple model-type assertions with just `pprint(available_models)` removes a quick sanity check; consider keeping a minimal validation (e.g., assert at least one LLM model exists) alongside the debug print to preserve early failure behavior for users following the demo.
- Now that datasets and benchmarks are accessed under `beta`/`alpha` namespaces in several places, it would be good to double-check that all usages across the repository (including any less-visible scripts or utilities) are consistently updated to avoid mixing legacy and new client APIs.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}