Skip to content

Optimize table fetches for explicit attribute lists via entries-search projection#210

Merged
PatrykGala merged 1 commit intomainfrom
pg/optimize-fetch-runs-table
Feb 12, 2026
Merged

Optimize table fetches for explicit attribute lists via entries-search projection#210
PatrykGala merged 1 commit intomainfrom
pg/optimize-fetch-runs-table

Conversation

@PatrykGala
Copy link
Contributor

@PatrykGala PatrykGala commented Feb 6, 2026

MR Summary

  • Added a fast path in fetch_experiments_table and fetch_runs_table for cases where attributes is an exact list.
  • Extended fetch_table() with exact_attribute_names and routed eligible requests through a single projected entries-search flow instead of the split label/definition/value pipeline.
  • Refactored search pagination/setup into reusable helpers (_build_entries_search_params, _fetch_entries_with_projection) and introduced TableSearchEntry parsing for direct table-row construction.
  • Added NEPTUNE_QUERY_ENTRIES_SEARCH_MAX_PROJECTION_ATTRIBUTES (default 1000) to cap projection size and keep fallback behavior for larger requests.
  • Kept backward-compatible fallback path when fast-path conditions are not met.
  • Added/updated unit tests to cover fast-path usage for experiments/runs and fallback behavior around projection limits.

@PatrykGala PatrykGala changed the title feat: optimize fetch runs table Optimize table fetches for explicit attribute lists via entries-search projection Feb 6, 2026
@PatrykGala PatrykGala force-pushed the pg/optimize-fetch-runs-table branch from 4dd75ce to d9d180b Compare February 9, 2026 11:31
Copy link

@maciejpyteloai maciejpyteloai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really good improvement. Left some minor comment, feel free to ignore if you disagree

sort_by_inference_result.emit_warnings()

if exact_attribute_names is not None:
exact_attribute_names_set = set(exact_attribute_names)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would move this to a helper function

)
NEPTUNE_QUERY_ATTRIBUTE_VALUES_BATCH_SIZE = EnvVariable[int]("NEPTUNE_QUERY_ATTRIBUTE_VALUES_BATCH_SIZE", int, 10_000)
NEPTUNE_QUERY_ENTRIES_SEARCH_MAX_PROJECTION_ATTRIBUTES = EnvVariable[int](
"NEPTUNE_QUERY_ENTRIES_SEARCH_MAX_PROJECTION_ATTRIBUTES", int, 1000

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you choose this value?

)


def test_fetch_experiments_table_uses_entries_search_fast_path_for_exact_attribute_list():

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe consider a separate test where fast path returns non-empty result to test new formatting logic (e.g. that sys/id that is in projection is used for building identifier, but not included in values...).

@PatrykGala PatrykGala added this pull request to the merge queue Feb 12, 2026
Merged via the queue into main with commit 5701269 Feb 12, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants