feat: add session step replay dataset loader and session-level metrics#743
Open
ajcasagrande wants to merge 2 commits intomainfrom
Open
feat: add session step replay dataset loader and session-level metrics#743ajcasagrande wants to merge 2 commits intomainfrom
ajcasagrande wants to merge 2 commits intomainfrom
Conversation
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@a7eefa2d0c2485c00ff10c1ba7dbf2d0901eb929Recommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@a7eefa2d0c2485c00ff10c1ba7dbf2d0901eb929Last updated for commit: |
ajcasagrande
force-pushed
the
ajc/session-step-replay
branch
from
5336ffb to
f0127f6
Compare
ajcasagrande
force-pushed
the
ajc/conversationp-context-mode
branch
from
87946ec to
59b6e84
Compare
Introduce ConversationContextMode enum (accumulate_all, drop_responses, standalone) to control how prior turns are accumulated in multi-turn conversations. Modes resolve with conversation > dataset default > accumulate_all precedence. Standalone replaces turn_list with only the current turn; drop_responses skips storing assistant responses. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
ajcasagrande
force-pushed
the
ajc/conversationp-context-mode
branch
from
59b6e84 to
27536c2
Compare
Add support for replaying captured agent sessions with candidate prompt selection. Includes SessionReplayDatasetLoader, session metrics processor, deterministic candidate selection via credit-issued random seeds, and completions endpoint multi-turn support. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
ajcasagrande
force-pushed
the
ajc/session-step-replay
branch
from
f0127f6 to
a7eefa2
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
ajcasagrande
force-pushed
the
ajc/conversationp-context-mode
branch
from
27536c2 to
9544238
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add support for replaying captured agent sessions with candidate prompt selection. Includes SessionStepReplayDatasetLoader, session metrics processor, deterministic candidate selection via credit-issued random seeds, and completions endpoint multi-turn support.