This appears to break the non-scenario pathway. And I'm running into issues running the built-in scenario.

Regarding the issues with the built-in profile:
The built-in scenario rag appears to use the old name for the profile, rate_type. The result of this is the command line still requesting that I specify the profile type despite it being included in the profile.

There is also an error trying to use the rag profile:

  File "/Users/joconnel/Documents/projects/ai/guidellm/src/guidellm/dataset/creator.py", line 94, in create
    dataset = cls.handle_create(
        data, data_args, processor, processor_args, random_seed
    )
  File "/Users/joconnel/Documents/projects/ai/guidellm/src/guidellm/dataset/in_memory.py", line 42, in handle_create
    data_dict = cls.format_data_dict(data)
  File "/Users/joconnel/Documents/projects/ai/guidellm/src/guidellm/dataset/in_memory.py", line 64, in format_data_dict
    raise TypeError(
    ...<2 lines>...
    )
TypeError: Unsupported data format. Expected Dict[str, Iterable[Any]], got <class 'dict'>

Are the built in scenarios up to date?

Regarding the issue with the non-scenario pathway, it's sending a list of integers to the profiles that need an integer, causing a validation failure:
guidellm benchmark run --target http://localhost:8000 --data "prompt_tokens=256,output_tokens=128" --rate-type constant --rate 2 --max-seconds 10

TypeError: Unable to apply constraint 'gt' to supplied value [2.0]

It's possible that some of this is user error, but if that's the case the input validation is not doing a good job.

Looks good to me. With the fix I pushed it works for me, though I haven't had the time to let a full run complete.
I have one comment, but I won't block on that. I am also willing to do that work for you if you like the idea.

Pull Request Overview

This PR refactors the scenario system in GuideLLM to enable scenarios functionality. The changes transition from a nested JSON structure to a flat string-based data format and introduce comprehensive scenario support across the codebase.

• Simplifies scenario JSON configuration files to use flat string data format instead of nested objects
• Adds comprehensive scenario support with validation, file loading, and decorator functionality
• Integrates scenarios into the CLI with builtin scenario selection and file-based configuration

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}