Add failure injection mode to simulator

[smarunich]

The initial PR for a discussion on failure mode as doing some tests locally with client and gateway testing, wdyt in general from the mode standpoint? how it should be introduced? open to the feedback and work on it.

Introduces a 'failure' mode to the simulator, allowing random injection of OpenAI API-compatible error responses for testing error handling. Adds configuration options for failure injection rate and specific failure types, implements error response logic, and updates documentation and tests to cover the new functionality.

This pull request adds a new "failure" simulation mode to the LLM-D inference simulator, enabling randomized injection of OpenAI-compatible API error responses for enhanced error-handling test scenarios. It introduces configuration options for controlling the failure injection rate and specifying which error types to inject, along with robust validation and documentation updates. The implementation includes a new error response system, supporting unit tests, and integration into the main completion request handler.

Simulator functionality expansion:

• Added a new failure simulation mode, which randomly injects OpenAI API-compatible error responses for testing purposes. This mode is selectable via the mode parameter and is documented in README.md. [1] [2]
• Introduced new configuration parameters: failure-injection-rate (controls probability of error injection) and failure-types (specifies which error types to inject), with validation for allowed values and documentation updates. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Failure injection implementation:

• Added pkg/common/failures.go, defining error types, failure injection logic, and random selection of error responses based on configuration.
• Integrated failure injection into the main completion handler in simulator.go, returning error responses when triggered. Added a dedicated method for sending structured error responses. [1] [2]

Testing and validation:

• Added unit tests in pkg/common/failures_test.go to verify failure injection logic and error response generation.

Other improvements:

• Updated error response handling in simulator.go to use a consistent error response structure.
• Minor fixes to test code for compatibility with new OpenAI Go SDK conventions. [1] [2] [3]

[irar2]

@smarunich Thank you very much for the PR, failure generation support is a part of our roadmap, thanks for picking this up!

Some more or less general comments, so I am writing them here and not in the code:

We think that failure generation should not be a mode. (We have to update the explanation of mode, it's misleading, it should be a completion response generation mode, and not the mode of the simulator).
failure-injection-rate = 0 will mean no failure injection
failure types - please define constants for the possible values

Please move failures.go to llm-d-inference-sim package.

Please don’t call rand.Seed() - use common.RandomInt which is protected by mutex and allows reproducible behavior.

"Rate limit reached for %s in organization org-xxx on requests per min (RPM): Limit 3, Used 3, Requested 1." and "The model ‘%s-nonexistent’ does not exist" can be defined as constants and reused.

We may only have one function sendCompletionError instead of two (sendCompletionError and sendFailureResponse) which gets common.FailureSpec and boolean which defines whether this is an injected failure for logging.

In ErrorResponse object field is missing. In the documentation that we found, it should be present. Could you please share a link to the relevant documentation?

[smarunich]

@irar2 thank you a lot for your feedback, please take an initial look and let me know what you think i have incorporated your feedback, also adding this reference vllm-project/vllm#12886 to the ErrorResponse conversation... i am not contributing to the projects frequently, but do have a huge respect to what you are doing with this project so attempted to do my best, open for any modifications, etc.

p.s. I am using the build with Envoy AI Gateway Demos for provider failover demo https://github.com/smarunich/envoy-ai-gateway-demos/tree/main/demos/03-provider-fallback also do have github actions run to showcase the run: https://github.com/smarunich/envoy-ai-gateway-demos/actions/runs/16978406193/job/48132952395?pr=3

[irar2]

Please update according to the general comment

[irar2]

@smarunich Thanks for the links and the changes. We really appreciate your contribution.

We did some more digging and found out that even though the hierarchical error structure is already in the vLLM current code, but not in the latest release. So, we need to stay with the current structure for now, and upgrade it later. I created issue #135 for this.

In addition, code field is an int in vLLM, please don't change that, and it's the same code as the HTTP code.

I don't think there is a need in your FailureSpec struct, you can simply use the existing CompletionError.
Your ErrorTypes don't match the openai types.
It would be nice to have a constructor that will set the error type according to the code:

func NewCompletionError(message string, code int, param *string) CompletionError {
	errorType := ""
	switch code {
	case 400:
		errorType = "BadRequestError"
	case 401:
		errorType = "AuthenticationError"
	case 403:
		errorType = "PermissionDeniedError"
	case 404:
		errorType = "NotFoundError"
	case 422:
		errorType = "UnprocessableEntityError"
	case 429:
		errorType = "RateLimitError"
	default:
		if code >= 500 {
			errorType = "InternalServerError"
		} else {
			errorType = "APIConnectionError"
		}
	}
	return CompletionError{
		Object:  "error",
		Message: message,
		Code:    code,
		Type:    errorType,
		Param:   param,
	}
}

and call this constructor to create an error to pass to sendCompletionError and to create a map of predefinedFailures.

Please run make lint, our automatic testing fails because of the lint errors.

Please see additional comments inside the code.

[irar2]

@smarunich We are planning a release soon, and would like to include this feature in it. Will you have time to continue with the PR in the next couple of days? If your schedule doesn't allow this, please let us know and we will continue with this PR.

[smarunich]

@smarunich We are planning a release soon, and would like to include this feature in it. Will you have time to continue with the PR in the next couple of days? If your schedule doesn't allow this, please let us know and we will continue with this PR.

@irar2 I was off last week, let me resume by today and update the thread.

[smarunich]

@smarunich We are planning a release soon, and would like to include this feature in it. Will you have time to continue with the PR in the next couple of days? If your schedule doesn't allow this, please let us know and we will continue with this PR.

@irar2 please feel free to take it forward, I have done few updates, but running out of cycles for this week as I have run into the conflicting priorities, I won't have cycle to work on this until very beginning of the next week, meanwhile I might miss on quality... don't want to delay you folks as truly appreciate your efforts, I would really appreciate if you can take this forward to completion!

[irar2]

@smarunich We are planning a release soon, and would like to include this feature in it. Will you have time to continue with the PR in the next couple of days? If your schedule doesn't allow this, please let us know and we will continue with this PR.

@irar2 please feel free to take it forward, I have done few updates, but running out of cycles for this week as I have run into the conflicting priorities, I won't have cycle to work on this until very beginning of the next week, meanwhile I might miss on quality... don't want to delay you folks as truly appreciate your efforts, I would really appreciate if you can take this forward to completion!

@smarunich Thanks a lot! We will take it from here then.

[irar2]

fixes #135

[mayabar]

please add json annotation

[mayabar]

move this test to failure_test please

[mayabar]

Hi Sergey @smarunich
Can you please fix the sing off problem , so we will be able to merge this PR
Ira is going to fix requested changes

[irar2]

@smarunich We added your signature to the extended comments of the squashed merge that we did.
Thank you very much for your contribution, it is much appreciated.

[smarunich]

@smarunich We added your signature to the extended comments of the squashed merge that we did. Thank you very much for your contribution, it is much appreciated.

thank you @irar2 @mayabar for taking this forward to completion, truly appreciated!