feat: Introduce pluggable queue framework (#1138)

This commit introduces a new pluggable framework for request queuing within the EPP Flow Control layer. This change establishes the core interfaces and initial implementations needed for sophisticated request management, prioritization, and fairness.

The key components of this framework are:

- **`framework.SafeQueue` Interface**: A new contract for concurrent-safe queue implementations. It defines a standard set of behaviors for adding, removing, peeking, and managing items, ensuring that all queue plugins are interchangeable.

- **Queue Plugin Implementations**:
  - **`listqueue`**: A simple, efficient FIFO queue based on `container/list`. Ideal for basic, fair queuing workloads.
  - **`maxminheap`**: A priority queue based on a max-min heap, allowing for O(1) access to both the highest and lowest priority items. This is suitable for advanced policies that require configurable ordering.

- **Plugin Registration**: A factory pattern (`queue.MustRegisterQueue`) allows new queue implementations to be discovered and registered at runtime, making the system extensible.

- **Comprehensive Testing**:
  - A new conformance test suite (`TestQueueConformance`) ensures that all registered queue plugins strictly adhere to the `SafeQueue` contract, covering lifecycle, ordering, edge cases, and concurrency.
  - A centralized benchmark suite (`BenchmarkQueues`) provides a fair, apples-to-apples performance comparison of all queue implementations across various workload patterns.

- **Core Type Refinements**: The `types` package has been updated to support this new framework, including a refined `QueueItemAccessor` interface and a new `QueueItemHandle` for opaque, safe item manipulation.

This framework decouples the core flow control logic from the specific queuing disciplines, enabling future work on advanced dispatch and displacement policies.

Author: LukeAVanDrie

pkg/epp/flowcontrol/framework/doc.go

@@ -0,0 +1,27 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package framework defines the core plugin interfaces for extending the `controller.FlowController`.
+//
+// It establishes the contracts that custom logic, such as queueing disciplines and dispatching policies, must adhere
+// to. By building on these interfaces, the Flow Control system can be extended and customized without modifying the
+// core controller logic.
+//
+// The primary interfaces defined here are:
+//   - `SafeQueue`: The contract for concurrent-safe queue implementations.
+//   - `ItemComparator`: The contract for policy-driven logic that defines the relative priority of items within a
+//     queue.
+package framework

pkg/epp/flowcontrol/framework/errors.go

@@ -0,0 +1,44 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package framework
+
+import (
+	"errors"
+)
+
+// `SafeQueue` Errors
+//
+// These errors relate to operations directly on a `SafeQueue` implementation. They are returned by `SafeQueue` methods
+// and might be handled or wrapped by the `ports.FlowRegistry`'s `ports.ManagedQueue` or the
+// `controller.FlowController`.
+var (
+	// ErrNilQueueItem indicates that a nil `types.QueueItemAccessor` was passed to `SafeQueue.Add()`.
+	ErrNilQueueItem = errors.New("queue item cannot be nil")
+
+	// ErrQueueEmpty indicates an attempt to perform an operation on an empty `SafeQueue` that requires one or more items
+	// (e.g., calling `SafeQueue.PeekHead()`).
+	ErrQueueEmpty = errors.New("queue is empty")
+
+	// ErrInvalidQueueItemHandle indicates that a `types.QueueItemHandle` provided to a `SafeQueue` operation (e.g.,
+	// `SafeQueue.Remove()`) is not valid for that queue, has been invalidated, or does not correspond to an actual item
+	// in the queue.
+	ErrInvalidQueueItemHandle = errors.New("invalid queue item handle")
+
+	// ErrQueueItemNotFound indicates that a `SafeQueue.Remove(handle)` operation did not find an item matching the
+	// provided, valid `types.QueueItemHandle`. This can occur if the item was removed by a concurrent operation.
+	ErrQueueItemNotFound = errors.New("queue item not found for the given handle")
+)

pkg/epp/flowcontrol/framework/mocks/mocks.go

@@ -0,0 +1,32 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package mocks
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/framework"
+)
+
+// MockItemComparator provides a mock implementation of the `framework.ItemComparator` interface.
+type MockItemComparator struct {
+	FuncV      framework.ItemComparatorFunc
+	ScoreTypeV string
+}
+
+func (m *MockItemComparator) Func() framework.ItemComparatorFunc { return m.FuncV }
+func (m *MockItemComparator) ScoreType() string                  { return m.ScoreTypeV }
+
+var _ framework.ItemComparator = &MockItemComparator{}

pkg/epp/flowcontrol/framework/plugins/queue/README.md

@@ -0,0 +1,114 @@
+# Flow Controller Queue Plugins (`plugins/queue/`)
+
+This directory contains concrete implementations of the `framework.SafeQueue` interface. This contract defines core,
+self-contained queue data structures used by the `controller.FlowController`.
+
+## Overview
+
+The `controller.FlowController` manages requests by organizing them into queues. Each logical "flow" (e.g., a specific
+model or workload) within a given priority band has its own `ports.ManagedQueue` instance, which wraps a
+`framework.SafeQueue`. This design allows the `controller.FlowController` to apply policies at both the inter-flow
+(across different flows) and intra-flow (within a single flow's queue) levels.
+
+The `framework.SafeQueue` interface abstracts the underlying data structure and its ordering logic. This pluggable
+design allows for:
+
+- **Different Queuing Disciplines**: A basic FIFO queue ([`listqueue`](./listqueue/)) is provided, but other disciplines
+  like priority queues ([`maxminheap`](./maxminheap/)) can be used for more complex ordering requirements.
+- **Specialized Capabilities**: Policies can declare `RequiredQueueCapabilities()` (e.g., `framework.CapabilityFIFO` or
+  `framework.CapabilityPriorityConfigurable`). The `ports.FlowRegistry` pairs the policy with a queue that provides the
+  necessary capabilities.
+- **Performance Optimization**: Different queue implementations offer varying performance characteristics, which can be
+  compared using the centralized benchmark suite to select the best fit for a given workload.
+
+## Contributing a New `SafeQueue` Implementation
+
+To contribute a new queue implementation, follow these steps:
+
+1.  **Define Your Implementation**
+    - Create a new Go package in a subdirectory (e.g., `mycustomqueue/`).
+    - Implement the `framework.SafeQueue` and `types.QueueItemHandle` interfaces.
+    - Ensure all methods of `framework.SafeQueue` are goroutine-safe, typically by using a `sync.Mutex` or
+      `sync.RWMutex`.
+    - If your queue declares `framework.CapabilityPriorityConfigurable`, it MUST use the `framework.ItemComparator`
+      passed to its constructor for all internal ordering logic.
+
+2. **Register Your Queue**
+    - In an `init()` function within your queue's Go file, call `queue.MustRegisterQueue()` with a unique name and a
+      constructor function that matches the `queue.QueueConstructor` signature.
+
+3.  **Add to the Conformance Test**
+    - Add a blank import for your new package to [`conformance_test.go`](./conformance_test.go). Your queue will then be
+      automatically included in the conformance suite, which validates the `SafeQueue` contract.
+
+4.  **Documentation**
+    - Add GoDoc comments to your new queue type, explaining its behavior, capabilities, and any trade-offs.
+
+5.  **Benchmarking**
+    - You do not need to write custom benchmarks. The centralized suite in [`benchmark_test.go`](./benchmark_test.go)
+      automatically includes any new queue implementation after it is registered. This ensures all queues are compared
+      fairly under the same conditions.
+
+## Benchmarking Strategy and Results
+
+A centralized benchmark suite runs against all registered `SafeQueue` implementations to provide a consistent
+performance comparison. To run the benchmarks, use the following command:
+
+```sh
+go test -bench=. -benchmem ./pkg/epp/flowcontrol/framework/plugins/queue/...
+```
+
+### Benchmark Scenarios
+
+The suite includes the following scenarios:
+
+- **`AddRemove`**: Measures throughput of tightly coupled `Add` and `Remove` operations under high parallelism. This
+  tests the raw overhead of the data structure and its locking mechanism for simple, transactional workloads.
+- **`AddPeekRemove`**: Measures performance of a sequential `Add` -> `PeekHead` -> `Remove` loop. This simulates a
+  common consumer pattern where a single worker inspects an item before processing it.
+- **`BulkAddThenBulkRemove`**: Tests performance of adding a large batch of items and then removing them all. This can
+  reveal how the data structure's performance changes as it grows and shrinks under load.
+- **`HighContention`**: Simulates a realistic workload with multiple concurrent producers (adding items) and consumers
+  (peeking and removing items) operating on the same queue.
+
+### Latest Results
+
+*Last Updated: 2025-07-10*
+*(CPU: AMD EPYC 7B12)*
+
+| Benchmark                   | Implementation | Iterations | ns/op   | B/op  | allocs/op |
+| --------------------------- | -------------- | ---------- | ------- | ----- | --------- |
+| **AddRemove**               | `ListQueue`    | 1,889,844  | 609.0   | 224   | 5         |
+|                             | `MaxMinHeap`   | 1,660,987  | 696.7   | 184   | 4         |
+| **AddPeekRemove**           | `ListQueue`    | 3,884,938  | 298.0   | 224   | 5         |
+|                             | `MaxMinHeap`   | 1,857,448  | 615.9   | 184   | 4         |
+| **AddPeekTailRemove**       | `ListQueue`    | 3,576,487  | 308.4   | 224   | 5         |
+|                             | `MaxMinHeap`   | 2,113,134  | 535.3   | 184   | 4         |
+| **BulkAddThenBulkRemove**   | `ListQueue`    | 24,032     | 49,861  | 24801 | 698       |
+|                             | `MaxMinHeap`   | 10,000     | 108,868 | 20787 | 597       |
+| **HighContention**          | `ListQueue`    | 484,574    | 2,328   | 896   | 20        |
+|                             | `MaxMinHeap`   | 84,806     | 18,679  | 783   | 16        |
+
+### Interpretation of Results
+
+The benchmark results highlight the trade-offs between the different queue implementations based on their underlying
+data structures:
+
+- **`ListQueue`**: As a linked list, it excels in scenarios involving frequent additions or removals from either end of
+  the queue (`AddPeekRemove`, `AddPeekTailRemove`), which are O(1) operations. Its performance is less competitive in
+  high-contention and bulk scenarios, which reflects the necessary per-item memory allocation and pointer manipulation
+  overhead.
+- **`MaxMinHeap`**: As a slice-based heap, it has a lower allocation overhead per operation, making it efficient for
+  high-throughput `AddRemove` cycles. Peeking and removing items involves maintaining the heap property, which has an
+  O(log n) cost, making individual peek operations slower than `ListQueue`.
+
+**Choosing a Queue:**
+
+The data suggests the following guidance:
+- For simple **FIFO** workloads where the primary operations are consuming from the head, `ListQueue` is a strong and
+  simple choice.
+- For workloads requiring **priority-based ordering** or those that are sensitive to allocation overhead under high
+  contention, `MaxMinHeap` is likely the more suitable option.
+
+These benchmarks provide a baseline for performance. The best choice for a specific use case will depend on the expected
+workload patterns.