Introduce a policy-driven quota system w/ real-time enforcement (#322)

## Summary

Adds a first-class, policy-driven quota system to Milo: a v1alpha1 API
surface for registering quota-managed resource types, allocating
capacity to consumers, creating claims at admission time, and enforcing
limits. Policies (CEL-based) automate both grant and claim creation.
Real-time enforcement occurs in the API server admission chain, with
decisions computed by a single-writer AllowanceBucket controller.
Tracing and Prometheus metrics are included.

## API Surface (v1alpha1)

See **[Quota API types and documentation]** for a detailed overview of
the API. Here's the brief overview:

- **ResourceRegistration** — declares a quota-managed resource type
(units, name-gen, constraints; default-deny surface).
- **ResourceGrant** — allocates capacity (static allowances) to a
consumer; only `Active=True` grants count.
- **AllowanceBucket** — aggregated view per consumer+resourceType:
`limit`, `allocated`, `available`, counts, and `contributingGrantRefs`.
- **ResourceClaim** — request for capacity (static amounts) evaluated
per create; status conveys `Granted` vs `Denied` with reasons.
- **GrantCreationPolicy** — watches “trigger” resources; when CEL
conditions match, renders and creates/updates/deletes `ResourceGrant`s
(supports hierarchical parent context).
- **ClaimCreationPolicy** — matches incoming creates by GVK + CEL;
renders a `ResourceClaim` used by admission for request-time
enforcement.


## Architecture & Flow

See the **[quota system architecture overview]** for diagrams and
end-to-end flow.

1. **Register types**: Operators create **ResourceRegistration** for
each quota-managed type.
2. **Provision capacity**: Operators/policies create **ResourceGrant**s
for consumers; a controller validates and sets `Active=True`.
3. **Aggregate state**: **AllowanceBucketController**
materializes/updates per-consumer buckets and becomes the **single
writer** of allocation fields (SSA field ownership).
4. **Enforce at admission (create)**: The **Admission Plugin**
intercepts object **create**, finds `Ready=True`
**ClaimCreationPolicy**(ies), renders a **ResourceClaim** (labeled
`auto-created=true`), waits for the bucket’s decision, and
allows/denies.
5. **Lifecycle maintenance**: If **Granted**, the **Ownership
Controller** sets ownerReferences to the created object; if **Denied**
and auto-created, the **Cleanup Controller** deletes the claim on a
safety timer.

## Components

### Policy evaluation & automation

- **Policy Engine** — CELEngine (conditions + name expr), TemplateEngine
(CEL-template rendering for claims/grants), in-memory cache for O(1)
policy lookups during admission; metrics for informer/workqueue
performance. See **[Policy engine for template rendering and
evaluation]**.

- **Policy Controllers & Dynamic Informers** — Ready/validation
controllers for Claim/Grant policies and a dynamic informer framework to
watch trigger GVKs at runtime; includes a **Grant Creation Executor**
and **ParentContext Resolver** for hierarchical targeting. See **[Policy
controllers and dynamic informer framework]**.

### Core enforcement

- **AllowanceBucketController** — centralizes limit/consumption
aggregation and produces grant/deny decisions; owns allocation fields
via **SSA**.
- **ResourceClaimController** — aggregates allocation status into a
clear `Granted` condition (separate from bucket writer).
- **ResourceGrantController** — validates grants against active
registrations; sets `Active` condition.
- **ResourceRegistrationController** — marks registrations `Active` once
valid. See **[Core quota controllers]**.

### Admission
- **Claim-Creation Admission Plugin** — positioned after authz and
before storage; matches policies, renders claims, **blocks** until
decision or timeout; uses a shared informer/watch to avoid N×
connections; exposes Prometheus metrics for latency, waiters, results.
See **[Admission plugin for request-time enforcement]**.

### Lifecycle hygiene
- **DeniedAutoClaimCleanupController** — deletes denied, auto-created
claims after a grace period; preserves manual claims.
- **ResourceClaimOwnershipController** — sets ownerReferences for
granted claims; dual-path (fast path vs. safety-net grace). See **[Claim
lifecycle controllers]**.

### Integration, config, and observability
- **System Integration** — registers plugin + all controllers, ensures
initialization order and graceful shutdown. See **[Integrate quota into
apiserver and controller manager]**.
- **System Configurations** — controller/plugin flags, RBAC roles, and
controller-manager wiring; explicit metrics coverage. See **[Quota
system configurations]**.
- **Tracing** — API server tracing support (OpenTelemetry) for admission
path visibility. See **[Distributed tracing support for API server]**.

### Validation & docs
- **Shared Validation Package** — unified CEL env, resource type
validation via informer cache, and template validators; reused by
engine, controllers, and admission. See **[Validation package for quota
resources]**.
- **CEL rule placement** — CEL checks on resource refs moved from CRD
schemas to controller/admission validation for flexibility. See
**[Remove CRD-level CEL validations on resource refs]**.
- **Docs & E2E** — architecture + API docs and end-to-end tests covering
dynamic informers, policy validation, grant automation, bucket
decisions, and admission wait/deny paths. See **[End-to-end tests for
quota system]**.

## Out of Scope (Not implemented in this series)

- **Usage-based / metered quotas** (time-series, rolling windows,
rate-limits).
- **Overage handling & burst semantics** (grace, debt, preemption).
- **Autoscaling of buckets or sharded accounting stores**.
- **UI/UX surfaces** (dashboards, self-service management).
- **Billing integration** (prices, entitlements, invoicing).
- **Policy catalogs/marketplace** beyond the policy CRDs themselves.

### Links
- #283
- #295
- #305
- #307
- #306
- #308
- #309
- #310
- #311
- #312
- #313
- #315
- #316
- #330
- #333

### Improvements

These improvements were made to the original set of PRs based on
feedback received or issues encountered during testing.

- #340 
- #342
- #330 
- #346 
- #351
- #353 

### Outstanding Items

These items were identified during review of the PRs above that need to
be addressed in follow up PRs.

- [X] Convert controllers to be multi-cluster aware
- [X] Update admission plugin to support managing claims in project
control planes

[Quota API types and documentation]:
#283
[Quota system architecture overview]:
#295
[Validation package for quota resources]:
#305
[Policy engine for template rendering and evaluation]:
#307
[Policy controllers and dynamic informer framework]:
#306
[Claim lifecycle controllers]:
#308
[Core quota controllers]: #309
[Admission plugin for request-time enforcement]:
#310
[Integrate quota into apiserver and controller manager]:
#311
[Distributed tracing support for API server]:
#312
[Quota system configurations]:
#313
[End-to-end tests for quota system]:
#315
[Operational visibility section]:
#316
[Remove CRD-level CEL validations on resource refs]:
#330
[Remove k8s native resource quota system]:
https://github.com/datum-cloud/milo/pull/pull/333

Author: scotwells

.dockerignore

@@ -22,6 +22,9 @@ dev-temp/
 
 # Configuration and manifests
 config/
+# Exception: CRD bootstrap package needs to be included for embedded CRDs
+!config/crd/bootstrap.go
+!config/crd/bases/
 Taskfile.yaml
 Taskfile.yml
 docker-compose.yaml

Taskfile.yaml

@@ -94,62 +94,46 @@ tasks:
         set -e
         echo "🚀 Deploying complete Milo control plane to test-infra cluster..."
 
-        # Deploy all components including etcd, API server, controller manager, webhooks, RBAC, and networking
-        # Everything is deployed in the milo-system namespace for simplified testing
         echo "📋 Deploying Milo control plane with etcd storage backend..."
         task test-infra:kubectl -- apply -k config/overlays/test-infra/
 
-        # Wait for etcd Helm release to be complete (ensures deployment is fully reconciled)
         echo "⏳ Waiting for etcd Helm release to be ready..."
         task test-infra:kubectl -- wait --for=condition=Ready helmrelease/etcd -n milo-system --timeout=300s
 
-        # Wait for etcd pod readiness (API server needs etcd to store data)
         echo "⏳ Waiting for etcd pod to be ready..."
         task test-infra:kubectl -- wait --for=condition=Ready pod -l app.kubernetes.io/component=etcd -n milo-system --timeout=180s
 
-        # Wait for API server to be ready (required for CRD installation)
-        # API server must be running before we can install custom resources
         echo "⏳ Waiting for API server to be ready..."
         task test-infra:kubectl -- wait --for=condition=Ready pod -l app.kubernetes.io/name=milo-apiserver -n milo-system --timeout=180s
 
-        # Install CRDs into Milo API server (defines custom resource schemas)
-        # Controller manager needs these CRDs to watch and reconcile custom resources
-        echo "📋 Installing core control plane CRDs into Milo API server..."
-        task kubectl -- apply -k config/crd/overlays/core-control-plane/
-        task kubectl -- wait --for=condition=Established customresourcedefinitions --all
+        echo "✅ Verifying CRDs were bootstrapped..."
+        CRD_COUNT=$(task kubectl -- get crd --no-headers 2>/dev/null | grep miloapis | wc -l || echo "0")
+        if [ "$CRD_COUNT" -eq "0" ]; then
+          echo "⚠️  Warning: No Milo CRDs found - checking API server logs..."
+        else
+          echo "✓ Found $CRD_COUNT Milo CRDs bootstrapped by API server"
+        fi
 
-        # Step 6b: Install infrastructure control plane CRDs into Milo API server
-        # This includes ProjectControlPlanes needed by the controller manager
         echo "📋 Installing infrastructure control plane CRDs into the infrastructure cluster..."
         task test-infra:kubectl -- apply -k config/crd/overlays/infra-control-plane/
         task test-infra:kubectl -- wait --for=condition=Established customresourcedefinitions projectcontrolplanes.infrastructure.miloapis.com
 
-        # Step 7: Verify CRDs are properly installed (sanity check)
-        echo "✅ Verifying CRDs are installed..."
-        CRD_COUNT=$(task kubectl -- get crd --no-headers | grep miloapis | wc -l || echo "0")
-        echo "Installed $CRD_COUNT Milo CRDs in API server"
-
-        # Step 7b: Create test users in Milo API server
         echo "👤 Creating test users in Milo API server..."
         task kubectl -- apply -f config/overlays/test-infra/resources/test-users.yaml
 
-        # Step 8: Wait for controller manager (now that CRDs exist for it to reconcile)
-        # Controller manager can only start successfully after CRDs are available
         echo "⏳ Waiting for controller manager to be ready..."
         task test-infra:kubectl -- wait --for=condition=Ready pod -l app.kubernetes.io/name=milo-controller-manager -n milo-system --timeout=120s
 
-        # Step 8b: Install webhook configurations to Milo API server
-        # Webhooks validate and mutate resources submitted to the Milo API server
         echo "🔒 Installing webhook configurations to Milo API server..."
         task kubectl -- apply -k config/webhook/
-        # Webhook configurations don't have conditions to wait for - they're ready immediately after creation
         echo "✅ Webhook configurations installed successfully"
 
         WEBHOOK_COUNT=$(task kubectl -- get mutatingwebhookconfigurations,validatingwebhookconfigurations --no-headers | grep resourcemanager.miloapis.com | wc -l || echo "0")
         echo "Installed $WEBHOOK_COUNT webhook configurations in Milo API server"
 
-        # Update kubeconfig for easy developer access
-        echo "📝 Updating kubeconfig for developer access..."
+        echo "⚙️  Installing service configurations to Milo API server..."
+        task kubectl -- apply -k config/services/
+        echo "✅ Service configurations installed successfully"
 
         echo ""
         echo "✅ Milo API server and storage deployed successfully!"
@@ -308,6 +292,7 @@ tasks:
     cmds:
       - task: generate:code
       - task: generate:docs
+      - task: generate:test-docs
 
   generate:code:
     desc: Generate code including deepcopy, objects, CRDs, and potentially protobuf marshallers
@@ -333,7 +318,7 @@ tasks:
       - "\"{{.TOOL_DIR}}/controller-gen\" webhook paths=\"./internal/webhooks/...\" output:dir=\"./config/webhook\""
       # Generate RBAC rules for the controllers.
       - echo "Generating RBAC rules for the controllers..."
-      - "\"{{.TOOL_DIR}}/controller-gen\" rbac:roleName=milo-controller-manager paths=\"./internal/controllers/...\" output:dir=\"./config/controller-manager/overlays/core-control-plane/rbac\""
+      - "\"{{.TOOL_DIR}}/controller-gen\" rbac:roleName=milo-controller-manager paths=\"./internal/controllers/...\" paths=\"./internal/quota/controllers/...\" output:dir=\"./config/controller-manager/overlays/core-control-plane/rbac\""
     silent: true
 
   generate:openapi:identity:
@@ -375,6 +360,47 @@ tasks:
         done;
     silent: true
 
+  generate:test-docs:
+    desc: Generate test documentation using Chainsaw build docs
+    deps:
+      - task: install-go-tool
+        vars:
+          NAME: chainsaw
+          PACKAGE: github.com/kyverno/chainsaw
+          VERSION: "{{.CHAINSAW_VERSION}}"
+    cmds:
+      - |
+        set -e
+        echo "Generating test documentation..."
+
+        # Find all test directories containing chainsaw-test.yaml files
+        TEST_DIRS=$(find test -name "chainsaw-test.yaml" -exec dirname {} \; | sort -u)
+
+        if [ -z "$TEST_DIRS" ]; then
+          echo "No chainsaw test files found"
+          exit 0
+        fi
+
+        # Store the absolute path to chainsaw
+        CHAINSAW_BIN="{{.TOOL_DIR}}/chainsaw"
+
+        # Generate documentation for each test directory
+        for test_dir in $TEST_DIRS; do
+          echo "Generating README.md for $test_dir..."
+
+          # Generate README.md in the test directory
+          (cd "$test_dir" && "$CHAINSAW_BIN" build docs --test-dir .)
+
+          if [ -f "$test_dir/README.md" ]; then
+            echo "  ✓ $test_dir/README.md"
+          else
+            echo "  ⚠️  No README.md generated for $test_dir"
+          fi
+        done
+
+        echo "✅ Test documentation generated in test directories"
+    silent: true
+
   install-go-tool:
     desc: Install a Go tool to {{.TOOL_DIR}}/{{.NAME}} (symlinked from {{.TOOL_DIR}}/{{.NAME}}-{{.VERSION}})
     silent: true

cmd/milo/apiserver/admission.go

@@ -7,25 +7,53 @@ import (
 	mutatingwebhook "k8s.io/apiserver/pkg/admission/plugin/webhook/mutating"
 	validatingwebhook "k8s.io/apiserver/pkg/admission/plugin/webhook/validating"
 	"k8s.io/kubernetes/pkg/kubeapiserver/options"
+
+	admissionquota "go.miloapis.com/milo/internal/quota/admission"
+)
+
+// GetMiloOrderedPlugins returns the complete ordered list of admission plugins,
+// including both Kubernetes built-in plugins and our custom Milo plugins.
+// Custom plugins are inserted before the webhook plugins as recommended.
+func GetMiloOrderedPlugins() []string {
+	// Start with Kubernetes' built-in ordered plugin list
+	plugins := make([]string, 0, len(options.AllOrderedPlugins)+1)
+
+	// Find where to insert our custom plugins
+	for _, plugin := range options.AllOrderedPlugins {
+		if plugin == "ValidatingAdmissionPolicy" {
+			// Insert our custom validating plugins here, before ValidatingAdmissionPolicy
+			// This ensures they run before the generic webhook validators
+			plugins = append(plugins, admissionquota.PluginName)
+		}
+		plugins = append(plugins, plugin)
+	}
+
+	return plugins
+}
+
+// MiloAdmissionPlugins lists all Milo-specific admission plugins
+var MiloAdmissionPlugins = sets.New[string](
+	admissionquota.PluginName, // ResourceQuotaEnforcement
+	// Add future Milo admission plugins here
 )
 
-// DefaultOffAdmissionPlugins get admission plugins off by default for kube-apiserver.
+// DefaultOffAdmissionPlugins returns admission plugins that should be disabled by default.
+// This keeps only essential plugins and our custom Milo plugins enabled.
 func DefaultOffAdmissionPlugins() sets.Set[string] {
-	defaultOnPlugins := sets.New[string](
-		lifecycle.PluginName, // NamespaceLifecycle
-		// defaulttolerationseconds.PluginName, // DefaultTolerationSeconds
-		mutatingwebhook.PluginName,   // MutatingAdmissionWebhook
-		validatingwebhook.PluginName, // ValidatingAdmissionWebhook
-		// resourcequota.PluginName,            // ResourceQuota
-		// certapproval.PluginName,              // CertificateApproval
-		// certsigning.PluginName,               // CertificateSigning
-		// ctbattest.PluginName,                 // ClusterTrustBundleAttest
-		// certsubjectrestriction.PluginName,    // CertificateSubjectRestriction
-		validatingadmissionpolicy.PluginName, // ValidatingAdmissionPolicy, only active when feature gate ValidatingAdmissionPolicy is enabled
+	// Plugins we want ON by default
+	defaultOnPlugins := sets.New(
+		lifecycle.PluginName,                 // NamespaceLifecycle
+		mutatingwebhook.PluginName,           // MutatingAdmissionWebhook
+		validatingwebhook.PluginName,         // ValidatingAdmissionWebhook
+		validatingadmissionpolicy.PluginName, // ValidatingAdmissionPolicy
 	)
 
-	universe := sets.New(options.AllOrderedPlugins...)
-	universe.Insert(lifecycle.PluginName)
+	// Add all Milo plugins to the enabled set
+	defaultOnPlugins = defaultOnPlugins.Union(MiloAdmissionPlugins)
+
+	// Get the complete plugin list including our custom plugins
+	allPlugins := sets.New(GetMiloOrderedPlugins()...)
 
-	return universe.Difference(defaultOnPlugins)
+	// Return plugins to turn OFF (all plugins minus the ones we want ON)
+	return allPlugins.Difference(defaultOnPlugins)
 }

cmd/milo/apiserver/config.go

@@ -4,12 +4,16 @@ import (
 	"net/http"
 	"time"
 
+	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
+	semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
+	"go.opentelemetry.io/otel/trace"
 	corev1 "k8s.io/api/core/v1"
 	apiextensionsapiserver "k8s.io/apiextensions-apiserver/pkg/apiserver"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apiserver/pkg/admission"
 	"k8s.io/apiserver/pkg/endpoints/filterlatency"
 	genericapifilters "k8s.io/apiserver/pkg/endpoints/filters"
+	"k8s.io/apiserver/pkg/endpoints/request"
 	genericfeatures "k8s.io/apiserver/pkg/features"
 	"k8s.io/apiserver/pkg/server"
 	genericfilters "k8s.io/apiserver/pkg/server/filters"
@@ -18,6 +22,7 @@ import (
 	"k8s.io/apiserver/pkg/util/webhook"
 	"k8s.io/client-go/discovery"
 	"k8s.io/client-go/rest"
+	"k8s.io/component-base/tracing"
 	utilversion "k8s.io/component-base/version"
 	aggregatorapiserver "k8s.io/kube-aggregator/pkg/apiserver"
 	aggregatorscheme "k8s.io/kube-aggregator/pkg/apiserver/scheme"
@@ -40,8 +45,10 @@ import (
 	"go.miloapis.com/milo/internal/apiserver/admission/initializer"
 	sessionsbackend "go.miloapis.com/milo/internal/apiserver/identity/sessions"
 	identitystorage "go.miloapis.com/milo/internal/apiserver/storage/identity"
+	admissionquota "go.miloapis.com/milo/internal/quota/admission"
 	identityapi "go.miloapis.com/milo/pkg/apis/identity"
 	identityopenapi "go.miloapis.com/milo/pkg/apis/identity/v1alpha1"
+	quotaapi "go.miloapis.com/milo/pkg/apis/quota"
 	datumfilters "go.miloapis.com/milo/pkg/server/filters"
 	openapicommon "k8s.io/kube-openapi/pkg/common"
 )
@@ -147,13 +154,16 @@ func NewConfig(opts options.CompletedOptions) (*Config, error) {
 		Options: opts,
 	}
 
-	// Initialize Milo scheme with identity APIs and install into legacy scheme
+	// Initialize Milo scheme with identity and quota APIs and install into legacy scheme
 	miloScheme := runtime.NewScheme()
 	identityapi.Install(miloScheme)
 	identityapi.Install(legacyscheme.Scheme)
+	quotaapi.Install(miloScheme)
+	quotaapi.Install(legacyscheme.Scheme)
 
 	apiResourceConfigSource := controlplane.DefaultAPIResourceConfigSource()
 	apiResourceConfigSource.DisableResources(corev1.SchemeGroupVersion.WithResource("serviceaccounts"))
+	apiResourceConfigSource.DisableResources(corev1.SchemeGroupVersion.WithResource("resourcequotas"))
 	// Enable identity group/version served by virtual storage
 	apiResourceConfigSource.EnableVersions(identityopenapi.SchemeGroupVersion)
 
@@ -195,6 +205,12 @@ func NewConfig(opts options.CompletedOptions) (*Config, error) {
 	c.ControlPlane = kubeAPIs
 	c.ControlPlane.Generic.EffectiveVersion = utilversion.DefaultKubeEffectiveVersion()
 
+	// Configure tracing for the loopback client to ensure trace context propagation
+	// to admission plugins that use dynamic clients (e.g., quota admission plugin)
+	if kubeAPIs.Generic.LoopbackClientConfig != nil && kubeAPIs.Generic.TracerProvider != nil {
+		kubeAPIs.Generic.LoopbackClientConfig.Wrap(tracing.WrapperFor(kubeAPIs.Generic.TracerProvider))
+	}
+
 	combinedInits := append(upstreamInits, loopbackInit)
 
 	authInfoResolver := webhook.NewDefaultAuthenticationInfoResolverWrapper(kubeAPIs.ProxyTransport, kubeAPIs.Generic.EgressSelector, kubeAPIs.Generic.LoopbackClientConfig, kubeAPIs.Generic.TracerProvider)
@@ -210,6 +226,15 @@ func NewConfig(opts options.CompletedOptions) (*Config, error) {
 	}
 	c.APIExtensions = apiExtensions
 
+	// Add readiness check for quota validator to ensure cache is synced before serving traffic
+	kubeAPIs.Generic.AddReadyzChecks(admissionquota.ReadinessCheck())
+
+	// Add post-start hook to bootstrap CRDs from embedded filesystem
+	// This installs all CRDs EXCEPT infrastructure.miloapis.com group, which should remain in the infrastructure cluster
+	kubeAPIs.Generic.AddPostStartHookOrDie("bootstrap-crds", func(ctx server.PostStartHookContext) error {
+		return bootstrapCRDsHook(ctx, kubeAPIs.Generic.LoopbackClientConfig)
+	})
+
 	// TODO(jreese) create an admission plugin that will prohibit the creation of
 	// a Secret with a type of `kubernetes.io/service-account-token`
 	c.APIExtensions.GenericConfig.DisabledPostStartHooks.Insert("start-legacy-token-tracking-controller")
@@ -311,7 +336,7 @@ func DefaultBuildHandlerChain(apiHandler http.Handler, c *server.Config) http.Ha
 	// }
 	handler = genericfilters.WithHTTPLogging(handler)
 	if c.FeatureGate.Enabled(genericfeatures.APIServerTracing) {
-		handler = genericapifilters.WithTracing(handler, c.TracerProvider)
+		handler = withCustomTracing(handler, c.TracerProvider)
 	}
 	handler = genericapifilters.WithLatencyTrackers(handler)
 	// WithRoutine will execute future handlers in a separate goroutine and serving
@@ -334,3 +359,54 @@ func DefaultBuildHandlerChain(apiHandler http.Handler, c *server.Config) http.Ha
 
 	return handler
 }
+
+// withCustomTracing provides tracing that always creates child spans (not links)
+// ensuring proper trace hierarchy for all requests including internal loopback requests.
+func withCustomTracing(handler http.Handler, tp trace.TracerProvider) http.Handler {
+	opts := []otelhttp.Option{
+		otelhttp.WithPropagators(tracing.Propagators()),
+		otelhttp.WithTracerProvider(tp),
+		otelhttp.WithServerName("milo-apiserver"),
+		otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string {
+			ctx := r.Context()
+			info, exist := request.RequestInfoFrom(ctx)
+			if !exist || !info.IsResourceRequest {
+				return r.Method
+			}
+			return getSpanNameFromRequestInfo(info, r)
+		}),
+		// Note: NOT using WithPublicEndpoint() to always create child spans instead of links
+	}
+
+	wrappedHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		// Add the http.target attribute to the span
+		if r.URL != nil {
+			trace.SpanFromContext(r.Context()).SetAttributes(semconv.HTTPTarget(r.URL.RequestURI()))
+		}
+		handler.ServeHTTP(w, r)
+	})
+
+	// With Noop TracerProvider, the otelhttp still handles context propagation.
+	// See https://github.com/open-telemetry/opentelemetry-go/tree/main/example/passthrough
+	return otelhttp.NewHandler(wrappedHandler, "MiloAPI", opts...)
+}
+
+// getSpanNameFromRequestInfo creates span names from Kubernetes request info
+func getSpanNameFromRequestInfo(info *request.RequestInfo, r *http.Request) string {
+	spanName := "/" + info.APIPrefix
+	if info.APIGroup != "" {
+		spanName += "/" + info.APIGroup
+	}
+	spanName += "/" + info.APIVersion
+	if info.Namespace != "" {
+		spanName += "/namespaces/{:namespace}"
+	}
+	spanName += "/" + info.Resource
+	if info.Name != "" {
+		spanName += "/" + "{:name}"
+	}
+	if info.Subresource != "" {
+		spanName += "/" + info.Subresource
+	}
+	return r.Method + " " + spanName
+}