[stale] [feature] Add ingestion batching (spans and meters)

[jp-agenta]

[STALE] [feature] Add ingestion batching (spans and meters)

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

GitHub CI seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull Request Overview

This is a stale feature branch that adds ingestion batching capabilities for spans and meters. The changes involve significant refactoring of evaluation entities, removal of legacy application code, updates to database migrations, authentication logic simplification, and entrypoint reorganization.

Key Changes:

• Renamed evaluation entities from Result/Metrics to Step/Metric (singular forms)
• Removed legacy application routers, models, and utilities
• Simplified user authentication by removing PostHog feature flag integration
• Refactored database migration utilities from async to sync patterns
• Updated entitlements system with soft-check caching mechanism

Reviewed Changes

Copilot reviewed 114 out of 2332 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
api/oss/src/apis/fastapi/evaluations/models.py	Renamed evaluation entities (Result→Step, Metrics→Metric) and updated query models
api/oss/src/apis/fastapi/applications/*	Removed legacy application router, models, and utilities
api/oss/src/apis/fastapi/annotations/*	Major refactor adding annotation utilities, expanding router with CRUD operations
api/oss/src/__init__.py	Simplified email blocking logic by removing PostHog integration
api/oss/docker/Dockerfile.*	Cleaned up Docker build steps and removed SDK installation
api/oss/databases/postgres/migrations/utils.py	Converted async database operations to synchronous patterns
api/oss/databases/postgres/migrations/tracing/versions/*	Fixed migration index creation and enum naming
api/oss/databases/postgres/migrations/core/versions/*	Removed several stale migration files
api/entrypoint.py	Reorganized service initialization and router mounting order
api/ee/src/utils/entitlements.py	Added soft-check caching mode for entitlements
api/ee/src/services/*	Updated service methods to use correct database managers and fixed parameter naming

Comments suppressed due to low confidence (1)

api/oss/databases/postgres/migrations/runner.py:1

• Entire migration runner file was deleted. This removes the ability to run database migrations programmatically, which may be needed for deployment or setup scripts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Enum names are swapped in the downgrade function. span_type_enum should use name='spantype' and trace_type_enum should use name='tracetype' to properly reverse the upgrade operation.

Suggested change

	span_type_enum = sa.Enum(SpanType, name="tracetype")
	trace_type_enum = sa.Enum(TraceType, name="spantype")
	span_type_enum = sa.Enum(SpanType, name="spantype")
	trace_type_enum = sa.Enum(TraceType, name="tracetype")

[Copilot]

The condition if count > 0 is missing a null check. The previous code had if (count or 0) > 0 which safely handles None values. If count is None, this will raise a TypeError.

Suggested change

	if count > 0:
	if (count or 0) > 0:

[Copilot]

URL parameters are no longer being URL-encoded. The previous code used quote() for token, email, and IDs. Without encoding, special characters in these values could break the URL or create security vulnerabilities.

[Copilot]

Return type annotation is incorrect. The function returns the result of loop.run_in_executor(), which is a Future/Task, not a Coroutine. The return type should be the actual return type of func or Any.

Suggested change

	from typing import Callable, Coroutine
	async def run_in_separate_thread(func: Callable, *args, **kwargs) -> Coroutine:
	from typing import Callable, Coroutine, Any
	async def run_in_separate_thread(func: Callable, *args, **kwargs) -> Any:

[Copilot]

Permission naming changed from plural (VIEW_APPLICATIONS) to singular forms. This is a breaking change that will affect existing permission checks throughout the codebase if not updated consistently.

The best way to fix this is to validate and constrain the user-provided evaluationTableId before using it to construct an internal API URL. Ideally, restrict the value to a string that matches an expected pattern—most likely a UUID (or whatever the allowed set of IDs is). This prevents attackers from injecting special characters or performing path traversal. We can implement this validation at the point where useEvaluationRunData is called, or at the beginning of the hook itself (preferable for defense-in-depth, and because we cannot assume calling code changes). Should the value fail validation, early-return or throw an error, or use a known safe value (e.g., null) for the request. We'll add a helper isValidEvaluationId function to do this check and only proceed if it passes; otherwise, we return early or handle the error gracefully.

The recommended fix is to validate and sanitize the evaluationId parameter before incorporating it into the URL for outgoing requests. Specifically, since evaluation IDs are likely opaque tokens (such as UUIDs), the code should check that each supplied ID matches an expected format before sending the request. This can be implemented by adding a utility function that verifies whether a string is a valid UUID (or whatever structure is expected for evaluation IDs). If validation fails, the code should either throw an error or avoid sending the request.

Edits should occur in web/ee/src/services/evaluations/api/index.ts, within the relevant functions (fetchEvaluation, fetchEvaluationStatus, fetchAllEvaluationScenarios, and the mapping inside fetchAllComparisonResults). Add a UUID validation function and call it before requests are made with user-supplied evaluation IDs. If an ID fails validation, throw an error instead of interpolating it into the URL.

A well-known library for UUID validation is validator, but you can use a simple regex or the built-in validate method from the uuid package (since uuid is already imported).

To fix this SSRF risk, we must ensure that the user-controlled values interpolated into the outgoing request URL (specifically, the evaluationId in /evaluations/${evaluationId}/evaluation_scenarios) are strictly validated or constrained to allow only legitimate, expected values such as UUIDs. The fix should be implemented in the frontend code before making the Axios request. The most robust approach is to run a validation (e.g., a regex match) that accepts only valid UUIDs or expected ID formats before passing them into the request, and to ignore or reject any IDs that do not match. This should be done in the fetchAllComparisonResults and fetchAllEvaluationScenarios functions. Add a helper for validation (e.g., isValidUUID) and apply it to filter IDs. If an ID does not validate, it should not be used in any request.

Required changes:

• Add a UUID validation method (can use a utility function or simple regex).
• In fetchAllComparisonResults, filter the incoming evaluationIds array for valid IDs only before proceeding.
• In fetchAllEvaluationScenarios, validate evaluationId before making any requests; throw an error or skip if invalid.

---

The best way to fix the SSRF issue is to validate or sanitize the scenarioId before including it as a path segment in the outgoing request URL. As a scenario ID should be a UUID, the fix is to ensure the input matches a valid UUID pattern before any outgoing request is made. If the validation fails, the code should avoid issuing the request (and possibly show an error or fallback accordingly). The changes should happen in the function updateScenarioStatusRemote in web/ee/src/services/evaluations/workerUtils.ts where scenarioId is used, and anywhere else in the request chain where scenario IDs enter the system from user input, although the main risk is at the HTTP request boundary. To implement, add a simple UUID validation method (e.g., using regex), and early-return/safeguard if the validation fails.

Specifically:

• Add a isValidUUID helper to validate scenarioId.
• At the beginning of updateScenarioStatusRemote, check that scenarioId is a valid UUID; if not, throw or early-return (do not send fetch request).
• Ensure imports (no new external dependencies needed, use regex).
• This change is all within the file web/ee/src/services/evaluations/workerUtils.ts.

---

To fix this problem, validate or sanitize the evaluationId value before using it in the outbound HTTP request URL. The recommended approach is to ensure it matches the expected format for an evaluation ID (such as a MongoDB ObjectID, a UUID, or an allowed-string regular expression).
Modify the function fetchLoadEvaluation in web/ee/src/services/human-evaluations/api/index.ts so that before the API call, it checks that evaluationId consists only of permitted characters (for example, /^[a-zA-Z0-9_-]{1,36}$/ or a stricter pattern, depending on the ID system in use).
If validation fails, log the error and return null or throw a harmless error.
To implement this, you may need to add an in-file helper function for validation, such as isSafeEvaluationId.

Edits will be made only to the fetchLoadEvaluation function in web/ee/src/services/human-evaluations/api/index.ts. No changes to other files are needed.

---

The best way to fix this is to avoid injecting untrusted input directly into the format string passed to console.error. Instead, use a static format string, and pass user data (such as evaluationId) as a variable argument. For this case, change:

console.error(`Error fetching evaluation ${evaluationId}:`, error)

to:

console.error("Error fetching evaluation %s:", evaluationId, error)

This way, even if evaluationId includes format characters (like %d), they're treated as string data, not format specifiers. No new imports are needed, and the change is entirely in web/ee/src/services/human-evaluations/api/index.ts, line 93.

---

To fix this SSRF risk, restrict the format and allowed values of evaluationTableId before it is used to construct the outgoing HTTP request path. The best way is to validate that evaluationTableId conforms to the allowed pattern (such as a UUID or a database ObjectId, if that's the case), and reject or return an error if not. This ensures that client-provided input cannot cause a request to an unintended endpoint. We can implement a function to validate the format (e.g., a UUID validator) and call it before making the outgoing request. If evaluationTableId does not match the allowed pattern, throw an error or reject the request early.

This fix affects fetchAllLoadEvaluationsScenarios in web/ee/src/services/human-evaluations/api/index.ts, adding a validation step on evaluationTableId before using it in the URL.

---

To fix this issue, we need to ensure that the evaluationScenarioId parameter used in the API endpoint URL is strictly validated or sanitized before use, so that only legitimate values (e.g., UUIDs, fixed-format IDs, etc.) are allowed. In general, the fix involves limiting path or query parameters inserted into outbound request URLs to known-safe, validated input values. The best course is to validate evaluationScenarioId (and possibly evaluationTableId) before making the API call with them: ensure they match the expected pattern (e.g., a UUID using regex, or that they appear in a known, safe list of scenario IDs for that evaluation). This validation should be added immediately prior to calling axios.put in updateEvaluationScenario in web/ee/src/services/human-evaluations/api/index.ts.

To implement this, add a simple validation—using a regex for UUIDs (assuming that's the allowed format, which is exceedingly common for such IDs in typical API design) or restricting to a known-safe list. If the validation fails, throw an error or return early, preventing the potentially unsafe API call.

---