Skip to content

fix: use correct DNS for evalhub #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tarilabs merged 1 commit into from
Mar 25, 2026
Merged

fix: use correct DNS for evalhub #134

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,9 +381,9 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu
coords.annotations.update(
{
"org.opencontainers.image.created": datetime.now(UTC).isoformat(),
"org.evalhub.benchmark": benchmark_id,
"org.evalhub.model": model_name,
"org.evalhub.job_id": job_id,
"io.github.eval-hub.benchmark": benchmark_id,
"io.github.eval-hub.model": model_name,
"io.github.eval-hub.job_id": job_id,
}
)
Comment on lines 381 to 388
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add backward-compatible annotation keys to avoid metadata contract break

This change silently renames externally consumed OCI annotation keys. If any consumer still reads org.evalhub.*, artifact processing can fail (integration reliability break). Write both namespaces during migration, then remove legacy keys in a later versioned change.

Proposed compatibility patch 
                 coords.annotations.update(
                     {
                         "org.opencontainers.image.created": datetime.now(UTC).isoformat(),
+                        # Legacy namespace (backward compatibility)
+                        "org.evalhub.benchmark": benchmark_id,
+                        "org.evalhub.model": model_name,
+                        "org.evalhub.job_id": job_id,
+                        # New reverse-DNS namespace
                         "io.github.eval-hub.benchmark": benchmark_id,
                         "io.github.eval-hub.model": model_name,
                         "io.github.eval-hub.job_id": job_id,
                     }
                 )

As per coding guidelines, "REVIEW PRIORITIES: 2. Architectural issues and anti-patterns; 3. Bug-prone patterns and error handling gaps."

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@main.py` around lines 381 - 388, The update currently writes new OCI
annotation keys via coords.annotations.update; to remain backward-compatible
also write the legacy org.evalhub.* keys alongside the new io.github.eval-hub.*
and org.opencontainers.image.created keys so consumers reading the old namespace
still see metadata. Modify the coords.annotations.update payload to include
duplicate legacy keys (e.g., "org.evalhub.image.created" or matching legacy
names for created/benchmark/model/job_id) set to the same values
(datetime.now(UTC).isoformat(), benchmark_id, model_name, job_id) so both
namespaces are written during migration; remove the legacy keys in a future,
explicitly versioned change.

oci_spec = OCIArtifactSpec(
Expand Down
Loading