Community-contributed evaluation framework adapters for eval-hub.
This repository contains adapters that integrate various evaluation frameworks with the eval-hub service. Each adapter implements the FrameworkAdapter pattern from the evalhub-sdk, enabling seamless integration with the eval-hub evaluation service.
| Framework | Container Image | Local | Kubernetes | Notes |
|---|---|---|---|---|
| LightEval | quay.io/evalhub/community-lighteval:latest |
✗ | ✓ | Lightweight evaluation framework for language models |
| GuideLLM | quay.io/evalhub/community-guidellm:latest |
✗ | ✓ | Performance benchmarking platform for LLM inference servers |
| MTEB | quay.io/evalhub/community-mteb:latest |
✗ | ✓ | Massive Text Embedding Benchmark for embedding models |
# Build specific adapter
make image-lighteval
make image-guidellm
# Build all adapters
make images
# Push to registry
make push-lighteval REGISTRY=quay.io/your-org VERSION=v1.0.0
make push-guidellm REGISTRY=quay.io/your-org VERSION=v1.0.0See CONTRIBUTING.md for guidelines on adding adapters.
See the LICENSE file for details.