Skip to content

feat: add support for edge ingestion and query - #176

Merged
Topper (toppercodes) merged 24 commits into
mainfrom
feat/edge-based-ingestion
Feb 2, 2026
Merged

feat: add support for edge ingestion and query#176
Topper (toppercodes) merged 24 commits into
mainfrom
feat/edge-based-ingestion

Conversation

@toppercodes

@toppercodes Topper (toppercodes) commented Jan 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Add support for regional edge endpoints for ingest and query operations, aligned with axiom-go PR #401.

Changes

Client API

  • edge parameter: Regional edge domain (e.g., eu-central-1.aws.edge.axiom.co)
  • edge_url parameter: Explicit edge URL (takes precedence over edge)
  • Environment variables: AXIOM_EDGE, AXIOM_EDGE_URL

Edge Routing

  • Ingest requests: https://{edge}/v1/ingest/{dataset}
  • Query requests: https://{edge}/v1/query/_apl
  • All other API operations (datasets, annotations, users, tokens) use main API

Validation

  • Edge endpoints require API tokens (xaat-), not personal tokens (xapt-)
  • PersonalTokenNotSupportedForEdgeError raised on edge ingest with personal token

Dataset Region

  • DatasetsClient.create() accepts optional region parameter for dataset locality

Example Usage

import axiom_py

# Using a regional edge domain
client = axiom_py.Client(edge="eu-central-1.aws.edge.axiom.co")

# Or using an explicit edge URL
client = axiom_py.Client(edge_url="https://custom-edge.example.com")

# Ingest and query use edge, other operations use main API
client.ingest_events("my-dataset", [{"foo": "bar"}])
client.query("['my-dataset'] | limit 10")

Breaking Changes

None - edge configuration is optional and existing behavior is preserved.

Topper (toppercodes) and others added 13 commits January 16, 2026 11:40
@toppercodes @ampagent
feat: add support for regional edge endpoints
2252532 
Adds support for Axiom's regional edge deployments for data ingestion,
matching the functionality implemented in Vector (vectordotdev/vector#24037).

## New Configuration Options

- `region`: Regional edge domain (e.g., 'eu-central-1.aws.edge.axiom.co')
  When set, data is sent to `https://{region}/v1/ingest/{dataset}`
- `url`: Smart path handling for backwards compatibility
  - URLs with custom paths are used as-is
  - URLs without paths append legacy format `/v1/datasets/{dataset}/ingest`

## Environment Variables

- `AXIOM_REGION`: New env var for regional edge configuration
- `AXIOM_URL`: Existing env var (unchanged behavior)

## Configuration Priority

url > region > default cloud endpoint

## Breaking Changes

- Parameter renamed from `url_base` to `url` (positional args still work)

## Validation

- `EdgeConfigError` raised when both `url` and `region` are specified

## Architecture

- Dual session design: API session for management operations,
  edge session for ingest/query operations
- Only ingest and query operations use edge endpoints
- All other API operations (datasets, annotations, users, tokens)
  continue to use api.axiom.co

Amp-Thread-ID: https://ampcode.com/threads/T-019bc691-bc45-718d-8f52-a4b5d139440c
Co-authored-by: Amp <amp@ampcode.com>
@toppercodes
fix: use api.axiom.co as default for backwards compatibility
568e7a0 
Edge endpoints use a different path format (/v1/ingest/{dataset}) than
the legacy API (/v1/datasets/{dataset}/ingest). For backwards compatibility,
the default should use api.axiom.co with the legacy path format.

Edge path format is only used when `region` is explicitly configured.
@toppercodes
style: fix ruff formatting in test_client.py
0f64d17
@toppercodes
fix: clear AXIOM_URL env var in edge configuration tests
bbf69b3 
The CI environment has AXIOM_URL set, which was causing EdgeConfigError
when tests tried to use the region parameter.
@toppercodes
style: reformat with ruff 0.6.4 to match CI
85c6464
@toppercodes
fix: use AXIOM_EDGE_URL env var instead of AXIOM_REGION
3159dda
@toppercodes
fix: rename env var to AXIOM_EDGE_REGION
3bfd03e
@toppercodes
fix: url takes precedence over region instead of raising error
8236a0f 
Matches Vector's behavior where url takes precedence if both are set,
rather than raising an error.
@toppercodes
fix: remove unused EdgeConfigError and AXIOM_EDGE_URL exports
92f8c93
@toppercodes
test: add edge integration tests
36f6048 
Add comprehensive integration tests for edge-based ingest and query:
- TestEdgeIntegration with ingest, query, and roundtrip tests
- Tests skip if AXIOM_EDGE_URL or AXIOM_EDGE_REGION not set
- Supports AXIOM_EDGE_TOKEN for separate edge authentication
- TestEdgeURLConfiguration for URL precedence unit test

Update CI workflow with edge secrets matching axiom-go:
- AXIOM_EDGE_URL, AXIOM_EDGE_REGION, AXIOM_EDGE_TOKEN
- AXIOM_EDGE_DATASET_REGION for dataset region config
@toppercodes
feat: add edge_url parameter for explicit edge endpoint
4e5f202 
Add AXIOM_EDGE_URL support to match axiom-go:
- edge_url parameter uses edge path format /v1/ingest/{dataset}
- url parameter uses legacy format /v1/datasets/{dataset}/ingest
- Priority: edge_url > region > url (for edge operations)

Update tests to clear AXIOM_EDGE_URL env var and add new test cases.
@toppercodes
feat: add edge_token parameter for edge authentication
a81a437 
Edge endpoints require an API token (not personal token). Add:
- edge_token parameter to Client (falls back to AXIOM_EDGE_TOKEN env var)
- Edge session uses edge_token for Authorization header
- Falls back to main token if edge_token not provided

This matches axiom-go's approach where edge operations use a
dedicated token separate from the main API token.
@toppercodes @ampagent
feat: align edge implementation with axiom-go PR #401
99005a3 
- Rename 'region' param to 'edge', AXIOM_EDGE_REGION to AXIOM_EDGE
- Remove edge_token param and AXIOM_EDGE_TOKEN (use single token)
- Add PersonalTokenNotSupportedForEdgeError for edge ingest validation
- Simplify to single session (remove dual session architecture)
- Fix edge query path to /v1/query/_apl
- Add region param to DatasetCreateRequest for dataset locality
- Update CI to use vars instead of secrets for edge config
- Add edge documentation to README

Amp-Thread-ID: https://ampcode.com/threads/T-019bfc15-cc0b-7248-843c-a5e234a18301
Co-authored-by: Amp <amp@ampcode.com>
@toppercodes Topper (toppercodes) changed the title feat: add support for regional edge endpoints feat: add support for edge ingestion and query Jan 26, 2026
Topper (toppercodes) and others added 5 commits January 26, 2026 21:07
@toppercodes @ampagent
fix: align CI with axiom-go matrix pattern and add AXIOM_EDGE_TOKEN s…
7e06f85 
…upport

- Use matrix with environment (development/staging) and slug (DEV/STAGING)
- Use vars for EDGE_URL, EDGE, EDGE_DATASET_REGION (account-level)
- Use secrets for EDGE_TOKEN (account-level)
- Add AXIOM_DATASET_SUFFIX for easier test cleanup
- Edge integration test uses AXIOM_EDGE_TOKEN for edge client
- Update helpers.py to use AXIOM_DATASET_SUFFIX in dataset names

Amp-Thread-ID: https://ampcode.com/threads/T-019bfc15-cc0b-7248-843c-a5e234a18301
Co-authored-by: Amp <amp@ampcode.com>
@toppercodes @ampagent
fix: do not auto-read edge config from environment
2621f96 
Edge configuration (AXIOM_EDGE_URL, AXIOM_EDGE) must be passed explicitly
when creating a Client. This prevents accidentally routing all requests
through edge when these env vars are set for edge-specific tests.

This matches axiom-go behavior where a separate edge client is created
with explicit edge options.

Amp-Thread-ID: https://ampcode.com/threads/T-019bfc15-cc0b-7248-843c-a5e234a18301
Co-authored-by: Amp <amp@ampcode.com>
@toppercodes @ampagent
fix: use proper APL syntax in logger test and add edge debug logging
bc66bbd 
- Logger test now uses ['dataset-name'] syntax for APL queries
- Add debug logging to edge integration test to diagnose region mismatch

Amp-Thread-ID: https://ampcode.com/threads/T-019bfc15-cc0b-7248-843c-a5e234a18301
Co-authored-by: Amp <amp@ampcode.com>
@toppercodes @ampagent
feat: align edge implementation with axiom-go PR #401
929e827 
- Use single token for both API and edge operations
- Remove AXIOM_EDGE env var, use only edge_url parameter
- Add personal token validation for edge ingest/query
- Add dataset region parameter for regional dataset creation
- Normalize empty string edge_url to None
- Add skip logic for edge tests when server ignores region param
- Add retry logic for edge query eventual consistency

Amp-Thread-ID: https://ampcode.com/threads/T-019c043c-0fdd-73e2-851a-b670728fb75a
Co-authored-by: Amp <amp@ampcode.com>
@toppercodes @ampagent
fix: add retry with exception handling for edge query tests
a9ac55d 
Handle AxiomError for 'invalid field' when schema not yet indexed.
Increase retry attempts to 10 with 2s delay for eventual consistency.

Amp-Thread-ID: https://ampcode.com/threads/T-019c043c-0fdd-73e2-851a-b670728fb75a
Co-authored-by: Amp <amp@ampcode.com>
@islameldigwi
Merge branch 'main' into feat/edge-based-ingestion
ac39f28
@islameldigwi
feat: implement edge support for async clients
24de108
@islameldigwi
fix formatting
ad50320
@islameldigwi
proper async tests
3b25b49
@thecraftman @claude
fix: resolve variable shadowing bug in edge integration tests
2b55d7d 
Changed variable name from `time` to `t` in test_edge_ingest_events()
and test_edge_ingest_and_query_roundtrip() to avoid shadowing the
`time` module, which caused time.sleep(2) calls to fail.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@thecraftman @claude
fix: add region validation to async edge integration tests
f854ec5 
Added missing region validation to TestAsyncEdgeIntegration.setUpClass
to match the sync TestEdgeIntegration behavior:
- Skip tests if AXIOM_EDGE_DATASET_REGION is not set
- Verify dataset was created in expected region before running tests
- Skip with clear message if region mismatch detected

This prevents "mismatched region" errors when the dataset is created
in a different region than the edge endpoint.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@toppercodes
Topper (toppercodes) merged commit 6d4f170 into main Feb 2, 2026
12 checks passed
@toppercodes
Topper (toppercodes) deleted the feat/edge-based-ingestion branch February 2, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ronoc Conor Curran (ronoc) ronoc approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@toppercodes @ronoc @islameldigwi @thecraftman