Add modern Uploads and Datasets APIs, deprecate legacy Table API#187
Merged
Add modern Uploads and Datasets APIs, deprecate legacy Table API#187
Conversation
This commit introduces the modern /v1/uploads and /v1/datasets API endpoints while maintaining backward compatibility with the legacy /table endpoints. Changes: - Add UploadsAPI with /v1/uploads/* endpoints for table management - Add DatasetsAPI with /v1/datasets/* endpoints for dataset discovery - Deprecate all TableAPI methods with migration guidance - Add comprehensive response models for new APIs - Update ExtendedAPI to include new APIs with proper MRO - Add unit and E2E tests for new functionality - Configure ruff to allow unittest-style assertions in tests All existing TableAPI methods remain functional but emit deprecation warnings pointing users to the new UploadsAPI methods.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| namespace=self.test_namespace, | ||
| table_name=self.test_table_name, | ||
| ) | ||
| self.assertIsInstance(delete_result, DeleteTableResponse) |
There was a problem hiding this comment.
Bug: Namespace Confusion Leads to Orphaned Tables
The test_upload_csv_and_delete test calls upload_csv() which returns only table_name in the response (not namespace), then attempts to delete the table using a hardcoded namespace "test". The CSV upload endpoint likely creates tables in the authenticated user's namespace, not "test", causing the delete operation to target the wrong namespace and potentially fail or leave orphaned tables.
- Remove double /v1 prefix from all routes (api_version already includes it) - Update Dataset models to match actual API responses (full_name instead of slug/namespace) - Update Table models to match actual API responses (full_name format) - Change owner from id to handle+type fields - Add nullable field to column models - Change InsertDataResponse to use 'name' field instead of 'table_name' - Update datasets integration tests to use full_name and required filters - Update uploads integration tests to use DUNE_NAMESPACE env var and handle CSV upload naming - Fix column types in test schemas (int -> integer)
- Change Dataset mocks to use full_name instead of slug/namespace - Update owner structure to use handle+type instead of id+handle - Add nullable field to all column mocks - Change metadata from description field to dict - Update TableElement mocks to use full_name format - Change InsertDataResponse mock to use 'name' instead of 'table_name' - Fix all route expectations to remove /v1 prefix (already in api_version)
va3093
force-pushed
the
add-uploads-datasets-api
branch
from
94ace33 to
9a497df
Compare
va3093
force-pushed
the
add-uploads-datasets-api
branch
from
9a497df to
1f6cb0a
Compare
norbertdurcansk
approved these changes
Nov 14, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The Dune API has evolved to use versioned endpoints (
/v1/uploads/*and/v1/datasets/*) as defined in the OpenAPI specification, but the Python client was still using legacy unversioned routes (/table/*). This PR modernizes the client to support the new API structure while maintaining full backward compatibility.Key findings from the OpenAPI spec analysis:
/v1/uploads/*endpoints provide comprehensive table management (list, create, upload CSV, insert, clear, delete)/v1/datasets/*endpoints enable dataset discovery and schema inspection across all dataset types/table/*endpoints are not present in the current OpenAPI spec, indicating they should be deprecatedWhat Changed
New APIs Added
UploadsAPI (
dune_client/api/uploads.py) - Modern table managementlist_uploads()- List all uploaded tables with paginationcreate_table()- Create empty table with defined schemaupload_csv()- Upload CSV with automatic schema inferenceinsert_data()- Insert data into existing table (CSV/NDJSON)clear_table()- Remove all data while preserving structuredelete_table()- Permanently delete table and dataDatasetsAPI (
dune_client/api/datasets.py) - Dataset discoverylist_datasets()- Browse datasets with filtering by owner/typeget_dataset()- Get detailed dataset info including schemaResponse Models
Added 17 new dataclass models to
models.py:DatasetType,DatasetOwner,DatasetColumn,Dataset,DatasetListResponse,DatasetResponseTableOwner,TableColumn,TableElement,UploadListResponse,UploadCreateResponse,CSVUploadResponse,InsertDataResponse,ClearTableResponse,DeleteTableResponseAll models follow existing patterns with
DataClassJsonMixinandfrom_dict()constructors.Backward Compatibility
TableAPI deprecation - All 5 methods in
dune_client/api/table.pynow use@deprecateddecorator:upload_csv()→ UseUploadsAPI.upload_csv()insteadcreate_table()→ UseUploadsAPI.create_table()insteadinsert_table()→ UseUploadsAPI.insert_data()insteadclear_data()→ UseUploadsAPI.clear_table()insteaddelete_table()→ UseUploadsAPI.delete_table()insteadDeprecation warnings include version (1.9.0) and clear migration guidance. All legacy methods remain fully functional.
Method Resolution Order (MRO)
Updated
ExtendedAPIinheritance inapi/extensions.py:UploadsAPIandDatasetsAPIbeforeTableAPIin the inheritance listtype: ignore[misc]to suppress expected mypy conflicts between incompatible method signaturesTesting
Unit tests (10 new tests, all passing):
tests/unit/test_uploads_api.py- Mock-based tests for all UploadsAPI methodstests/unit/test_datasets_api.py- Mock-based tests for all DatasetsAPI methodsE2E integration tests:
tests/e2e/test_uploads_integration.py- Real API tests for table lifecycletests/e2e/test_datasets_integration.py- Real API tests for dataset discoveryConfiguration
Updated
pyproject.toml:PT009to test file ignores to allow unittest-style assertionsVerification
✅ All 48 unit tests pass
✅ MyPy strict type checking passes (21 source files)
✅ Ruff formatting check passes
✅ Ruff linting passes
✅ Deprecation warnings work correctly
✅ New API methods accessible via
DuneClient✅ Zero breaking changes - full backward compatibility maintained