datastax
diff --git a/‎CHANGES
Lines changed: 43 additions & 5 deletions b/‎CHANGES
Lines changed: 43 additions & 5 deletions
diff --git a/‎README.md
Lines changed: 135 additions & 1 deletion b/‎README.md
Lines changed: 135 additions & 1 deletion
@@ -1,8 +1,43 @@
-main
-====
-Exceptions
-    - `DataAPIResponseException.error_descriptors` is a property (computed from detailed_error_descriptors)
+v 2.0.0
+==========
+Collections admit `DataAPIMap` in writing and support `DataAPIDate` as well (the latter coming with the same timezone caveats as `datetime.date`)
+(plus all of the v2 pre-releases below)
+
+
+v 2.0.0rc2
+==========
+Cursors (class hierarchy revised to accommodate `find_and_rerank`, plus other changes):
+    - renamed the 'FindCursorState' enum to `CursorState`
+    - renamed the abstract ur-class 'FindCursor' => `AbstractCursor`
+    - `clone` method does not strip the mapping anymore, rather retains it
+    - removed the `alive` sugar property (use `cursors.state != CursorState.CLOSED`)
+Support for reranker header-based authentication:
+    - new authentication classes `RerankingHeadersProvider`, `RerankingAPIKeyHeaderProvider`
+    - introduced `reranking_api_key` parameter for APIOptions, `{get|create}_{table|collection}` database methods, collection/table `with_options` and `to_[a]sync` methods
+Support for "findRerankingProviders" API command in Database Admin classes:
+    - classes class hierarchy: `RerankingProviderParameter`, `RerankingProviderModel`, `RerankingProviderToken`, `RerankingProviderAuthentication`, `RerankingProvider`, `FindRerankingProvidersResult` to express the response
+    - database admin `find_reranking_providers`/`async_find_reranking_providers` methods implemented.
+Support for findAndRerank in collections:
+    - new classes `CollectionLexicalOptions`, `CollectionRerankOptions`, `RerankServiceOptions` for create_collection
+    - new `lexical` and `rerank` entries in CollectionDefinition (+builder interface management)
+    - `find_and_rerank` method for collections
+    - cursor classes `[Async]CollectionFindAndRerankCursor` added
+    - findAndRerank cursors return the new `RerankedResult` construct by default (modulo custom mappings)
+Maps for tables expressed as list of pairs (association lists):
+    - support for automatic handling of DataAPIMaps (+possibly dicts) in the proper table payload portions
+    - introduced serdes option `encode_maps_as_lists_in_tables` (default to "NEVER") to control this
+Exceptions, major rework of `[Table|Collection]InsertManyException`, `CollectionUpdateManyException` and `CollectionDeleteManyException` ("bulk operations")
+    - All astrapy exceptions derive directly from `Exception` (and not 'ValueError` anymore)
     - better string representation of `DataAPIDetailedErrorDescriptor`
+    - `DataAPIDetailedErrorDescriptor` removed.
+    - 'CumulativeOperationException` removed.
+    - The four `CollectionInsertManyException`, `CollectionUpdateManyException`, `CollectionDeleteManyException`, `TableInsertManyException` classes now inherit directly from `DataAPIResponseException`.
+    - New semantics and structure for `[Collection|Table]InsertManyException`: they have members `inserted_ids`(/`inserted_id_tuples`) and an `exceptions` list for the root cause(s)
+    - New semantics and structure for `Collection[Update|Delete]ManyException`: they have members `partial_result` and a single-exception `cause`. They are now raised consistently for API exceptions occurring during the respective methods.
+Arbitrary field names and dot-escaping:
+    - offering utilities `astrapy.utils.document_paths.escape_field_names/unescape_field_path`
+    - `distinct` methods can accept a list of (literal) str|int as well as a(n escaped) identifier string
+Improved StrEnum matching for e.g. better coercion of TableIndexType (and future enum with e.g. dashes in values)
 Spawner methods for databases/admins standardized; they don't issue DevOps API calls.
     - removed `normalize_region_for_id` utility method, not used anymore.
     - `AstraDBAdmin.get_[async]_database()`:
@@ -29,6 +64,7 @@ Replaced the ValueErrors not directly coming from function calls/constructors wi
 Collection and Table `insert_many` methods employ returnDocumentResponses under the hood.
 maintenance: switch to DSE6.9 for local non-Astra testing.
 
+
 v 2.0.0rc1
 ==========
 Support for TIMEUUID and COUNTER columns:
@@ -41,6 +77,7 @@ restore support for Python 3.8, 3.9
 maintenance: full restructuring of tests and CI (tables+collections on same footing+other)
 maintenance: adopt `blockbuster` in async tests to detect (and bust) any blocking call
 
+
 v 2.0.0-preview
 ===============
 Introduction of full Tables support.
@@ -77,7 +114,7 @@ Reworked and enriched `FindCursor` interface:
     - Cursor classes renamed to `[Async]CollectionCursor`
     - Base class for all (find) cursors renamed to `FindCursor`
     - introduced `map` and `to_list` methods
-    - `cursor.state` now has values in `FindCursorState` enum (take `cursor.state.value` for a string)
+    - `cursor.state` now has values in `CursorState` enum (take `cursor.state.value` for a string)
     - 'cursor.address' is removed from the API
     - `cursor.rewind()` returns None, mutates cursor in-place
     - removed 'cursor.distinct()': use the  corresponding collection(/table) method.
@@ -199,6 +236,7 @@ Internal restructuring/maintenance things:
     - removal of unused imports from toplevel `__init__.py` (ids, constants, cursors)
     - simplified timeout management classes and representations
 
+
 v. 1.5.2
 ========
 Bugfix: `Database.get_collection` uses callers inheritance (same for async)
 
@@ -93,6 +93,56 @@ Next steps:
 - [AstraPy reference](https://docs.datastax.com/en/astra-api-docs/_attachments/python-client/astrapy/index.html)
 - Package on [PyPI](https://pypi.org/project/astrapy/)
 
+### Server-side embeddings
+
+AstraPy works with the "vectorize" feature of the Data API. This means that one can define server-side computation for vector embeddings and use text strings in place of a document vector, both in writing and in reading.
+The transformation of said text into an embedding is handled by the Data API, using a provider and model you specify.
+
+```python
+my_collection = database.create_collection(
+    "my_vectorize_collection",
+    definition=(
+        CollectionDefinition.builder()
+        .set_vector_service(
+            provider="example_vendor",
+            model_name="embedding_model_name",
+            authentication={"providerKey": "<STORED_API_KEY_NAME>"}  # if needed
+        )
+        .build()
+    )
+)
+
+my_collection.insert_one({"$vectorize": "text to make into embedding"})
+
+documents = my_collection.find(sort={"$vectorize": "vector search query text"})
+```
+
+See the [Data API reference](https://docs.datastax.com/en/astra-db-serverless/databases/embedding-generation.html)
+for more on this topic.
+
+### Hybrid search
+
+AstraPy supports the supports the "find and rerank" Data API command,
+which performs a hybrid search by combining results from a lexical search
+and a vector-based search in a single operation.
+
+```python
+r_results = my_collection.find_and_rerank(
+    sort={"$hybrid": "query text"},
+    limit=10,
+    include_scores=True,
+)
+
+for r_result in r_results:
+    print(r_result.document, r_results.scores)
+```
+
+The Data API must support the primitive (and one must not have
+disabled the feature at collection-creation time).
+
+See the Data API reference, and the docstring for the `find_and_rerank` method,
+for more on this topic.
+
 ### Using Tables
 
 The example above uses a _collection_, where schemaless "documents" can be stored and retrieved.
@@ -184,7 +234,17 @@ for result in cursor:
 my_table.drop()
 ```
 
-For more on Tables, consult the [Data API documentation about Tables](https://docs.datastax.com/en/astra-db-serverless/api-reference/tables.html).
+For more on Tables, consult the [Data API documentation about Tables](https://docs.datastax.com/en/astra-db-serverless/api-reference/tables.html). Note that most features of Collections, with due modifications, hold for Tables as well (e.g. "vectorize", i.e. server-side embeddings).
+
+#### Maps as association lists
+
+In the Data API, table `map` columns with key of a type other than text
+have to be expressed as association lists,
+i.e. nested lists of lists: `[[key1, value1], [key2, value2], ...]`.
+
+AstraPy objects can be configured to always do so automatically, for a seamless
+experience.
+See the API Option `serdes_options.encode_maps_as_lists_in_tables` for details.
 
 ### Usage with HCD and other non-Astra installations
 
@@ -393,6 +453,25 @@ my_collection.update_one(
 my_collection.insert_one({"_id": uuid8()})
 ```
 
+### Escaping field names
+
+Field names containing special characters (`.` and `&`) must be correctly escaped
+in certain Data API commands. It is a responsibility of the user to ensure escaping
+is done when needed; however, AstraPy offers utilities to escape sequences of "path
+segments" and -- should it ever be needed -- unescape path-strings back into
+literal segments:
+
+```python
+from astrapy.utils.document_paths import escape_field_names, unescape_field_path
+
+print(escape_field_names("f1", "f2", 12, "g.&3"))
+# prints: f1.f2.12.g&.&&3
+print(escape_field_names(["f1", "f2", 12, "g.&3"]))
+# prints: f1.f2.12.g&.&&3
+print(unescape_field_path("a&&&.b.c.d.12"))
+# prints: ['a&.b', 'c', 'd', '12']
+```
+
 ## For contributors
 
 First install poetry with `pip install poetry` and then the project dependencies with `poetry install --with dev`.
@@ -468,6 +547,23 @@ poetry run pytest [...] -o log_cli=0
 poetry run pytest [...] -o log_cli=1 --log-cli-level=10
 ```
 
+### Special tests (2025-03-25, Temporary provisions)
+
+Running special tests taking `find_and_rerank` into account, until dev/prod/local discrepancies resolved.
+
+**Prod** (usual CI) just runs as is and skips f.a.r.r.
+
+**Dev** (manual CI on a hybrid-capable cloud Data API). One must:
+
+1. launch integration tests with `ASTRAPY_TEST_FINDANDRERANK=y`
+2. ... but also setting "ASTRAPY_TEST_FINDANDRERANK_SUPPRESS_LEXICAL=y" to suppress actual non-null `"$lexical"` sorts, if not rolled out yet.
+  
+**Local** (manual CI on a hybrid-capable locally-running Data API). One must:
+
+1. launch integration tests with `ASTRAPY_TEST_FINDANDRERANK=y`
+2. ... but also with `ASTRAPY_FINDANDRERANK_USE_RERANKER_HEADER=y` to pass a reranker API key where needed
+3. ... which requires an environment variable `HEADER_RERANKING_API_KEY_NVIDIA` to be set with the `AstraCS:...` dev token.
+
 ## Appendices
 
 ### Appendix A: quick reference for key imports
@@ -497,12 +593,29 @@ Constants for data-related use:
 from astrapy.constants import (
     DefaultIdType,
     Environment,
+    MapEncodingMode,
     ReturnDocument,
     SortMode,
     VectorMetric,
 )
 ```
 
+Cursor for find-like operations:
+
+```python
+from astrapy.cursors import (
+  AbstractCursor,
+  AsyncCollectionFindAndRerankCursor,
+  AsyncCollectionFindCursor,
+  AsyncTableFindCursor,
+  CollectionFindAndRerankCursor,
+  CollectionFindCursor,
+  CursorState,
+  RerankedResult,
+  TableFindCursor,
+)
+```
+
 ObjectIds and UUIDs:
 
 ```python
@@ -553,8 +666,14 @@ from astrapy.info import (
     AlterTableAddVectorize,
     AlterTableDropColumns,
     AlterTableDropVectorize,
+    AstraDBAdminDatabaseInfo,
+    AstraDBDatabaseInfo,
     CollectionDefaultIDOptions,
     CollectionDefinition,
+    CollectionDescriptor,
+    CollectionInfo,
+    CollectionLexicalOptions,
+    CollectionRerankOptions,
     CollectionVectorOptions,
     ColumnType,
     CreateTableDefinition,
@@ -563,14 +682,29 @@ from astrapy.info import (
     EmbeddingProviderModel,
     EmbeddingProviderParameter,
     EmbeddingProviderToken,
+    FindEmbeddingProvidersResult,
+    FindRerankingProvidersResult,
+    ListTableDefinition,
+    ListTableDescriptor,
+    RerankingProvider,
+    RerankingProviderAuthentication,
+    RerankingProviderModel,
+    RerankingProviderParameter,
+    RerankingProviderToken,
+    RerankServiceOptions,
+    TableAPIIndexSupportDescriptor,
+    TableAPISupportDescriptor,
     TableBaseIndexDefinition,
     TableIndexDefinition,
+    TableIndexDescriptor,
     TableIndexOptions,
+    TableInfo,
     TableKeyValuedColumnType,
     TableKeyValuedColumnTypeDescriptor,
     TablePrimaryKeyDescriptor,
     TableScalarColumnTypeDescriptor,
     TableUnsupportedColumnTypeDescriptor,
+    TableUnsupportedIndexDefinition,
     TableValuedColumnType,
     TableValuedColumnTypeDescriptor,
     TableVectorColumnTypeDescriptor,