refactor entity update tests, amend readme, add USE_CASES.md

Author: Matthias Zimmermann

README.md

@@ -32,10 +32,6 @@ alice = NamedAccount.create('Alice')
 client = Arkiv(account=alice)
 print(f"Connected: {client.is_connected()}")
 
-# Fund the account
-client.node.fund_account(alice)
-print(f"Balance: {client.eth.get_balance(alice.address)}")
-
 # Create entity with data and annotations
 entity_key, tx_hash = client.arkiv.create_entity(
     payload=b"Hello World!",
@@ -45,18 +41,48 @@ entity_key, tx_hash = client.arkiv.create_entity(
 
 # Check and print entity key
 exists = client.arkiv.entity_exists(entity_key)
-print(f"Created entity: {entity_key}, exists={exists}")
+print(f"Created entity: {entity_key} (exists={exists}), creation TX: {tx_hash}")
 
 # Get individual entity and print its details
 entity = client.arkiv.get_entity(entity_key)
 print(f"Entity: {entity}")
 
-# TODO
-# Clean up - delete entities
+# Clean up - delete entity
 client.arkiv.delete_entity(entity_key)
-print("Entities deleted")
+print("Entity deleted")
+```
+
+### Web3 Standard Support
+```python
+from web3 import HTTPProvider
+provider = HTTPProvider('https://kaolin.hoodi.arkiv.network/rpc')
+
+# Arkiv 'is a' Web3 client
+client = Arkiv(provider)
+balance = client.eth.get_balance(client.eth.default_account)
+tx = client.eth.get_transaction(tx_hash)
+```
+
+### Arkiv Module Extension
+```python
+from arkiv import Arkiv
+from arkiv.account import NamedAccount
+
+account = NamedAccount.from_wallet('Alice', wallet, 's3cret')
+client = Arkiv(provider, account = account)
+
+entity_key, tx_hash = client.arkiv.create_entity(
+    payload=b"Hello World!",
+    annotations={"type": "greeting", "version": 1},
+    btl = 1000
+)
+
+entity = client.arkiv.get_entity(entity_key)
+exists = client.arkiv.exists(entity_key)
 ```
 
+## Advanced Features
+
 ### Provider Builder
 
 The snippet below demonstrates the creation of various nodes to connect to using the `ProviderBuilder`.
@@ -82,35 +108,179 @@ provider_kaolin_ws = ProviderBuilder().kaolin().ws().build()
 provider_custom = ProviderBuilder().custom("https://my-rpc.io").build()
 ```
 
-### Web3 Standard Support
+## Arkiv Topics/Features
+
+### Deprecate BTL
+
+BTL (Blocks-To-Live) should be replaced with explicit `expires_at_block` values for predictability and composability.
+
+Relative `BTL` depends on execution timing and creates unnecessary complexity:
+- An entity created with `btl=100` will have different expiration blocks depending on when the transaction is mined
+- Extending entity lifetimes requires fetching the entity, calculating remaining blocks, and adding more—a race-prone pattern
+- Creates asymmetry between write operations (which use `btl`) and read operations (which return `expires_at_block`)
+
+Absolute `expires_at_block` is predictable, composable, and matches what you get when reading entities:
+- Deterministic regardless of execution timing
+- Maps directly to `Entity.expires_at_block` field returned by queries
+- Enables clean compositional patterns like `replace(entity, expires_at_block=entity.expires_at_block + 100)`
+- Aligns write API with read API, making the SDK more intuitive
+
+With `expires_at_block`, updating entities becomes cleaner:
+
 ```python
-from web3 import HTTPProvider
-provider = HTTPProvider('https://kaolin.hoodi.arkiv.network/rpc')
+from dataclasses import replace
 
-# Arkiv 'is a' Web3 client
-client = Arkiv(provider)
-balance = client.eth.get_balance(client.eth.default_account)
-tx = client.eth.get_transaction(tx_hash)
+# Fetch entity
+entity = client.arkiv.get_entity(entity_key)
+
+# Modify payload and extend expiration by 100 blocks
+updated_entity = replace(
+    entity,
+    payload=b"new data",
+    expires_at_block=entity.expires_at_block + 100
+)
+
+# Update entity
+client.arkiv.update_entity(updated_entity)
 ```
 
-### Arkiv Module Extension
+### Query DSL
+
+To make querying entities as simple and natural as possible, rely on a suitable and existing query DSL. Since Arkiv currently uses a SQL database backend and is likely to support SQL databases in the future, the Arkiv query DSL is defined as a **subset of the SQL standard**.
+
+**Rationale:**
+- Leverages existing SQL knowledge - no new language to learn
+- Well-defined semantics and broad tooling support
+- Natural fit for relational data structures
+- Enables familiar filtering, joining, and aggregation patterns
+
+**Example:**
 ```python
-from arkiv import Arkiv
-from arkiv.account import NamedAccount
+# Query entities using SQL-like syntax
+results = client.arkiv.query(
+    "SELECT entity_key, payload WHERE annotations.type = 'user' AND annotations.age > 18 ORDER BY annotations.name"
+)
+```
 
-account = NamedAccount.from_wallet('Alice', wallet, 's3cret')
-client = Arkiv(provider, account = account)
+### Paging
 
-entity_key, tx_hash = client.arkiv.create_entity(
-    payload=b"Hello World!",
-    annotations={"type": "greeting", "version": 1},
-    btl = 1000
+Paging of query results is currently in development.
+
+**Requirements:**
+- Support cursor-based pagination for consistent results
+- Configurable maximum page size (a page might contain fewer entities depending on actual data used for the representation per entity)
+- Return page metadata (total count, has_next_page, cursor)
+
+**Example:**
+```python
+# Fetch first page
+page = client.arkiv.query("SELECT * FROM entities", max_page_size=100)
+
+# Fetch next page using cursor
+next_page = client.arkiv.query("SELECT * FROM entities", cursor=page.next, max_page_size=100)
+```
+
+### Sorting
+
+Querying entities should support sorting results by one or more fields.
+
+**Requirements:**
+- Sort by annotations (string and numeric)
+- Sort by metadata (owner, expires_at_block)
+- Support ascending and descending order
+- Multi-field sorting with priority
+
+**Example:**
+```python
+# SQL-style sorting
+results = client.arkiv.query(
+    "SELECT * FROM entities ORDER BY annotations.priority DESC, annotations.name ASC"
 )
+```
+
+### Projections
+
+The transfer of large entities or many entities consumes considerable bandwidth. Which information per entity is most valuable is use-case specific and should be specified by the application.
+
+**Let users decide which parts of an entity to return:**
+- **Payload** - Binary data (can be large)
+- **Annotations** - Key-value metadata
+- **Metadata** - Owner, expiration, timestamps
+
+**Current implementation:**
+The SDK already supports projections via the `fields` parameter using bitmask flags:
+```python
+from arkiv.types import PAYLOAD, ANNOTATIONS, METADATA
+
+# Fetch only annotations (minimal bandwidth)
+entity = client.arkiv.get_entity(entity_key, fields=ANNOTATIONS)
+
+# Fetch payload and metadata (skip annotations)
+entity = client.arkiv.get_entity(entity_key, fields=PAYLOAD | METADATA)
 
+# Fetch everything (fields = ALL is default)
 entity = client.arkiv.get_entity(entity_key)
-exists = client.arkiv.exists(entity_key)
 ```
 
+### Entity Existence Check
+
+Make testing whether an entity exists for a specific entity key as efficient as possible.
+
+**Current implementation:**
+```python
+exists = client.arkiv.entity_exists(entity_key)  # Returns bool
+```
+
+**Options for optimization:**
+- Dedicated lightweight RPC endpoint (current approach)
+- Fold into unified query RPC with minimal projection
+- Support batch existence checks for multiple keys
+
+**Example batch API:**
+```python
+# Check multiple entities at once
+existence_map = client.arkiv.entities_exist([key1, key2, key3])
+# Returns: {key1: True, key2: False, key3: True}
+```
+
+### Other Features
+
+- **Ownership Transfer**: The creating account is the owner of the entity.
+Only the owner can update the entity (payload, annotations, expires_at_block).
+A mechanism to transfer entity ownership should be provided.
+  ```python
+  # Proposed API
+  client.arkiv.transfer_entity(entity_key, new_owner_address)
+  ```
+
+- **Creation Flags**: Entities should support creation-time flags with meaningful defaults.
+Flags can only be set at creation and define entity behavior:
+  - **Read-only**: Once created, entity data cannot be changed by anyone (immutable)
+  - **Unpermissioned extension**: Entity lifetime can be extended by anyone, not just the owner
+  ```python
+  # Proposed API
+  client.arkiv.create_entity(
+      payload=b"data",
+      annotations={"type": "public"},
+      expires_at_block=future_block,
+      flags=EntityFlags.READ_ONLY | EntityFlags.PUBLIC_EXTENSION
+  )
+  ```
+
+- **ETH Transfers**: Arkiv chains should support ETH (or native token like GLM) transfers for gas fees and value transfer.
+  ```python
+  # Already supported via Web3.py compatibility
+  tx_hash = client.eth.send_transaction({
+      'to': recipient_address,
+      'value': client.to_wei(1, 'ether'),
+      'gas': 21000
+  })
+  ```
+
+- **Offline Entity Verification**: Provide cryptographic verification of entity data without querying the chain.
+  - Signature verification for entity authenticity
+  - Minimal trust assumptions for light clients
+
 ## Development Guide
 
 ### Branches, Versions, Changes