docs: upd api ref + 0.4.0 changelog

Author: pikulev

docs/api.md

@@ -9,165 +9,145 @@ This page provides a detailed guide to the main functions, classes, and extensib
 ### `parse`
 
 ```python
-from hario_core import parse
+from hario_core.parse import parse
 ```
 
 Parses a HAR file from a path, bytes, or file-like object and returns a validated `HarLog` model. Automatically selects the correct Pydantic model for each entry (including extensions).
 
 **Signature:**
 ```python
-def parse(
-    src: str | Path | bytes | bytearray | IO[Any],
-    *,
-    entry_model_selector: Callable[[dict[str, Any]], type[Entry]] = entry_selector,
-) -> HarLog
+def parse(src: str | Path | bytes | bytearray | IO[Any]) -> HarLog
 ```
-
 - `src`: Path, bytes, or file-like object containing HAR JSON.
-- `entry_model_selector`: Optional. Function to select the Pydantic model for each entry (default: registry-based selector).
 
 **Returns:**
 - `HarLog` — a validated Pydantic model with `.entries` (list of `Entry` or extension models).
 
 **Example:**
 ```python
-model = parse("example.har")
+har_log = parse("example.har")
 for entry in har_log.entries:
     print(entry.request.url)
 ```
 
 ---
 
-## Entry Model Registration
+### `validate`
+
+Validates a HAR dict (already loaded from JSON) and returns a `HarLog` model.
+
+**Signature:**
+```python
+def validate(har_dict: dict) -> HarLog
+```
+
+---
 
 ### `register_entry_model`
 
 Register a custom Pydantic model and detector function for new HAR entry formats (e.g., Safari, proprietary extensions).
 
 **Signature:**
 ```python
-def register_entry_model(
-    detector: Callable[[dict[str, Any]], bool],
-    model: type[Entry],
-) -> None
+def register_entry_model(detector: Callable[[dict], bool], model: type[Entry]) -> None
 ```
-
 - `detector`: Function that takes an entry dict and returns True if the model should be used.
 - `model`: Pydantic model class to use for matching entries.
 
 **Example:**
 ```python
-from hario_core.models.har_1_2 import Entry
-from pydantic import Field
+from hario_core.models import Entry
 
-class SafariEntry(Entry):
-    webkit_trace: dict = Field(alias="_webkitTrace")
+class CustomEntry(Entry):
+    x_custom: str
 
-def is_safari_entry(entry_json):
-    return "_webkitTrace" in entry_json
+def is_custom_entry(entry):
+    return "x-custom" in entry
 
-register_entry_model(is_safari_entry, SafariEntry)
+register_entry_model(is_custom_entry, CustomEntry)
+```
+
+---
+
+### `entry_selector`
+
+Selects the appropriate Entry model for a given entry dict (based on registered detectors).
+
+**Signature:**
+```python
+def entry_selector(entry_dict: dict) -> type[Entry]
 ```
 
 ---
 
 ## Data Models
 
-All core data structures are implemented as Pydantic models in `hario_core.models.har_1_2`.
+All core data structures are implemented as Pydantic models in `hario_core.models`.
 
 - `Entry`: Pydantic model for a HAR entry (fields: request, response, timings, cache, etc.).
 - `HarLog`: Pydantic model for the HAR log (fields: version, creator, entries, etc.).
+- `DevToolsEntry`: Chrome DevTools extension entry model.
 
 **Example:**
 ```python
-from hario_core.models.har_1_2 import HarLog, Entry
+from hario_core.models import HarLog, Entry
 
 har_log = HarLog.model_validate(har_json["log"])
 for entry in har_log.entries:
     assert isinstance(entry, Entry)
     print(entry.request.url)
 ```
 
-### `Transformer`
-A transformer is a function that takes a dict (parsed HAR entry) and returns a dict (possibly mutated/transformed).
-
-```python
-def my_transformer(data: dict[str, Any]) -> dict[str, Any]:
-    # mutate data
-    return data
-```
-
-### `EntryIdFn`
-A function that takes an `Entry` and returns a string ID.
-
 ---
 
-## ID Generation
+## Transformers & ID Generators
 
-### `by_field`
+### `Transformer`
+A transformer is a callable that takes a dict (parsed HAR entry) and returns a dict (possibly mutated/transformed).
 
-Returns a deterministic ID function based on specified fields of a HAR entry.
+### `set_id`
+Sets an ID field in each entry using a provided function.
 
 **Signature:**
 ```python
-def by_field(fields: list[str]) -> EntryIdFn
+def set_id(id_fn: Callable[[dict], str], id_field: str = "id") -> Transformer
 ```
 
-**Example:**
+### `by_field`
+Returns a deterministic ID function based on specified fields of a HAR entry.
+
+**Signature:**
 ```python
-from hario_core.utils import by_field
-id_fn = by_field(["request.url", "startedDateTime"])
+def by_field(fields: list[str]) -> Callable[[dict], str]
 ```
 
 ### `uuid`
-
 Returns a function that generates a random UUID for each entry.
 
 **Signature:**
 ```python
-def uuid() -> EntryIdFn
+def uuid() -> Callable[[dict], str]
 ```
 
-**Example:**
-```python
-from hario_core.utils import uuid
-id_fn = uuid()
-```
-
----
-
-## Transformers
-
-Transformers are functions that mutate or normalize HAR entry data for storage or analysis.
-
-
 ### `flatten`
-
 Flattens nested structures in a HAR entry to a flat dict with keys joined by separator. If a list is encountered, array_handler is called (default: str). Useful for exporting to CSV, analytics, or custom DB schemas.
 
 **Signature:**
 ```python
-def flatten(
-    separator: str = ".",
-    array_handler: Callable[[list, str], Any] = None,
-) -> Transformer
+def flatten(separator: str = ".", array_handler: Callable[[list, str], Any] = None) -> Transformer
 ```
 - `separator`: Separator for keys (default: '.')
 - `array_handler`: Function (lambda arr, path) -> value. Default is str(arr)
 
 **Example:**
 ```python
 def header_handler(arr, path):
-    # Each header becomes a separate key by name
     return {f"{path}.{item['name']}": item["value"] for item in arr if isinstance(item, dict) and "name" in item and "value" in item}
 
 flat_entry = flatten(array_handler=header_handler)(entry)
-# flat_entry['request.headers.user-agent'] == 'Mozilla/5.0 ...'
-# flat_entry['request.headers.:authority'] == 'test.test'
 ```
 
 ### `normalize_sizes`
-
 Normalizes negative size fields in request/response to zero.
 
 **Signature:**
@@ -176,7 +156,6 @@ def normalize_sizes() -> Transformer
 ```
 
 ### `normalize_timings`
-
 Normalizes negative timing fields in entry.timings to zero.
 
 **Signature:**
@@ -188,45 +167,88 @@ def normalize_timings() -> Transformer
 
 ## Pipeline
 
-### `Pipeline`
-
-A high-level class for processing HAR data: transforming and assigning IDs. You must pass a parsed `HarLog` object (see `parse`).
+### `PipelineConfig`
+Configuration for the Pipeline processor.
 
 **Signature:**
 ```python
-class Pipeline:
-    def __init__(
-        self,
-        id_fn: EntryIdFn,
-        id_field: str = "id",
-        transformers: Sequence[Transformer] = (),
-    ) -> None
-    def process(self, har_log: HarLog) -> list[dict[str, Any]]
+from hario_core.transform import PipelineConfig
+
+config = PipelineConfig(
+    batch_size=1000,                # entries per batch
+    processing_strategy="process", # "sequential", "thread", "process", "async"
+    max_workers=4                   # number of parallel workers (if applicable)
+)
 ```
 
-- `id_fn`: Function to generate an ID for each entry.
-- `id_field`: Field name for the generated ID (default: "id").
-- `transformers`: List of transformer functions to apply to each entry.
+- `batch_size`: int, default 20000
+- `processing_strategy`: str, one of "sequential", "thread", "process", "async"
+- `max_workers`: int | None, number of parallel workers (for thread/process)
 
 ---
 
-## Example: Full Pipeline
+### `Pipeline`
+A high-level class for processing HAR entry dicts: transforming and assigning IDs.
 
+**Signature:**
 ```python
-from hario_core import Pipeline, by_field, flatten, normalize_sizes, parse
+from hario_core.transform import Pipeline, PipelineConfig
 
 pipeline = Pipeline(
-    id_fn=by_field(["request.url", "startedDateTime"]),
-    transformers=[flatten(), normalize_sizes()],
+    transformers=[...],
+    config=PipelineConfig(...)
 )
 
-model = parse("example.har")
-result_dict = pipeline.process(model)
+results = pipeline.process(entries)  # entries: list[dict]
+```
+- `transformers`: List of transformer functions to apply to each entry.
+- `config`: PipelineConfig instance (optional, default: sequential, batch_size=20000)
+- `process(entries)`: entries must be a list of dicts (e.g., from HarLog.model_dump()["entries"])
+
+---
+
+### Example: Full Pipeline
+
+```python
+from hario_core.parse import parse
+from hario_core.transform import Pipeline, PipelineConfig, by_field, flatten, normalize_sizes, set_id
+
+har_log = parse("example.har")
+entries = har_log.model_dump()["entries"]
+
+pipeline = Pipeline([
+    set_id(by_field(["request.url", "startedDateTime"])),
+    flatten(),
+    normalize_sizes(),
+])
+results = pipeline.process(entries)
+```
 
-for entry in result_dict:
-    print(entry["id"], entry["request"]["url"])
+---
+
+### Example: Parallel Processing with Custom Batch Size and Workers
+
+```python
+from hario_core.transform import Pipeline, PipelineConfig, flatten
+
+config = PipelineConfig(
+    processing_strategy="process",  # or "thread"
+    batch_size=20,                  # process 20 entries per batch
+    max_workers=6                   # use 6 parallel workers
+)
+
+pipeline = Pipeline([
+    flatten(),
+], config=config)
+results = pipeline.process(entries)
 ```
 
+#### Available Processing Strategies
+- `sequential` (default): Process entries one by one in a single thread. Best for small datasets or debugging.
+- `thread`: Parallel processing using threads. Useful for I/O-bound tasks or when GIL is not a bottleneck.
+- `process`: Parallel processing using multiple processes. Recommended for CPU-bound tasks and large datasets.
+- `async`: Asynchronous processing (if your transformers support async). For advanced use cases with async I/O.
+
 ---
 
 ## Chrome DevTools Extension Example
@@ -235,8 +257,8 @@ You can use the Chrome DevTools HAR extension models to validate and work with H
 
 **Example:**
 ```python
-from hario_core.models.extensions.chrome_devtools import DevToolsEntry
-from hario_core.models.har_1_2 import HarLog
+from hario_core.models import DevToolsEntry
+from hario_core.models import HarLog
 
 # Suppose har_json is a dict loaded from a Chrome DevTools HAR file
 har_log = HarLog.model_validate(har_json["log"])

docs/changelog.md

@@ -1,5 +1,15 @@
 # Changelog
 
+### v0.4.0
+- BREAKING: Pipeline now requires a list of transformers and a PipelineConfig instance (no more id_fn/id_field in constructor).
+- BREAKING: Pipeline.process now expects a list of dicts (e.g., from HarLog.model_dump()["entries"]).
+- New: PipelineConfig class for configuring batch size, processing strategy (sequential/thread/process/async), and max_workers.
+- New: Parallel and batch processing strategies for large HAR files (process, thread, async).
+- New: Benchmarks and benchmarking scripts for pipeline performance (see `benchmarks/`).
+- New: All transformers (`flatten`, `normalize_sizes`, `normalize_timings`, `set_id`) are now implemented as picklable callable classes, fully compatible with multiprocessing.
+- New: `set_id` transformer for assigning IDs to entries using any function (e.g., by_field, uuid).
+- Internal: Test suite and samples updated for new API and real-world HAR compatibility.
+
 ### v0.3.1
 - FIX real-world HAR compatibility: made nested fields like `postData.params` optional in models, so parsing DevTools and other real HAR files is more robust.
 - All test samples are now based on real HAR data with valid `pages` and `pageref` links.