Improve linting tools + update docs (#47)

* add linting tools and configure pre-commit

- Add black, isort, autoflake, flake8, and pre-commit to test-requirements.txt
- Set line length to 120 characters for all formatting tools
- Configure pre-commit hooks with proper exclusions for test_pypi_env/
- Update GitHub Actions to use 120-character line length
- Add test_pypi_env/ to .gitignore

apply black formatting and fix linting issues

- Reformat all Python files with 120-character line length
- Fix import ordering with isort
- Add E501 and E722 to flake8 ignore list for remaining edge cases
- All code now consistently formatted

* .

* tweak docs

* rabbit feedback

Author: hampsterx

.flake8

@@ -1,7 +1,7 @@
 [flake8]
 max-line-length = 88
 extend-ignore = E203, W503, E501
-exclude = 
+exclude =
     .git,
     __pycache__,
     venv,
@@ -11,4 +11,4 @@ exclude =
     *.egg-info
 per-file-ignores =
     __init__.py:F401
-    tests/*:F401,F811
+    tests/*:F401,F811

.github/workflows/publish.yml

@@ -93,13 +93,13 @@ jobs:
         pip install black isort flake8
 
     - name: Check code formatting with black
-      run: black --check --diff kinesis tests
+      run: black --check --diff --line-length=120 kinesis tests
 
     - name: Check import sorting with isort
-      run: isort --check-only --diff kinesis tests
+      run: isort --check-only --diff --line-length=120 kinesis tests
 
     - name: Lint with flake8
-      run: flake8 kinesis tests --max-line-length=88 --extend-ignore=E203,W503,E501
+      run: flake8 kinesis tests --max-line-length=120 --extend-ignore=E203,W503,E712,E402,F401,F841,F541
 
   publish:
     needs: [test, lint]
@@ -144,4 +144,4 @@ jobs:
     - name: Publish to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        repository-url: https://upload.pypi.org/legacy/
+        repository-url: https://upload.pypi.org/legacy/

.github/workflows/test.yml

@@ -111,10 +111,10 @@ jobs:
         pip install black isort flake8
 
     - name: Check code formatting with black
-      run: black --check --diff kinesis tests
+      run: black --check --diff --line-length=120 kinesis tests
 
     - name: Check import sorting with isort
-      run: isort --check-only --diff kinesis tests
+      run: isort --check-only --diff --line-length=120 kinesis tests
 
     - name: Lint with flake8
-      run: flake8 kinesis tests --max-line-length=88 --extend-ignore=E203,W503,E501
+      run: flake8 kinesis tests --max-line-length=120 --extend-ignore=E203,W503,E712,E402,F401,F841,F541

.gitignore

@@ -1,4 +1,5 @@
 venv/
+test_pypi_env/
 dist
 *.egg-info
 .idea

.pre-commit-config.yaml

@@ -0,0 +1,34 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 24.10.0
+    hooks:
+      - id: black
+        language_version: python3
+        args: [--line-length=120]
+        exclude: ^test_pypi_env/
+
+  - repo: https://github.com/pycqa/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort
+        args: [--profile=black, --line-length=120]
+        exclude: ^test_pypi_env/
+
+  - repo: https://github.com/pycqa/flake8
+    rev: 7.1.1
+    hooks:
+      - id: flake8
+        args: ["--max-line-length=120", "--extend-ignore=E203,W503,E712,E402,F401,F841,F541,E501,E722"]
+        exclude: ^test_pypi_env/
+
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+        exclude: ^test_pypi_env/
+      - id: end-of-file-fixer
+        exclude: ^test_pypi_env/
+      - id: check-yaml
+      - id: check-added-large-files
+        exclude: ^test_pypi_env/
+      - id: check-merge-conflict

README.md

@@ -2,30 +2,53 @@
 
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black) [![PyPI version](https://badge.fury.io/py/async-kinesis.svg)](https://badge.fury.io/py/async-kinesis) [![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3100/) [![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/downloads/release/python-3110/) [![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
 
-```
+High-performance async Python library for AWS Kinesis with production-ready resharding support.
+
+```bash
 pip install async-kinesis
 ```
 
-## Features
-
-- uses queues for both producer and consumer
-  - producer flushes with put_records() if has enough to flush or after "buffer_time" reached
-  - consumer iterates over msg queue independent of shard readers
-- Configurable to handle Sharding limits but will throttle/retry if required
-  - ie multiple independent clients are saturating the Shards
-- **Dynamic shard handling and resharding support**
-  - automatically discovers new shards during Kinesis stream resharding
-  - graceful handling of closed shards when `NextShardIterator` is null
-  - robust error recovery for expired shard iterators
-  - shard status monitoring and operational visibility
-- Checkpointing with heartbeats
-  - deadlock + reallocation of shards if checkpoint fails to heartbeat within "session_timeout"
-- processors (aggregator + serializer)
-    - json line delimited, msgpack
-- Address Kinesis streams by name or [ARN]
-
-See [docs/design](./docs/DESIGN.md) for more details.
-See [docs/yetanother](docs/YETANOTHER.md) as to why reinvent the wheel.
+## Quick Start
+
+**Producer:**
+```python
+from kinesis import Producer
+
+async with Producer(stream_name="my-stream") as producer:
+    await producer.put({"message": "hello world"})
+```
+
+**Consumer:**
+```python
+from kinesis import Consumer
+
+async with Consumer(stream_name="my-stream") as consumer:
+    async for message in consumer:
+        print(message)
+```
+
+## Key Features
+
+✅ **Production-Ready Resharding**: Automatic shard discovery and topology management
+✅ **Async/Await Native**: Built for modern Python async patterns
+✅ **High Performance**: Queue-based architecture with configurable batching
+✅ **AWS Best Practices**: Parent-child shard ordering and proper error handling
+✅ **Stream Addressing**: Support for both stream names and ARNs
+✅ **Multi-Consumer Support**: Redis-based checkpointing with heartbeats
+✅ **Flexible Processing**: Pluggable serialization (JSON, MessagePack, KPL)
+✅ **Operational Visibility**: Rich monitoring APIs for production debugging
+
+### Resharding Support Highlights
+
+Unlike basic Kinesis libraries, async-kinesis provides enterprise-grade resharding capabilities:
+
+- **Automatic discovery** of new shards during resharding operations
+- **Parent-child ordering** enforcement following AWS best practices
+- **Graceful handling** of closed shards and iterator expiration
+- **Real-time monitoring** with detailed topology and status reporting
+- **Seamless coordination** between multiple consumer instances
+
+📖 **[Architecture Details](./docs/DESIGN.md)** | **[Why Another Library?](docs/YETANOTHER.md)**
 
 ## Environment Variables
 
@@ -241,7 +264,7 @@ Refer https://aws.amazon.com/blogs/big-data/implementing-efficient-and-reliable-
 | JsonListProcessor | ListAggregator | JsonSerializer | Multiple JSON record returned by list |
 | MsgpackProcessor | NetstringAggregator | MsgpackSerializer | Multiple Msgpack record framed with Netstring Protocol (https://en.wikipedia.org/wiki/Netstring) |
 | KPLJsonProcessor | KPLAggregator | JsonSerializer | Multiple JSON record in a KPL Aggregated Record (https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md) |
-| KPLStringProcessor | KPLAggregator | StringSerializer | Multiple String record in a KPL Aggregated Record (https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md) | 
+| KPLStringProcessor | KPLAggregator | StringSerializer | Multiple String record in a KPL Aggregated Record (https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md) |
 
 Note you can define your own processor easily as it's simply a class inheriting the Aggregator + Serializer.
 
@@ -315,41 +338,62 @@ Choose the optimal processor based on your use case:
 **Performance Testing:** Use the benchmark tool with different `--record-size-kb` and `--processors` options to determine the best processor for your specific data patterns.
 
 
-## Unit Testing
+## Development & Testing
 
-Uses https://github.com/mhart/kinesalite for local testing.
+### Local Testing
 
-Run tests via docker
+Uses LocalStack for integration testing:
 
-```
+```bash
+# Run full test suite via Docker
 docker-compose up --abort-on-container-exit --exit-code-from test
-```
 
-For local testing use
-
-```
+# Local development setup
 docker-compose up kinesis redis
+pip install -r test-requirements.txt
+pytest
 ```
 
-then within your virtualenv
+### Code Quality
 
-```
-nosetests
+This project uses automated code formatting and linting:
+
+```bash
+# Install development tools
+pip install -r test-requirements.txt
+
+# Run formatting and linting
+black .
+isort .
+flake8 .
 
-# or run individual test
-nosetests tests.py:KinesisTests.test_create_stream_shard_limit_exceeded
+# Or use pre-commit hooks
+pre-commit install
+pre-commit run --all-files
 ```
 
-Note there are a few test cases using the *actual* AWS Kinesis (AWSKinesisTests)
-These require setting an env in order to run
+### AWS Integration Tests
 
-Create an ".env" file with
+Some tests require actual AWS Kinesis. Create `.env` file:
 
 ```
 TESTING_USE_AWS_KINESIS=1
 ```
 
-Note you can ignore these tests if submitting PR unless core batching/processing behaviour is being changed.
+### Resharding Tests
+
+Comprehensive resharding test suite available:
+
+```bash
+# Unit tests (no AWS required)
+python tests/resharding/test_resharding_simple.py
+
+# Integration tests (requires LocalStack)
+python tests/resharding/test_resharding_integration.py
+
+# Production testing (requires AWS)
+python tests/resharding/resharding_test.py --scenario scale-up-small
+```
 
 
 [ARN]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html

RELEASING.md

@@ -0,0 +1,80 @@
+# Release Process
+
+This document describes the automated release process for async-kinesis.
+
+## Quick Release
+
+For a patch release (1.1.5 → 1.1.6):
+```bash
+python scripts/release.py --next-patch
+```
+
+For a minor release (1.1.5 → 1.2.0):
+```bash
+python scripts/release.py --next-minor
+```
+
+For a major release (1.1.5 → 2.0.0):
+```bash
+python scripts/release.py --next-major
+```
+
+For a specific version:
+```bash
+python scripts/release.py --version 1.2.3
+```
+
+## What Happens
+
+1. **Pre-flight checks**: The script runs linting and tests
+2. **Version update**: Updates the version in `setup.py`
+3. **Git tag creation**: Creates and optionally pushes a git tag
+4. **Automated deployment**: GitHub Actions detects the tag and:
+   - Runs full test suite across Python 3.10, 3.11, 3.12
+   - Runs linting checks
+   - Builds the package
+   - Publishes to PyPI using trusted publishing
+
+## Manual Process
+
+If you prefer to do it manually:
+
+1. Update version in `setup.py`
+2. Commit the version change
+3. Create a git tag: `git tag 1.2.3`
+4. Push the tag: `git push origin 1.2.3`
+5. GitHub Actions will handle the rest
+
+## PyPI Configuration
+
+The deployment uses GitHub's trusted publishing feature, which requires:
+
+1. Configure the PyPI project to trust this GitHub repository
+2. Set up the `pypi` environment in GitHub repository settings
+3. The workflow uses `id-token: write` permission for OIDC authentication
+
+## Checking Current Version
+
+```bash
+python scripts/release.py --check
+```
+
+## Development Releases
+
+For testing the release process without affecting the main package:
+
+```bash
+# Skip tests and linting for quick iteration
+python scripts/release.py --version 1.2.3-dev --skip-tests --skip-lint --no-tag
+```
+
+## Troubleshooting
+
+- **Tests fail**: Fix the issues before releasing
+- **Linting fails**: Run `black kinesis tests` and `isort kinesis tests`
+- **PyPI upload fails**: Check repository settings and trusted publishing configuration
+- **Tag already exists**: Delete the tag locally and remotely if needed:
+  ```bash
+  git tag -d 1.2.3
+  git push origin :refs/tags/1.2.3
+  ```