Feat : Add ClickHouse Backend Support

.github/copilot-instructions.md

@@ -0,0 +1,118 @@
+# Cryptofeed AI Coding Instructions
+
+## Project Overview
+Cryptofeed is a normalized cryptocurrency exchange feed handler library that aggregates real-time market data (trades, order books, tickers) from 40+ exchanges via WebSocket and REST APIs, exposing them through a unified callback-based interface.
+
+## Architecture
+
+### Core Components
+- **FeedHandler** (`cryptofeed/feedhandler.py`): Main orchestrator managing multiple feeds, async event loop, and lifecycle (startup/shutdown signals)
+- **Feed** (`cryptofeed/feed.py`): Per-exchange handler inheriting from Exchange, manages channel subscriptions, reconnection logic, and message parsing
+- **Exchange** (`cryptofeed/exchange.py`): Base class defining interface: `websocket_channels`, `rest_endpoints`, symbol mapping, order validation
+- **Connection** (`cryptofeed/connection.py`): HTTP/WebSocket client abstractions (`HTTPSync`, `HTTPAsyncConn`, `WSAsyncConn`) with raw data collection hooks
+- **Types** (`cryptofeed/types.pyx`): Cython-compiled data classes (Trade, Ticker, OrderBook, Candle, etc.) for performance-critical paths; use `Decimal` for prices
+
+### Data Flow
+1. FeedHandler adds feeds → Feed initializes symbol mapping → WebSocket/REST connections open
+2. Messages arrive → Feed's channel-specific parser (e.g., `parse_l2_book`) deserializes and normalizes
+3. Normalized objects (Trade, Ticker) passed to user-provided async/sync callbacks via Callback wrapper
+4. Optional backends (Kafka, Redis, Postgres, etc.) persist data
+
+## Key Patterns & Conventions
+
+### Exchange Implementation
+- **Inherit from Feed, not Exchange directly** (Feed extends Exchange)
+- Define class attributes:
+  - `id = 'EXCHANGE_NAME'` (from `defines.py`)
+  - `websocket_endpoints = [WebsocketEndpoint(...)]` / `rest_endpoints = [RestEndpoint(...)]`
+  - `websocket_channels = {TRADES: 'channel_name', L2_BOOK: '...'}` (map cryptofeed types to exchange channel names)
+- Implement `_parse_symbol_data(cls, data: dict) -> Tuple[Dict, Dict]` to normalize exchange symbols to Symbol objects
+- Implement channel parsers (e.g., `async def _parse_trades(self, msg, receipt_timestamp)`) that deserialize exchange JSON into cryptofeed types
+- Use `self.normalized_symbol_mapping` (str → Symbol) for lookups
+
+### Symbol & Type System
+- Use `Symbol` objects (not strings) for type safety: `Symbol(base_asset, quote_asset, type=SPOT|FUTURES|PERPETUAL, expiry_date=None)`
+- All prices/amounts are `Decimal`, never float (for precision)
+- Timestamp normalization: exchanges often send milliseconds; normalize to seconds with `cls.timestamp_normalize()`
+
+### Callbacks & Async Patterns
+- Callbacks are user functions called with signature: `callback(data_object, receipt_timestamp)`
+- Can be `async def` or sync; wrapper in `Callback` handles execution in executor if needed
+- Lightweight callbacks preferred; FeedHandler is paused during callback execution
+
+### Configuration
+- Config loaded from `config.yaml` or passed as dict to FeedHandler/Exchange constructors
+- API keys stored under exchange IDs in lowercase (e.g., `config['binance']` with `.key_id`, `.key_secret`, `.key_passphrase`)
+- Supports sandbox mode via `sandbox=True` parameter
+
+## Developer Workflow
+
+### Testing
+- Run unit tests: `pytest tests/unit/` 
+- Integration tests with live API: `pytest tests/integration/` (requires API keys)
+- No formal test suite runner; use pytest directly
+
+### Code Style & Linting
+- **Type annotations required** on public methods (see existing exchanges for examples)
+- Prefer long lines (130+ chars) over line wrapping; isort for import formatting
+- Run `flake8` before PRs to catch style issues
+- Import order: stdlib → third-party → `cryptofeed`
+
+### Adding an Exchange
+1. Create `cryptofeed/exchanges/new_exchange.py`
+2. Implement Feed subclass with required attributes and parsers
+3. Add sample data file in `sample_data/` for testing (format: `EXCHANGE_NAME.0`, `EXCHANGE_NAME.ws.1.0`)
+4. Update `cryptofeed/exchanges/__init__.py` to export the class
+5. Reference API docs in PR; include link in README
+
+### Cython Types
+- `types.pyx` compiled with `cythonize` in setup.py; edit as `.pyx`, not `.py`
+- Use `@cython.freelist(n)` for frequently allocated types (Trade, Candle)
+- Compile with: `python setup.py build_ext --inplace`
+
+### Backend Implementation
+"Backends" in cryptofeed are not just database connectors—they're **callback wrappers** (middleware) that can aggregate, transform, persist, or format data from any channel. They follow two patterns:
+
+**Pattern 1: Data Persistence** (e.g., Kafka, Postgres, Redis)
+- Inherit from a base backend class or implement async callback interface
+- Implement `async def __call__(self, data, receipt_timestamp)` to receive normalized objects (Trade, Ticker, OrderBook, etc.)
+- Batch/buffer writes for efficiency; handle connection lifecycle (start/stop)
+- Example: `cryptofeed/backends/kafka.py` buffers trades and flushes to Kafka
+
+**Pattern 2: Data Transformation/Aggregation** (e.g., OHLCV, Throttle, Renko)
+- Inherit from `AggregateCallback` in `cryptofeed/backends/aggregate.py`
+- Wrap and chain to other handlers: `self.handler(data, receipt_timestamp)`
+- Aggregate across time windows or price movements before forwarding
+- Example: `Throttle` allows 1 callback per time window; `OHLCV` aggregates trades into OHLCV bars
+
+**Adding a Backend:**
+1. Create `cryptofeed/backends/my_backend.py`
+2. Implement class with `async def __call__(self, data, receipt_timestamp)` 
+3. Optional: add `start()` and `stop()` lifecycle methods for resource management
+4. Pass to FeedHandler: `fh.add_feed(Exchange(..., callbacks={TRADES: MyBackend(...), TICKER: MyBackend(...)}))`
+5. Backend receives each data type independently; use `hasattr(data, 'attribute')` to handle multiple types if needed
+
+### Import Structure
+- Exchange-specific symbols mapped through `cryptofeed/defines.py` (BINANCE, KRAKEN, etc.)
+- Channel constants: L2_BOOK, L3_BOOK, TRADES, TICKER, FUNDING, LIQUIDATIONS, CANDLES, etc.
+- Callback types in `callback.py` (TradeCallback, TickerCallback, etc.)
+
+## Cross-Component Communication
+- **Symbol resolution**: Feed queries Symbols registry (singleton) to map exchange symbols ↔ normalized formats
+- **Raw data collection**: Optional raw_data_callback on Connection class intercepts ALL HTTP/WebSocket traffic for debugging
+- **Backends**: Decouple data persistence; passed as callbacks (e.g., KafkaBackend implements async write logic)
+- **NBBO**: Synthetic feed aggregating best bids/asks across multiple exchanges (see `nbbo.py` and `examples/demo_nbbo.py`)
+
+## Common Pitfalls
+- Forgetting `async def` on parsers that call I/O operations
+- Using float for prices instead of Decimal → precision loss
+- Not normalizing exchange timestamps to UTC seconds
+- Missing `checkpoint_validation` or `cross_check` flags for exchanges with weak message ordering
+- Hardcoding depth limits instead of checking exchange's `valid_depths` parameter
+
+## Quick Reference Files
+- Channel definitions: `cryptofeed/defines.py`
+- Callback signatures: `cryptofeed/callback.py`
+- Example integration: `examples/demo.py`, `examples/demo_binance_authenticated.py`
+- Configuration template: `config.yaml`
+- Documentation index: `docs/README.md`

CHANGES.md

@@ -1,5 +1,9 @@
 ## Changelog
 
+### 2.4.2 (TBD)
+ * Feature: ClickHouse backend support for storing real-time cryptocurrency data
+ * Update: Added copilot-instructions.md
+
 ### 2.4.1 (2025-02-08)
  * Update: Added `is_data_json` to `write()` in `HTTPSync` from `connection.py` to support JSON payloads (#1071)
  * Bugfix: Handle empty nextFundingRate in OKX

INSTALL.md

@@ -31,6 +31,11 @@ for the exhaustive list of these *extra* dependencies.
 
          pip install --user --upgrade cryptofeed[arctic]
 
+* ClickHouse backend  
+  To install Cryptofeed along with [ClickHouse](https://clickhouse.com/) in one bundle:
+
+         pip install --user --upgrade cryptofeed[clickhouse]
+
 * Google Cloud Pub / Sub backend
 
          pip install --user --upgrade cryptofeed[gcp_pubsub]
@@ -65,8 +70,9 @@ for the exhaustive list of these *extra* dependencies.
          pip install --user --upgrade cryptofeed[zmq]
 
 If you have a problem with the installation/hacking of Cryptofeed, you are welcome to:
-* open a new issue: https://github.com/bmoscon/cryptofeed/issues/
+
+* open a new issue: <https://github.com/bmoscon/cryptofeed/issues/>
 * join us on Slack: [cryptofeed-dev.slack.com](https://join.slack.com/t/cryptofeed-dev/shared_invite/enQtNjY4ODIwODA1MzQ3LTIzMzY3Y2YxMGVhNmQ4YzFhYTc3ODU1MjQ5MDdmY2QyZjdhMGU5ZDFhZDlmMmYzOTUzOTdkYTZiOGUwNGIzYTk)
-* or on GitHub Discussion: https://github.com/bmoscon/cryptofeed/discussions
+* or on GitHub Discussion: <https://github.com/bmoscon/cryptofeed/discussions>
 
 Your Pull Requests are also welcome, even for minor changes.

LICENSE

@@ -1,4 +1,6 @@
-Copyright (C) 2017-2025 Bryant Moscon - [email protected]
+# License
+
+Copyright (C) 2017-2025 Bryant Moscon - <[email protected]>
 
  Permission is hereby granted, free of charge, to any person obtaining a copy
  of this software and associated documentation files (the "Software"), to
@@ -17,7 +19,7 @@ Copyright (C) 2017-2025 Bryant Moscon - [email protected]
 
  3. The end-user documentation included with the redistribution, if any, must
     include the following acknowledgment: "This product includes software
-    developed by Bryant Moscon (http://www.bryantmoscon.com/)", in the same
+    developed by Bryant Moscon (<http://www.bryantmoscon.com/>)", in the same
     place and form as other third-party acknowledgments. Alternately, this
     acknowledgment may appear in the software itself, in the same form and
     location as other such third-party acknowledgments.
@@ -27,7 +29,6 @@ Copyright (C) 2017-2025 Bryant Moscon - [email protected]
     other dealings in this Software without prior written authorization from
     the author.
 
-
  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL

README.md

@@ -1,4 +1,5 @@
 # Cryptocurrency Exchange Feed Handler
+
 [![License](https://img.shields.io/badge/license-XFree86-blue.svg)](LICENSE)
 ![Python](https://img.shields.io/badge/Python-3.8+-green.svg)
 [![PyPi](https://img.shields.io/badge/PyPi-cryptofeed-brightgreen.svg)](https://pypi.python.org/pypi/cryptofeed)
@@ -37,7 +38,7 @@ Handles multiple cryptocurrency exchange data feeds and returns normalized and s
 * [Huobi](https://www.hbg.com/)
 * [Huobi DM](https://www.huobi.com/en-us/markets/hb_dm/)
 * Huobi Swap (Coin-M and USDT-M)
-* [Independent Reserve](https://www.independentreserve.com/) 
+* [Independent Reserve](https://www.independentreserve.com/)
 * [Kraken](https://www.kraken.com/)
 * [Kraken Futures](https://futures.kraken.com/)
 * [KuCoin](https://www.kucoin.com/)
@@ -48,7 +49,6 @@ Handles multiple cryptocurrency exchange data feeds and returns normalized and s
 * [ProBit](https://www.probit.com/)
 * [Upbit](https://sg.upbit.com/home)
 
-
 ## Basic Usage
 
 Create a FeedHandler object and add subscriptions. For the various data channels that an exchange supports, you can supply callbacks for data events, or use provided backends (described below) to handle the data for you. Start the feed handler and you're done!
@@ -76,10 +76,8 @@ fh.run()
 
 Please see the [examples](https://github.com/bmoscon/cryptofeed/tree/master/examples) for more code samples and the [documentation](https://github.com/bmoscon/cryptofeed/blob/master/docs/README.md) for more information about the library usage.
 
-
 For an example of a containerized application using cryptofeed to store data to a backend, please see [Cryptostore](https://github.com/bmoscon/cryptostore).
 
-
 ## National Best Bid/Offer (NBBO)
 
 Cryptofeed also provides a synthetic [NBBO](examples/demo_nbbo.py) (National Best Bid/Offer) feed that aggregates the best bids and asks from the user specified feeds.
@@ -123,18 +121,19 @@ Cryptofeed supports the following channels from exchanges:
 * BALANCES - Updates on wallet funds
 * FILLS - User's executed trades
 
-
 ## Backends
 
 Cryptofeed supports `backend` callbacks that will write directly to storage or other interfaces.
 
 Supported Backends:
+
 * Redis (Streams and Sorted Sets)
 * [Arctic](https://github.com/manahl/arctic)
 * ZeroMQ
 * UDP Sockets
 * TCP Sockets
 * Unix Domain Sockets
+* [ClickHouse](https://clickhouse.com/)
 * [InfluxDB v2](https://github.com/influxdata/influxdb)
 * MongoDB
 * Kafka
@@ -144,36 +143,40 @@ Supported Backends:
 * GCP Pub/Sub
 * [QuestDB](https://questdb.io/)
 
-
 ## Installation
 
 **Note:** cryptofeed requires Python 3.8+
 
 Cryptofeed can be installed from PyPi. (It's recommended that you install in a virtual environment of your choosing).
 
-    pip install cryptofeed
+```bash
+pip install cryptofeed
+```
 
 Cryptofeed has optional dependencies, depending on the backends used. You can install them individually, or all at once. To install Cryptofeed along with all its optional dependencies in one bundle:
 
-    pip install cryptofeed[all]
+```bash
+pip install cryptofeed[all]
+```
 
 If you wish to clone the repository and install from source, run this command from the root of the cloned repository.
 
-    python setup.py install
+```bash
+python setup.py install
+```
 
 Alternatively, you can install in 'edit' mode (also called development mode):
 
-    python setup.py develop
+```bash
+python setup.py develop
+```
 
 See more discussion of package installation in [INSTALL.md](https://github.com/bmoscon/cryptofeed/blob/master/INSTALL.md).
 
-
-
 ## Rest API
 
 Cryptofeed supports some REST interfaces for retrieving real-time and historical data, as well as order placement and account management. These are integrated into the exchange classes directly. You can view the supported methods by calling the `info()` method on any exchange. The methods for interacting with the exchange RET endpoints exist in two flavors, the synchronous methods (suffixed with `_sync`) as well as the asynchronous which can be utilized with asyncio. For more information see the [documentation](docs/rest.md).
 
-
 ## Future Work
 
 There are a lot of planned features, new exchanges, etc planned! If you'd like to discuss ongoing development, please join the [discord](https://discord.gg/zaBYaGAYfR) or open a thread in the [discussions](https://github.com/bmoscon/cryptofeed/discussions) in GitHub.