Feature/smart ohlcv conversion (#24)

feat: add smart OHLCV conversion with automatic detection and analysis (#24)

## Overview
Comprehensive enhancement of OHLCV data handling with intelligent automatic conversion, format detection, and market analysis capabilities.

## Core Changes

### New DataConverter Module (`src/pynecore/core/data_converter.py`)
- Automatic format detection for CSV, JSON, and TXT files
- Smart symbol and provider detection from filename patterns
  - Supports exchanges: BINANCE, BYBIT, CAPITALCOM, etc.
  - Handles forex pairs and crypto symbols
  - Complex pattern matching (e.g., `ccxt_BYBIT_BTC_USDT.csv`)
- Automatic SymInfo configuration file generation

### Enhanced OHLCV File Processing (`src/pynecore/core/ohlcv_file.py`)
- 1080 lines changed with major enhancements:
- **Tick Size Detection**: Multi-method analysis using histogram clustering
- **Trading Hours Detection**: Automatic detection from trading data
  - Distinguishes 24/7 crypto from business hours
  - Smart data sufficiency checks
- **Interval Auto-Correction**: Fixes gaps at data start
- **TXT File Support**: Custom parser replacing csv module
- **Improved CSV Output**: Better formatting for readability
- **Refactored Timestamp Parsing**: 80+ lines of duplication removed

### CLI Command Updates
- **data.py** (201 lines changed):
  - `convert-from`: Auto-detects symbol, provider, timeframe
  - `convert-to`: Accepts direct file paths
  - Removed redundant parameter requirements
- **run.py**: Automatic conversion of non-OHLCV files

### Test Coverage
- **test_001_ohlcv_file.py**: 898 lines added for OHLCV enhancements
- **test_004_data_converter.py**: 297 lines of new tests
- **test_005_symbol_type_detection.py**: 229 lines for symbol detection

## Minor Fixes
- pytest.ini: Fixed test discovery glob pattern
- Removed redundant type checks and validation
- Cast buffer in struct.unpack to fix IDE warnings

## Documentation
- Updated CLI documentation for data and run commands
- Added examples for automatic detection features

## Breaking Changes
- `pyne data convert-from`: Removed timeframe parameter (auto-detected)
- `pyne data convert-to`: Changed to accept direct file paths

Author: wallneradam

docs/cli/data.md

@@ -140,25 +140,33 @@ PyneCore uses a binary format (`.ohlcv`) for storing OHLCV data efficiently. How
 
 ### Converting to Other Formats
 
-The `convert-to` command converts PyneCore format to CSV or JSON:
+The `convert-to` command converts PyneCore OHLCV format to CSV or JSON:
 
 ```bash
-pyne data convert-to PROVIDER [OPTIONS]
+pyne data convert-to OHLCV_FILE [OPTIONS]
 ```
 
+Where `OHLCV_FILE` is the path to the OHLCV file to convert.
+
 Options:
-- `--symbol`, `-s`: Symbol to convert
-- `--timeframe`, `-tf`: Timeframe in TradingView format
-- `--format`, `-f`: Output format (csv, json)
+- `--format`, `-f`: Output format (csv or json, default: csv)
 - `--as-datetime`, `-dt`: Save timestamp as datetime instead of UNIX timestamp
 
+The command automatically:
+- Adds `.ohlcv` extension if not specified
+- Creates output file with the same name but different extension
+- Looks in `workdir/data/` if only filename is provided
+
 Example:
 ```bash
-# Convert Bitcoin data to CSV
-pyne data convert-to ccxt --symbol "BINANCE:BTC/USDT" --timeframe "1D" --format "csv"
+# Convert OHLCV file to CSV
+pyne data convert-to BTCUSDT_1D.ohlcv
+
+# Convert to JSON with human-readable dates
+pyne data convert-to BTCUSDT_1D.ohlcv --format json --as-datetime
 
-# Convert with human-readable dates
-pyne data convert-to ccxt --symbol "BINANCE:BTC/USDT" --timeframe "1D" --format "csv" --as-datetime
+# Short form (extension optional)
+pyne data convert-to BTCUSDT_1D -f csv -dt
 ```
 
 ### Converting from Other Formats
@@ -172,21 +180,44 @@ pyne data convert-from FILE_PATH [OPTIONS]
 Where `FILE_PATH` is the path to the CSV or JSON file to convert.
 
 Options:
-- `--provider`, `-p`: Data provider name (can be any name, defaults to "custom")
-- `--symbol`, `-s`: Symbol name
-- `--timeframe`, `-tf`: Timeframe in TradingView format
-- `--fmt`, `-f`: Input format (csv, json) - defaults to the file extension if not specified
+- `--provider`, `-p`: Data provider name (defaults to auto-detected from filename)
+- `--symbol`, `-s`: Symbol name (defaults to auto-detected from filename)
 - `--timezone`, `-tz`: Timezone of the timestamps (defaults to UTC)
 
+**Automatic Detection Features:**
+- **Symbol Detection**: The command automatically detects symbols from common filename patterns
+- **Provider Detection**: Recognizes provider names in filenames (BINANCE, BYBIT, CAPITALCOM, etc.)
+- **Format Support**: Supports CSV and JSON files, auto-detected from file extension
+
+**Filename Pattern Examples:**
+- `BTCUSDT.csv` → Symbol: BTC/USDT
+- `EUR_USD.csv` → Symbol: EUR/USD  
+- `ccxt_BYBIT_BTC_USDT.csv` → Symbol: BTC/USDT, Provider: bybit
+- `BINANCE_ETHUSDT_1h.csv` → Symbol: ETH/USDT, Provider: binance
+- `capitalcom_EURUSD.csv` → Symbol: EUR/USD, Provider: capitalcom
+
 Example:
 ```bash
-# Convert CSV to PyneCore format
-pyne data convert-from ./data/btcusd.csv --symbol "CUSTOM:BTC/USD" --timeframe "1D"
+# Convert CSV with automatic detection
+pyne data convert-from ./data/BTCUSDT.csv  # Auto-detects BTC/USDT
+
+# Override auto-detected values if needed
+pyne data convert-from ./data/btcusd.csv --symbol "BTC/USD" --provider "kraken"
 
 # Convert with timezone specification
-pyne data convert-from ./data/eurusd.csv --symbol "CUSTOM:EUR/USD" --timeframe "60" --timezone "Europe/London"
+pyne data convert-from ./data/eurusd.csv --timezone "Europe/London"
 ```
 
+**Generated TOML Configuration:**
+
+After conversion, a TOML configuration file is automatically generated with:
+- **Smart Symbol Type Detection**: Automatically identifies forex, crypto, or other asset types
+- **Tick Size Analysis**: Analyzes price data to determine the minimum price increment
+- **Opening Hours Detection**: Detects trading hours from actual trading activity
+- **Interval Detection**: Automatically determines the timeframe from timestamp intervals
+
+The generated TOML file includes all detected information and can be manually adjusted if needed.
+
 ## Data File Structure
 
 PyneCore uses a structured approach to store OHLCV data:

docs/cli/run.md

@@ -27,7 +27,7 @@ pyne run SCRIPT DATA [OPTIONS]
 
 Where:
 - `SCRIPT`: Path to the PyneCore script (.py) or Pine Script (.pine) file
-- `DATA`: Path to the OHLCV data (.ohlcv) file
+- `DATA`: Path to the data file (.ohlcv, .csv, .json, or .txt)
 - `OPTIONS`: Additional options to customize the execution
 
 ## Simple Example
@@ -81,15 +81,61 @@ Example with API key:
 pyne run my_strategy.pine eurusd_data.ohlcv --api-key "your-api-key"
 ```
 
+## Automatic Data Conversion
+
+The `run` command now supports automatic conversion of non-OHLCV data formats. When you provide a CSV, JSON, or TXT file, the system automatically:
+
+1. **Detects the file format** from the extension
+2. **Analyzes the filename** to extract symbol and provider information
+3. **Converts the data** to OHLCV format
+4. **Generates a TOML configuration** with detected parameters
+5. **Runs the script** with the converted data
+
+### Supported Formats and Detection
+
+The automatic conversion supports:
+- **CSV files**: Standard comma-separated values
+- **JSON files**: JSON formatted OHLCV data
+- **TXT files**: Tab, semicolon, or pipe-delimited data (coming soon)
+
+### Filename Pattern Detection
+
+The system recognizes common filename patterns:
+- `BTCUSDT.csv` → Symbol: BTC/USDT
+- `EUR_USD.json` → Symbol: EUR/USD
+- `ccxt_BYBIT_BTC_USDT.csv` → Symbol: BTC/USDT, Provider: bybit
+- `BINANCE_ETHUSDT_1h.csv` → Symbol: ETH/USDT, Provider: binance
+
+### Example with Automatic Conversion
+
+```bash
+# Run a script with CSV data (automatic conversion)
+pyne run my_strategy.py BTCUSDT.csv
+
+# The system will:
+# 1. Detect BTC/USDT as the symbol
+# 2. Convert CSV to OHLCV format
+# 3. Generate BTCUSDT.toml with symbol info
+# 4. Run the script with converted data
+```
+
+### Advanced Analysis During Conversion
+
+When converting data, the system performs advanced analysis:
+- **Tick Size Detection**: Analyzes price movements to determine minimum price increment
+- **Trading Hours Detection**: Identifies when the market is actively trading
+- **Interval Auto-Correction**: Detects and fixes incorrect timeframe settings
+- **Symbol Type Detection**: Identifies forex, crypto, or other asset types
+
 ## Command Arguments
 
 The `run` command has two required arguments:
 
 - `SCRIPT`: The script file to run. If only a filename is provided, it will be searched in the `workdir/scripts/` directory.
-- `DATA`: The OHLCV data file to use. If only a filename is provided, it will be searched in the `workdir/data/` directory.
+- `DATA`: The data file to use. Supports .ohlcv, .csv, .json formats. If only a filename is provided, it will be searched in the `workdir/data/` directory.
 
 <small>
-Note: you don't need to write the `.py` and `.ohlcv` extensions in the command.
+Note: you don't need to write the file extensions in the command.
 </small>
 
 ## Command Options

pytest.ini

@@ -6,4 +6,4 @@ log_cli_level = DEBUG
 log_cli_format = %(asctime)s %(levelname)6s %(module_func_line)30s - %(message)s
 log_cli_date_format = %Y-%m-%d %H:%M:%S
 
-addopts = --import-mode=importlib -rs -x --spec --ignore-glob="**/data/**"
+addopts = --import-mode=importlib -rs -x --spec --ignore-glob="**/data/*modified.py"