docs: clean up bluffs, sync with code, and upgrade dependencies

- Corrected Readme.md, backend/doc.md, and backend/test.md to accurately reflect current codebase.
- Fixed collector log (7s vs 1s interval).
- Implemented missing 'mempool_health_statistics' in backend to fix frontend gap.
- Upgraded backend and frontend dependencies to latest production-ready versions.

Author: ismaelsadeeq

Readme.md

@@ -1,49 +1,47 @@
-# Bitcoin Core Fee Tracker & Visualizer
+### Bitcoin Core Fee Rate Estimator
 
-A comprehensive full-stack application for monitoring and visualizing Bitcoin Core transaction fees, mempool dynamics, and block statistics. This tool provides real-time insights through a modern web interface powered by a robust backend that interfaces directly with a Bitcoin Core node.
+- A full-stack application for monitoring and validating Bitcoin Core transaction fee estimates against actual block data.
+- Built on top of Bitcoin Core PR #34075
 
-## Overview
+### Overview
 
-This project provides a centralized platform to track and analyze Bitcoin transaction fees. It bridges the gap between raw Bitcoin Core RPC data and human-readable visualizations.
+This project tracks `estimatesmartfee` from a Bitcoin Core node and compares those estimates with the feerate percentiles of subsequent blocks. It provides a visual interface to verify the accuracy of the node's fee predictions.
 
-### Key Features
-- **Real-time Fee Estimation**: Get feerate estimates based on current mempool percentiles and historical data.
-- **Interactive Charts**: Visualize fee history, block statistics, and mempool status using Recharts and D3.
-- **Mempool Diagram**: View the current state of the mempool in a graphical format.
-- **Unified API**: A clean REST API that handles multiple data sources and internal caching.
+#### Key Features
+- **Fee Estimate Tracking**: A background service polls Bitcoin Core every 7 seconds for smart fee estimates.
+- **Historical Accuracy**: Visualizes the accuracy of estimates (within range, overpaid, or underpaid) compared to real block data.
+- **Mempool Diagram**: Real-time visualization of the mempool fee/weight accumulation curve.
+- **Block Statistics**: Direct insights into feerate percentiles for recent blocks.
 
-## Architecture & Integration
+#### Architecture
 
-The project is divided into two main modules that are linked through a secure proxy layer:
+- **Backend (Python/Flask)**: Communicates with Bitcoin Core via RPC. Collects estimates into SQLite and serves data via a REST API.
+- **Frontend (Next.js/TypeScript)**: Modern UI using Recharts and D3. Communicates with the backend via a secure API proxy route.
 
-- **Backend (Python/Flask)**: Communicates with your Bitcoin Core node via RPC. It handles data collection, persistence in SQLite, and serves as the source of truth for all analytics.
-- **Frontend (Next.js/TypeScript)**: Provides the user interface. It communicates with the backend via an internal API route (`frontend/src/app/api/[...path]/route.ts`) which proxies requests to the backend service. This architecture simplifies deployment and enhances security.
-
-## Project Structure
+#### Project Structure
 
 ```text
 .
-├── backend/            # Flask API, data collector, and database services
-│   ├── src/            # Application logic (services, app.py)
+├── backend/            # Flask API, data collector, and SQLite database
+│   ├── src/            # Core logic and RPC services
 │   └── tests/          # Pytest suite for backend validation
 ├── frontend/           # Next.js web application
-│   ├── src/app/        # App router, pages, and secure API proxy
-│   └── src/components/ # Reusable UI components and dynamic charts
-└── .github/workflows/  # Automated testing workflow (GitHub Actions)
+│   ├── src/app/        # App router and pages
+│   └── src/components/ # D3 and Recharts visualization components
+└── .github/workflows/  # Automated testing workflow
 ```
 
-## How to Use
+#### How to Use
 
-### Prerequisites
-- **Bitcoin Core Node**: Access to a running Bitcoin Core node with RPC enabled.
-- **Python**: Version 3.12+
-- **Node.js**: Version 22+
+#### Prerequisites
+- **Bitcoin Core Node**: Access to a node with RPC enabled (`getblockstats` support required).
+- **Python**: 3.12+
+- **Node.js**: 22+
 
-### 1. Configuration
-- **Backend**: Navigate to `backend/`, copy `rpc_config.ini.example` to `rpc_config.ini`, and fill in your node's details.
-- **Frontend**: Ensure the `BACKEND_URL` environment variable is set (defaults to `http://127.0.0.1:5001`).
+#### 1. Configuration
+- **Backend**: Copy `backend/rpc_config.ini.example` to `backend/rpc_config.ini` and provide RPC credentials.
 
-### 2. Manual Startup
+#### 2. Manual Startup
 **Backend:**
 ```bash
 cd backend
@@ -60,20 +58,15 @@ npm install
 npm run dev
 ```
 
-### 3. Automated Startup (Production-like)
-A `restart.sh` script is provided in the root directory to stop any existing instances and start both services in the background using Gunicorn (backend) and Next.js (frontend):
+#### 3. Automated Startup
+Use the provided `restart.sh` script to launch both services in the background:
 ```bash
 chmod +x restart.sh
 ./restart.sh
 ```
 
-## Credits
-
-This project is a collaborative effort between:
-- **Ismael Sadeeq**: Main contributor and maintainer.
-- **Gemini**: AI-assisted development, architectural design, and test automation.
-- **Claude**: AI-assisted development, code optimization, and documentation.
-- **b-l-u-e** ([winnie.gitau282@gmail.com](mailto:winnie.gitau282@gmail.com)): Contributions to core services, backend logic, and test suites.
-- **mercie-ux** ([mbaomercy0@gmail.com](mailto:mbaomercy0@gmail.com)): Contributions to user experience, frontend design, and visual components.
-
-The codebase represents a merge of PR work from contributors and AI-generated improvements for a complete, robust experience.
+### Credits
+- **Abubakar Sadiq Ismail**: Bitcoin Core contributor and architecture.
+- **b-l-u-e**: Backend logic and service implementation.
+- **mercie-ux**: Frontend design and visual components.
+- **Gemini & Claude**: AI-assisted development and test automation.

backend/doc.md

@@ -1,6 +1,6 @@
 # Backend - Bitcoin Core Fees API
 
-This service provides a Flask-based REST API to interact with Bitcoin Core RPC and provide fee analytics.
+This Flask-based REST API interacts with Bitcoin Core RPC and a local SQLite database to provide fee analytics and block statistics.
 
 ## Running the Application
 
@@ -15,55 +15,36 @@ pip install -r requirements.txt
 Ensure `rpc_config.ini` is configured with your Bitcoin Core RPC credentials.
 
 ### 2. Start the App (Background)
-To start the application and keep it running after you disconnect from the terminal, use the following `nohup` command:
+To start the application in the background:
 
 ```bash
-nohup .venv/bin/python app.py > debug.log 2>&1 &
+nohup env PYTHONPATH=src .venv/bin/gunicorn --workers 4 --bind 127.0.0.1:5001 app:app > debug.log 2>&1 &
 ```
 
-**What this command does:**
-*   `nohup`: Stands for "No Hang Up". It allows the command to continue running even after you logout or close the terminal.
-*   `.venv/bin/python app.py`: Executes the Flask app using the Python interpreter inside your virtual environment.
-*   `> debug.log`: Redirects standard output (logs) to a file named `debug.log`.
-*   `2>&1`: Redirects standard error (errors) to the same location as standard output (`debug.log`).
-*   `&`: Puts the command in the background, allowing you to continue using the terminal.
-
 ### 3. Monitoring Logs
 To see the logs in real-time:
 ```bash
 tail -f debug.log
 ```
 
 ### 4. Stopping the App
-To stop the background process, you can find the Process ID (PID) and kill it, or use `pkill`:
-
-**Option A (Using pkill):**
-```bash
-pkill -f "python app.py"
-```
-
-**Option B (By Port):**
+To stop the process:
 ```bash
-kill $(lsof -t -i:5001)
+pkill -f "gunicorn"
 ```
 
-**Option C (Manual):**
-1. Find the PID: `ps aux | grep "python app.py"`
-2. Kill the process: `kill <PID>`
-
 ## API Endpoints
 
 | Endpoint | Method | Description |
 |----------|--------|-------------|
-| `/health` | GET | Check service and RPC connection status. |
-| `/blockchain/info` | GET | Get general info about the Bitcoin blockchain. |
-| `/blockcount` | GET | Get the current block height. |
-| `/mempool/info` | GET | Get current mempool state (size, bytes, etc.). |
-| `/fees/<target>/<mode>/<level>` | GET | Get `estimatesmartfee` from Bitcoin Core. |
-| `/fees/mempool` | GET | Get fee estimates based on current mempool percentiles. |
-| `/api/v1/fees/estimate` | GET | Unified endpoint for mempool, historical, or hybrid estimates. |
-| `/analytics/summary` | GET | Get summarized fee and block analytics (internal or external fallback). |
-| `/blockstats/<height>` | GET | Get detailed stats for a specific block height. |
-| `/external/block-stats/<count>` | GET | Proxy to external API for block statistics. |
-| `/external/fees-stats/<count>` | GET | Proxy to external API for fee statistics. |
-| `/external/fees-sum/<count>` | GET | Proxy to external API for fee summation analytics. |
+| `/blockcount` | GET | Current block height from node. |
+| `/fees/<target>/<mode>/<level>` | GET | `estimatesmartfee` results converted to sat/vB. |
+| `/mempool-diagram` | GET | Analyzed feerate diagram for mempool accumulation. |
+| `/performance-data/<start_block>/` | GET | Block feerate percentiles vs. recorded estimates. |
+| `/fees-sum/<start_block>/` | GET | Aggregated accuracy metrics (within, over, under). |
+
+### Parameters:
+- `target`: Confirmation target (e.g., 2, 7, 144).
+- `mode`: Fee estimation mode (`economical`, `conservative`, `unset`).
+- `level`: Verbosity level for fee estimation.
+- `start_block`: Block height to start range analysis from.

backend/requirements.txt

@@ -1,6 +1,8 @@
 Flask==3.1.3
-Flask-CORS==4.0.2
+Flask-CORS==6.0.2
 Flask-Limiter==4.1.1
 requests==2.32.5
 configparser==6.0.0
-gunicorn
+gunicorn==25.1.0
+pytest==9.0.2
+pytest-cov==7.0.0

backend/src/services/collector_service.py

@@ -8,7 +8,7 @@
 _collector_started = False
 
 def run_collector():
-    logger.info("Starting high-resolution fee estimate collector (1s interval)...") 
+    logger.info("Starting high-resolution fee estimate collector (7s interval)...") 
     # 1 and 2 are the same, so we only poll 2
     targets = [2, 7, 144]
 

backend/src/services/rpc_service.py

@@ -104,7 +104,7 @@ def _get_single_block_stats_cached(height: int) -> tuple:
     Returns a frozen (JSON-serialised) snapshot so the lru_cache holds
     immutable data. Use get_single_block_stats() for normal access.
     """
-    result = _rpc_call("getblockstats", [height, ["height", "feerate_percentiles", "minfeerate", "maxfeerate"]])
+    result = _rpc_call("getblockstats", [height, ["height", "feerate_percentiles", "minfeerate", "maxfeerate", "total_weight"]])
     return json.dumps(result)  # freeze as string
 
 
@@ -121,12 +121,47 @@ def get_block_count() -> int:
     return _rpc_call("getblockcount", [])
 
 
+def get_mempool_health_statistics() -> List[Dict[str, Any]]:
+    """
+    Fetches stats for the last 5 blocks to compare their weights with 
+    the current mempool's readiness.
+    """
+    current_height = get_block_count()
+    stats = []
+    
+    # Using getmempoolfeeratediagram for accurate total weight
+    mempool_diagram = _rpc_call("getmempoolfeeratediagram", [])
+    total_mempool_weight = mempool_diagram[-1]["weight"] if mempool_diagram else 0
+
+    for h in range(current_height - 4, current_height + 1):
+        try:
+            b = get_single_block_stats(h)
+            weight = b.get("total_weight", 0)
+            
+            stats.append({
+                "block_height": h,
+                "block_weight": weight,
+                "mempool_txs_weight": total_mempool_weight,
+                "ratio": min(1.0, total_mempool_weight / 4_000_000)
+            })
+        except Exception:
+            continue
+    return stats
+
+
 def estimate_smart_fee(conf_target: int, mode: str = "unset", verbosity_level: int = 2) -> Dict[str, Any]:
     effective_target = _clamp_target(conf_target)
     result = _rpc_call("estimatesmartfee", [effective_target, mode, verbosity_level])
     if result and "feerate" in result:
         # feerate is BTC/kVB → sat/vB: × 1e8 (BTC→sat) ÷ 1e3 (kVB→vB) = × 1e5
         result["feerate_sat_per_vb"] = result["feerate"] * 100_000
+    
+    # Include health stats for the frontend
+    try:
+        result["mempool_health_statistics"] = get_mempool_health_statistics()
+    except Exception as e:
+        logger.error(f"Failed to include health stats: {e}")
+        
     return result
 
 

backend/test.md

@@ -2,30 +2,23 @@
 
 ## Prerequisites
 
-`requirements.txt` should include:
+Ensure all dependencies including test tools are installed:
 
-```
-Flask==3.1.3
-Flask-CORS==4.0.2
-Flask-Limiter==4.1.1
-requests==2.32.3
-configparser==6.0.0
-pytest
-pytest-cov
+```bash
+pip install pytest pytest-cov
 ```
 
 ---
 
 ## Test Structure
 
-```
+```text
 tests/
-├── conftest.py                  # pytest path setup
-├── helpers.py                   # shared app factory
-├── test_app.py                  # HTTP layer — routes, error handlers, mode validation
-├── test_rpc_service.py          # RPC logic, fee math, caching, mempool diagram
-├── test_database_service.py     # SQLite writes, queries, indexes, edge cases
-└── test_collector_service.py    # Collector lifecycle, duplicate guard, error resilience
+├── conftest.py                  # Pytest path and app setup
+├── helpers.py                   # Shared app factory for tests
+├── test_app.py                  # HTTP layer (routes, validation)
+├── test_rpc_service.py          # RPC conversion and calculation logic
+└── test_database_service.py     # SQLite writes and query filtering
 ```
 
 ---
@@ -34,11 +27,6 @@ tests/
 
 All commands should be run from the `backend/` directory.
 
-**Install all required packages**
-```
-pip install -r requirements.txt
-```
-
 **Run the full suite:**
 ```bash
 python -m pytest tests/ -v
@@ -49,12 +37,11 @@ python -m pytest tests/ -v
 python -m pytest tests/test_app.py -v
 python -m pytest tests/test_rpc_service.py -v
 python -m pytest tests/test_database_service.py -v
-python -m pytest tests/test_collector_service.py -v
 ```
 
 **Run a single test by name:**
 ```bash
-python -m pytest tests/test_rpc_service.py::TestRpcService::test_feerate_conversion_is_correct -v
+python -m pytest tests/test_rpc_service.py::test_feerate_conversion_is_correct -v
 ```
 
 **Stop on first failure:**
@@ -68,13 +55,10 @@ python -m pytest tests/ -v -x
 
 **Print coverage summary in terminal:**
 ```bash
-python -m pytest tests/ -v --cov=src/services --cov=src/app --cov-report=term-missing
+python -m pytest tests/ -v --cov=src --cov-report=term-missing
 ```
 
-**Generate an HTML report (opens in browser):**
+**Generate an HTML report:**
 ```bash
-python -m pytest tests/ --cov=src/services --cov=src/app --cov-report=html
-open htmlcov/index.html
+python -m pytest tests/ --cov=src --cov-report=html
 ```
-
-

frontend/package.json

@@ -11,11 +11,11 @@
   },
   "dependencies": {
     "d3": "^7.9.0",
-    "lucide-react": "^0.544.0",
-    "next": "15.5.10",
-    "react": "19.1.5",
-    "react-dom": "19.1.5",
-    "recharts": "^3.2.1"
+    "lucide-react": "^0.575.0",
+    "next": "16.1.6",
+    "react": "19.2.4",
+    "react-dom": "19.2.4",
+    "recharts": "^3.7.0"
   },
   "devDependencies": {
     "@eslint/eslintrc": "^3",