Merge pull request #10 from Ganymede-Bio/fix-release-ci

Fix release ci

Author: bensonlee5

.github/workflows/release.yml

@@ -142,6 +142,7 @@ jobs:
     - name: Create GitHub Release
       uses: softprops/action-gh-release@v1
       with:
+        tag_name: v${{ needs.build.outputs.version }}
         name: v${{ needs.build.outputs.version }}
         body_path: release_notes.md
         files: |

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.5.0
+    rev: v5.0.0
     hooks:
       - id: trailing-whitespace
       - id: end-of-file-fixer
@@ -18,14 +18,14 @@ repos:
         args: ['--pytest-test-first']
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.1.15
+    rev: v0.12.7
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
 
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.5.1
+    rev: v1.17.0
     hooks:
       - id: mypy
         args: [--config-file=pyproject.toml]

CHANGELOG.md

@@ -5,13 +5,28 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.3.4] - 2025-07-29
+
+### Added
+- BoxTableDetector for high-confidence detection of tables with complete borders (95% confidence)
+- Headers are now always extracted from the first row of detected tables
+- Header extraction support for all detection methods (SimpleCaseDetector, IslandDetector, BoxTableDetector)
+
+### Changed
+- Improved header detection to focus on bold text and data type differences
+- Reduced emphasis on background color for header detection (based on user feedback)
+- Updated detection pipeline to include box table detection as a fast path
+
+### Fixed
+- Fixed header extraction in island detection - headers were not being returned
+- Fixed sheet_data parameter bug in SimpleCaseDetector.convert_to_table_info
+- Fixed has_headers property to correctly transfer from islands to TableInfo objects
+
 ## [0.3.1] - 2025-07-29
 
 ### Changed
-- **Project Rename**: Renamed from GridPorter to GridGulp
-  - Updated all package references throughout codebase
-  - Renamed source directory from `src/gridporter/` to `src/gridgulp/`
-  - Updated project metadata and documentation
 - CI improvements:
   - Added Python version matrix testing (3.10, 3.11, 3.12, 3.13)
   - Updated ruff target version to py310 (minimum supported)

CLAUDE.md

@@ -10,9 +10,10 @@ The system follows a hierarchical detection strategy:
 
 1. **File Type Detection**: Use file magic and content analysis to determine actual file type
 2. **Single Table Check**: Fast check if file/sheet contains only one table (handles ~80% of cases)
-3. **Excel Metadata**: For Excel files, check native table objects and named ranges
-4. **Island Detection**: Algorithm to find disconnected data regions for multi-table sheets
-5. **Heuristic Analysis**: Apply header/format analysis for improved accuracy
+3. **Box Table Detection**: For Excel files, detect tables with complete borders (95% confidence)
+4. **Excel Metadata**: For Excel files, check native table objects and named ranges
+5. **Island Detection**: Algorithm to find disconnected data regions for multi-table sheets
+6. **Heuristic Analysis**: Apply header/format analysis for improved accuracy
 
 ### Detection Components
 
@@ -21,12 +22,21 @@ The system follows a hierarchical detection strategy:
 - Fast path optimization for common cases
 - Uses gap detection and data density analysis
 - Returns high confidence scores for clear single-table layouts
+- Always extracts headers from the first row
+
+#### BoxTableDetector
+- Detects tables with complete borders on all four sides
+- Assigns 95% confidence to these tables (addresses user feedback)
+- Verifies data density to avoid empty bordered regions
+- Ideal for formatted Excel tables with clear boundaries
+- Extracts headers with formatting-based detection
 
 #### IslandDetector
 - Identifies multiple disconnected data regions
 - Creates binary mask of non-empty cells
 - Uses connected component analysis
 - Handles complex multi-table layouts
+- Always extracts headers from first row of each island
 
 #### ExcelMetadataExtractor
 - Extracts Excel ListObjects (native tables)
@@ -47,25 +57,26 @@ class TableInfo(BaseModel):
     range: CellRange = Field(..., description="Table boundaries")
     confidence: float = Field(..., ge=0.0, le=1.0)
     detection_method: str
-    headers: list[str] | None = None
+    headers: list[str] | None = None  # Always extracted from first row
+    has_headers: bool = True  # Header detection confidence
     shape: tuple[int, int] = Field(..., description="(rows, columns)")
 ```
 
 ### File Handling Strategy
 
 #### Excel Files
-- Use openpyxl for .xlsx/.xlsm/.xlsb files
+- Use openpyxl for .xlsx/.xlsm files
 - Use xlrd for legacy .xls files
 - Preserve formatting metadata for detection
 - Handle multiple sheets independently
-- Support for python-calamine for fast parsing
 
 #### CSV/Text Files
 - Auto-detect delimiter using csv.Sniffer
 - Sophisticated encoding detection (BOM, chardet, pattern-based)
 - Handle various delimiters (comma, tab, pipe, semicolon)
 - Support UTF-8, UTF-16 (LE/BE), Latin-1, and more
-- Detect header rows using heuristics
+- Detect header rows using heuristics (bold text, data type differences)
+- Background color is no longer a primary header indicator
 
 #### File Type Detection
 - Check file signatures before trusting extensions

NOTICE

@@ -27,8 +27,6 @@ rich (https://github.com/Textualize/rich)
 Copyright (c) 2020 Will McGugan
 Licensed under the MIT License.
 
-python-calamine (https://github.com/dimastbk/python-calamine)
-Licensed under the MIT License.
 
 ## BSD Licensed Dependencies
 

README.md

@@ -1,12 +1,22 @@
 # GridGulp
 
+[![PyPI version](https://badge.fury.io/py/gridgulp.svg)](https://pypi.org/project/gridgulp/)
+[![Python Versions](https://img.shields.io/pypi/pyversions/gridgulp.svg)](https://pypi.org/project/gridgulp/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://ganymede-bio.github.io/gridgulp/)
+
 Automatically detect and extract tables from Excel, CSV, and text files.
 
 ## What is GridGulp?
 
-GridGulp finds tables in your spreadsheets - even when there are multiple tables on one sheet or when tables don't start at cell A1. It comes with reasonable defaults and is fully configurable.
+GridGulp finds tables in your spreadsheets, even when
+
+- there are multiple tables on one sheet
+- tables don't start at cell A1
+- file extensions do not reflect its file type
+- the file encoding is opaque
 
-**Supported formats:** `.xlsx`, `.xls`, `.xlsm`, `.xlsb`, `.csv`, `.tsv`, `.txt`
+**Supported formats:** `.xlsx`, `.xls`, `.xlsm`, `.csv`, `.tsv`, `.txt`
 
 ## Installation
 
@@ -16,22 +26,38 @@ pip install gridgulp
 
 ## Quick Start
 
+### Trying GridGulp Out
+
+To quickly try GridGulp on some spreadsheets, clone the repo, place example spreadsheets in the examples/ directory, and run
+
+```bash
+python scripts/test_example_files.py
+```
+
+You will receive output that looks like, representing identified ranges:
+
+📁 tests/manual
+----------------------------------------------------------------------------------------------------
+✓ sample.xlsx                              | Tables: 1  | Time: 1.099s | Size: 122.6KB | Method: magika
+  📄 Sheet: Sheet
+     └─ A1:CV203        | 203×100 | Conf: 70%
+
+
 ### Table Ranges vs DataFrames
 
 GridGulp provides two ways to work with detected tables:
 
-1. **Table Ranges** - Lightweight metadata about where tables are located (e.g., "A1:E100")
+1. **Table Ranges** - JSON metadata about where tables are located (e.g., "A1:E100")
    - Fast and memory-efficient
-   - Perfect for mapping table locations or visualizing spreadsheet structure
+   - Perfect for agent use as tools - mapping table locations or visualizing spreadsheet structure
    - No actual data is loaded into memory
 
 2. **DataFrames** - The actual data extracted from those ranges as pandas DataFrames
    - Contains the full data with proper types
    - Ready for analysis, transformation, or export
-   - Requires more memory but provides full data access
 
 Choose based on your needs:
-- Use **ranges only** when you need to know where tables are or want to process them later
+- Use **ranges only** when you need to know where tables are and want to submit to other tasks - for example, a downstream process to infer purpose / intent based on data content
 - Use **DataFrames** when you need to analyze or transform the actual data
 
 ### Getting Table Ranges Only
@@ -185,14 +211,14 @@ if all_dataframes:
 - **Smart Headers** - Detects single and multi-row headers automatically
 - **Multiple Tables** - Handles sheets with multiple separate tables
 - **Quality Scoring** - Confidence scores for each detected table
-- **Fast** - Processes most files in under a second
+- **Fast** - Processes 1M+ cells/second for simple tables, 100K+ cells/second for complex tables
 
 ## Documentation
 
 - [Full Usage Guide](docs/USAGE_GUIDE.md) - Detailed examples and configuration
 - [API Reference](docs/API_REFERENCE.md) - Complete API documentation
 - [Architecture](docs/ARCHITECTURE.md) - How GridGulp works internally
-- [Testing Guide](docs/TESTING_WITH_SCRIPT.md) - Test spreadsheets in bulk with the unified test script
+- [Testing Guide](docs/TESTING_GUIDE.md) - Test spreadsheets in bulk with the unified test script
 
 ## License
 

docs/API_REFERENCE.md

@@ -54,7 +54,6 @@ class Config(BaseModel):
     max_sheets: int = 10               # Max sheets to process
 
     # Performance
-    excel_reader: str = "calamine"     # "calamine" or "openpyxl"
     max_memory_mb: int = 1000          # Max memory usage
     chunk_size: int = 10000            # Streaming chunk size
 
@@ -180,7 +179,8 @@ class TableInfo(BaseModel):
     suggested_name: str | None = None  # Optional name
     confidence: float                  # Detection confidence (0-1)
     detection_method: str              # Method used
-    headers: list[str] | None = None   # Column headers
+    headers: list[str] | None = None   # Column headers (always extracted from first row)
+    has_headers: bool = True           # Whether headers were detected (confidence)
     data_preview: list[dict] | None = None  # Sample data
 
     @property
@@ -244,7 +244,7 @@ class FileType(str, Enum):
     XLSX = "xlsx"      # Modern Excel
     XLS = "xls"        # Legacy Excel
     XLSM = "xlsm"      # Excel with macros
-    XLSB = "xlsb"      # Excel binary
+    XLSB = "xlsb"      # Excel binary (detected but not supported)
     CSV = "csv"        # Comma-separated
     TSV = "tsv"        # Tab-separated
     TXT = "txt"        # Text file
@@ -262,6 +262,8 @@ class SimpleCaseDetector:
     def detect_simple_table(self, sheet_data: SheetData) -> SimpleCaseResult:
         """Detect a single table starting near A1.
 
+        Headers are always extracted from the first row of detected tables.
+
         Args:
             sheet_data: Sheet data to analyze
 
@@ -273,6 +275,37 @@ class SimpleCaseDetector:
         """Check if sheet is a simple single-table case."""
 ```
 
+### BoxTableDetector
+
+High-confidence detection for tables with complete borders.
+
+```python
+class BoxTableDetector:
+    def __init__(self,
+                 min_table_size: tuple[int, int] = (2, 2),
+                 box_confidence: float = 0.95):
+        """Initialize box table detector.
+
+        Args:
+            min_table_size: Minimum (rows, cols) for valid table
+            box_confidence: Confidence score for bordered tables (default: 0.95)
+        """
+
+    def detect_box_tables(self, sheet_data: SheetData) -> list[TableInfo]:
+        """Detect tables with complete borders on all four sides.
+
+        This detector assigns high confidence (95%) to tables that have
+        borders on all sides, addressing cases where formatting clearly
+        indicates table boundaries.
+
+        Args:
+            sheet_data: Sheet data to analyze
+
+        Returns:
+            List of TableInfo objects with high confidence
+        """
+```
+
 ### IslandDetector
 
 Multi-table detection using connected components.
@@ -294,6 +327,9 @@ class IslandDetector:
     def detect_islands(self, sheet_data: SheetData) -> list[DataIsland]:
         """Detect disconnected data regions.
 
+        Headers are always extracted from the first row of each detected
+        island, regardless of header detection confidence.
+
         Args:
             sheet_data: Sheet data to analyze
 
@@ -584,6 +620,7 @@ MIN_TABLE_SIZE = (2, 2)
 
 # Detection methods
 DETECTION_SIMPLE_CASE = "simple_case_fast"
+DETECTION_BOX_TABLE = "box_table_detection"
 DETECTION_ISLAND = "island_detection_fast"
 DETECTION_METADATA = "excel_metadata"
 

docs/ARCHITECTURE.md

@@ -30,8 +30,9 @@ GridGulp is a streamlined table detection framework that uses proven algorithms
 │                 Detection Pipeline                      │
 │  ┌─────────────────────────────────────────────────┐    │
 │  │ 1. SimpleCaseDetector (single table near A1)    │    │
-│  │ 2. IslandDetector (multi-table detection)       │    │
-│  │ 3. ExcelMetadataExtractor (ListObjects)         │    │
+│  │ 2. BoxTableDetector (tables with complete borders)│    │
+│  │ 3. IslandDetector (multi-table detection)       │    │
+│  │ 4. ExcelMetadataExtractor (ListObjects)         │    │
 │  └─────────────────────────────────────────────────┘    │
 ├─────────────────────────────────────────────────────────┤
 │                   Output Models                         │
@@ -88,12 +89,21 @@ GridGulp is a streamlined table detection framework that uses proven algorithms
 - **Performance**: < 1ms for most sheets
 - **Accuracy**: 100% for standard tables
 - **Algorithm**: Find data bounds, check density
+- **Headers**: Always extracted from first row
+
+#### BoxTableDetector
+- **Use Case**: Tables with complete borders on all four sides
+- **Performance**: < 10ms for most sheets
+- **Accuracy**: 95% confidence for bordered tables
+- **Algorithm**: Detect cells with borders on all sides, verify data density
+- **Headers**: Extracted with formatting-based detection
 
 #### IslandDetector
 - **Use Case**: Multiple disconnected tables
 - **Performance**: < 100ms for complex sheets
 - **Accuracy**: 95%+ for well-formatted data
 - **Algorithm**: Connected component analysis
+- **Headers**: Always extracted, with header detection confidence
 
 #### ExcelMetadataExtractor
 - **Use Case**: Excel tables with defined ListObjects
@@ -142,8 +152,16 @@ for sheet in file_data.sheets:
     # Try simple case first (fast path)
     if simple_detector.is_simple_case(sheet):
         tables = [simple_detector.detect_simple_table(sheet)]
+    # Try box detection for Excel files with formatting
+    elif file_type in [FileType.XLSX, FileType.XLS]:
+        box_tables = box_detector.detect_box_tables(sheet)
+        if box_tables:
+            tables = box_tables
+        else:
+            # Fall back to island detection
+            tables = island_detector.detect_tables(sheet)
     else:
-        # Fall back to island detection
+        # Use island detection for other formats
         tables = island_detector.detect_tables(sheet)
 ```