feat(pkg-py): Replace pandas with narwhals by cpsievert · Pull Request #175 · posit-dev/querychat

cpsievert · 2025-12-19T23:04:30Z

This PR removes pandas as a required dependency and replaces it with narwhals, a lightweight DataFrame abstraction layer that supports both pandas and polars backends. Users can now choose their preferred DataFrame library.

Motivation

Reduced dependencies: pandas is a heavy dependency; users who prefer polars shouldn't need to install pandas
Flexibility: Some users prefer polars for its performance characteristics
Future-proofing: narwhals provides a stable API that works across DataFrame libraries

Changes

Dependencies

Removed pandas from required dependencies
Added pandas and polars as optional dependencies
Install with pip install querychat[pandas] or pip install querychat[polars]

API Changes (Breaking)

execute_query(), get_data(), and df() now return narwhals DataFrames instead of pandas DataFrames
Call .to_native() on returned DataFrames to get the underlying pandas/polars DataFrame

Internal Changes

New _df_compat.py module handles backend selection (prefers polars when available)
df_to_html() generates HTML directly without pandas dependency
DataFrameSource accepts pandas, polars, or narwhals DataFrames

Tests

Added test_df_compat.py for the compatibility layer
Added test_dataframe_source.py with comprehensive DataFrameSource tests
Polars-specific tests are skipped when polars isn't installed

Migration Guide

# Before
df = querychat.df()  # pandas DataFrame
df.head()

# After
df = querychat.df()  # narwhals DataFrame
df.head()            # still works - narwhals has similar API

# If you need a native pandas/polars DataFrame:
native_df = querychat.df().to_native()

cpsievert · 2025-12-19T23:17:05Z

pkg-py/src/querychat/_datasource.py

-    """A DataSource implementation that wraps a pandas DataFrame using DuckDB."""
+    """A DataSource implementation that wraps a DataFrame using DuckDB."""

-    _df: nw.DataFrame | nw.LazyFrame


I'm pretty sure it's going to make sense for us to have a separate LazyFrameSource, which I'll do in a follow up PR (before the next release). The benefit being that we can be more lazy about computation, and possibly have .df() also return a LazyFrame in that scenario

makes sense!

cpsievert · 2025-12-19T23:18:10Z

pkg-py/src/querychat/_datasource.py


-        # Ensure we're working with a DataFrame, not a LazyFrame
-        ndf = (
-            self._df.head(10).collect()


Note that downstream calculation of ranges and unique values wasn't working properly because they were based on the first 10 rows -- I'll address this when doing the new LazyFrameSource implementation

Remove pandas as a required dependency in favor of narwhals, which provides a unified DataFrame interface supporting both pandas and polars backends. Changes: - Add _df_compat.py module with read_csv, read_sql, and duckdb_result_to_nw helpers - Update DataSource classes to return narwhals DataFrames - Update df_to_html to generate HTML without pandas dependency - Make pandas and polars optional dependencies - Add comprehensive tests for DataFrameSource and df_compat module Users can now install with either `pip install querychat[pandas]` or `pip install querychat[polars]`. Use `.to_native()` on returned DataFrames to get the underlying pandas or polars DataFrame. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Resolved conflicts: - _datasource.py: Combined narwhals abstraction layer with security configs, check_query validation, and test_query method - build.qmd: Kept include directive approach from main 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

cpsievert · 2026-01-07T16:37:36Z

pkg-py/src/querychat/_utils.py

+    columns = df_short.columns
+    rows = df_short.rows()
+
+    # Build HTML table
+    html_parts = ['<table border="1" class="dataframe table table-striped">']
+
+    # Header
+    html_parts.append("  <thead>")
+    html_parts.append('    <tr style="text-align: right;">')
+    html_parts.extend(f"      <th>{_escape_html(col)}</th>" for col in columns)
+    html_parts.append("    </tr>")
+    html_parts.append("  </thead>")
+
+    # Body
+    html_parts.append("  <tbody>")
+    for row in rows:
+        html_parts.append("    <tr>")
+        html_parts.extend(f"      <td>{_escape_html(str(val))}</td>" for val in row)
+        html_parts.append("    </tr>")
+    html_parts.append("  </tbody>")
+
+    html_parts.append("</table>")
+    table_html = "\n".join(html_parts)


I feel a bit torn on whether it's worth a great_tables dependency here? It is pretty low dependency, so maybe it's worth it?

I think it makes sense to use great_tables if it drops in nicely

gadenbuie

Nice!

gadenbuie · 2026-01-07T19:42:23Z

pkg-py/src/querychat/_datasource.py

-    """A DataSource implementation that wraps a pandas DataFrame using DuckDB."""
+    """A DataSource implementation that wraps a DataFrame using DuckDB."""

-    _df: nw.DataFrame | nw.LazyFrame


makes sense!

gadenbuie · 2026-01-07T19:44:03Z

pkg-py/src/querychat/_utils.py

+    columns = df_short.columns
+    rows = df_short.rows()
+
+    # Build HTML table
+    html_parts = ['<table border="1" class="dataframe table table-striped">']
+
+    # Header
+    html_parts.append("  <thead>")
+    html_parts.append('    <tr style="text-align: right;">')
+    html_parts.extend(f"      <th>{_escape_html(col)}</th>" for col in columns)
+    html_parts.append("    </tr>")
+    html_parts.append("  </thead>")
+
+    # Body
+    html_parts.append("  <tbody>")
+    for row in rows:
+        html_parts.append("    <tr>")
+        html_parts.extend(f"      <td>{_escape_html(str(val))}</td>" for val in row)
+        html_parts.append("    </tr>")
+    html_parts.append("  </tbody>")
+
+    html_parts.append("</table>")
+    table_html = "\n".join(html_parts)


I think it makes sense to use great_tables if it drops in nicely

Replace manual HTML table construction in df_to_html() with great_tables GT class for richer, styled table output in chat messages. - Add great-tables>=0.16.0 as a dependency - Simplify df_to_html() to use GT().as_raw_html() - Remove manual _escape_html helper (great_tables handles escaping) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Passing self.client() (calling the method) resulted in tools being registered twice - once in client() and again in mod_server. Instead, pass self.client (the method reference) so mod_server can call it with the update_dashboard and reset_dashboard callbacks. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

cpsievert requested a review from Copilot December 19, 2025 23:04

This comment was marked as resolved.

Sign in to view

cpsievert force-pushed the feat/narwhals-df branch 2 times, most recently from 09816ed to 1081e4d Compare December 19, 2025 23:11

cpsievert commented Dec 19, 2025

View reviewed changes

cpsievert force-pushed the feat/narwhals-df branch 2 times, most recently from 65e778b to 167317d Compare December 19, 2025 23:38

cpsievert force-pushed the feat/narwhals-df branch from 167317d to 0ffa6e7 Compare December 19, 2025 23:43

cpsievert requested a review from Copilot December 19, 2025 23:48

cpsievert marked this pull request as ready for review December 19, 2025 23:48

cpsievert requested a review from gadenbuie December 19, 2025 23:48

This comment was marked as resolved.

Sign in to view

Update CLAUDE.md

288cd55

cpsievert force-pushed the feat/narwhals-df branch from 98ef15d to 288cd55 Compare December 19, 2025 23:49

cpsievert changed the title ~~feat(pkg-py): Replace pandas dependency with narwhals abstraction layer~~ feat(pkg-py): Replace pandas with narwhals Dec 19, 2025

cpsievert commented Jan 7, 2026

View reviewed changes

gadenbuie approved these changes Jan 7, 2026

View reviewed changes

cpsievert and others added 3 commits January 7, 2026 15:56

Merge branch 'main' into feat/narwhals-df

6f187df

cpsievert merged commit 53c9872 into main Jan 7, 2026
6 checks passed

cpsievert deleted the feat/narwhals-df branch January 7, 2026 22:50

cpsievert mentioned this pull request Jan 7, 2026

py: Create a proper LazyFrame DataSource #189

Closed

cpsievert mentioned this pull request Jan 16, 2026

(py) Drop pandas dependency #105

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pkg-py): Replace pandas with narwhals#175

feat(pkg-py): Replace pandas with narwhals#175
cpsievert merged 6 commits intomainfrom
feat/narwhals-df

cpsievert commented Dec 19, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

cpsievert Dec 19, 2025 •

edited

Loading

Uh oh!

gadenbuie Jan 7, 2026

Uh oh!

cpsievert Dec 19, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

cpsievert Jan 7, 2026

Uh oh!

gadenbuie Jan 7, 2026

Uh oh!

gadenbuie left a comment

Uh oh!

gadenbuie Jan 7, 2026

Uh oh!

gadenbuie Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cpsievert commented Dec 19, 2025

Motivation

Changes

Dependencies

API Changes (Breaking)

Internal Changes

Tests

Migration Guide

Uh oh!

This comment was marked as resolved.

Uh oh!

cpsievert Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gadenbuie Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

cpsievert Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

cpsievert Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

gadenbuie Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

gadenbuie left a comment

Choose a reason for hiding this comment

Uh oh!

gadenbuie Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

gadenbuie Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cpsievert Dec 19, 2025 •

edited

Loading