feat: Use `_repr_html_` when native supports it #2776

dangotbanned · 2025-07-03T13:07:24Z

What type of PR is this? (check all applicable)

Related issues

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

I discovered this method in (#2572), when I was trying to work out why polars.Expr looked so much better that what I had 😅

Thinking we can get more immediate benefits now by allowing this option when a backend supports it for:

DataFrame
- (pandas, polars)

LazyFrame
- (pandas, polars)

Series
- (~~pandas~~, polars)

- Related #1702 - https://ipython.readthedocs.io/en/stable/config/integrating.html#rich-display - https://github.com/pandas-dev/pandas/blob/22f12fc5d3f7fda3f198760204e7c13150c78581/pandas/core/frame.py#L1189-L1232 - https://github.com/pola-rs/polars/blob/8011fa34e0c5f1270ef52e2d3b0b2946bb2faa72/py-polars/polars/dataframe/frame.py#L1580-L1605

dangotbanned · 2025-07-03T19:16:07Z

narwhals/_utils.py

+    style_css = (
+        ".dataframe caption { "
+        "caption-side: bottom; "
+        "text-align: center; "
+        "font-weight: bold; "
+        "padding-top: 8px;"
+        "}"
+    )


If anyone has any suggestions for styling - feel free to experiment/comment 🙂

The only decision I'd made so far was putting the <caption> below the table

With the default polars formatting, it appeared between the table and the shape tuple when above - which I thought looked odd

That's very reasonable!

Realized I never followed this up with an example

Now that I'm looking at it again, maybe above isn't so bad?

- `pandas` reuses the eager version - `pyarrow` doesn't support - `ibis` requires changing global config, so skipping that - `dask` does have a `_repr_html_`, but doesn't parse well

dangotbanned · 2025-07-09T16:21:14Z

narwhals/_utils.py

+    if header == "Narwhals LazyFrame" and "LazyFrame" in native_html:
+        html = native_html.replace("LazyFrame", "LazyFrame.to_native()")
+        return f"{html}<p><b>{header}</b></p>"


Had to add this branch for pl.LazyFrame as it wasn't parsing with my naive wrapper:

import io import xml.etree.ElementTree as ET import polars as pl data = {"a": [1, 2, 3], "b": ["fdaf", "fda", "cf"]} ldf = pl.LazyFrame(data) >>> ET.parse(io.StringIO(ldf._repr_html_())) ParseError: junk after document element: line 1, column 25

Seems to fail on the first <p> in https://github.com/pola-rs/polars/blob/dfa5efe71156c654a1ba3a54b865eae723a818e9/py-polars/polars/lazyframe/frame.py#L783

- `pandas` only supports it for `pd.DataFrame`

dangotbanned · 2025-07-09T16:53:14Z

Possible follow-ups

Just some loose ideas, nothing I'm planning to work on any time soon 😅

Support dask.dataframe.DataFrame._repr_html_
It has one, but it is a bit strange and I wasn't able to parse it (3769326)
Normalize the styling
- The code in polars is fairly short - vendoring it so we can render all the eager backends the same isn't that unrealistic and would be a nice DX improvement
- https://github.com/pola-rs/polars/blob/dfa5efe71156c654a1ba3a54b865eae723a818e9/py-polars/polars/dataframe/_html.py

dangotbanned · 2025-08-02T20:43:55Z

@MarcoGorelli, @FBruzzesi

Thought I'd do one last check before closing this one, here's a few options to choose from:

Don't support this
Do it, but less (defer entirely to polars, pandas)
Do it, but style differently (feat: Use _repr_html_ when native supports it #2776 (comment))
Do it, but increase the scope and make pyarrow + pd.Series look pretty too (feat: Use _repr_html_ when native supports it #2776 (comment))

No worries if we don't want it 🙂

FBruzzesi

@dangotbanned thank to your ping - I got reminded that I once started to look at this, and never finished. I am not against having a good support, yet I am not very useful nor opinionated about this.

I would try to aim for a pareto optimum that balances usefulness and maintainability 😂

FBruzzesi · 2025-07-18T20:02:25Z

narwhals/_utils.py

+    header: Literal["Narwhals DataFrame", "Narwhals LazyFrame", "Narwhals Series"],
+    native_html: str,
+) -> str | None:  # pragma: no cover
+    if header == "Narwhals LazyFrame" and "LazyFrame" in native_html:
+        html = native_html.replace("LazyFrame", "LazyFrame.to_native()")
+        return f"{html}<p><b>{header}</b></p>"


I am mostly nitpicking here but... isn't the header actually a footer? 😂

You're quite right 😂

It started as a header until I ran into (#2776 (comment))

I should've updated that to footer or caption

Oh I forgot, the name header actually came from generate_repr

narwhals/narwhals/_utils.py

Line 1515 in e32e139

def generate_repr(header: str, native_repr: str) -> str:

Anyway - updated it in (57e333d)

FBruzzesi · 2025-07-18T20:04:08Z

narwhals/_utils.py

+        tree.getroot().insert(0, style)
+    buf = io.BytesIO()
+    tree.write(buf, "utf-8", method="html")
+    return buf.getvalue().decode()


Everything else in this function is a new language to me - I am not very helpful

Ah yeah xml.etree.elementtree is a bit of a strange one

I had to learn a bit of lxml once to fix a particularly broken file.
The API of that is based on this stdlib module, but was more ergonoic than this mess 😄

To simplify this:

Element: Is a HTML Element

Tree: Refers to a document/webpage, but in this case it is just a table

So I'm essentially doing a fancy find/replace, but trying to preserve the structure of the document

FBruzzesi · 2025-08-02T21:23:23Z

narwhals/_utils.py

+    style_css = (
+        ".dataframe caption { "
+        "caption-side: bottom; "
+        "text-align: center; "
+        "font-weight: bold; "
+        "padding-top: 8px;"
+        "}"
+    )


That's very reasonable!

dangotbanned · 2025-08-03T12:51:28Z

#2776 (comment)

Do it, but increase the scope and make pyarrow + pd.Series look pretty too

I've started #2925 and came up against the pyarrow.ChunkedArray repr again, while writing an example 🤦‍♂️

import pyarrow as pa

import narwhals as nw

>>> nw.Series.from_iterable("a", [4, 1, 3, 2], dtype=nw.UInt32, backend=pa)
┌───────────────────────────────────────────────────────┐
|                    Narwhals Series                    |
|-------------------------------------------------------|
|<pyarrow.lib.ChunkedArray object at 0x0000017129497880>|
|[                                                      |
|  [                                                    |
|    4,                                                 |
|    1,                                                 |
|    3,                                                 |
|    2                                                  |
|  ]                                                    |
|]                                                      |
└───────────────────────────────────────────────────────┘

Even if we don't go ahead with _repr_html_ - I'd really like to be displaying Series.name and Series.dtype in __repr__

The polars one manages to fit in both of those + shape, while taking up waaaay less horizontal space and 1 fewer lines:

>>> nw.Series.from_iterable("a", [4, 1, 3, 2], dtype=nw.UInt32, backend="polars")
┌─────────────────┐
| Narwhals Series |
|-----------------|
|shape: (4,)      |
|Series: 'a' [u32]|
|[                |
|        4        |
|        1        |
|        3        |
|        2        |
|]                |
└─────────────────┘

#2776 (comment)

Related #2776 (comment)

dangotbanned · 2025-08-15T14:05:30Z

#2776 (comment)

If we just wanted pa.ChunkedArray to look nicer, I've got a very naive new repr (not html) for nw.Series:

shape: (365,)
dtype: Datetime(time_unit='us', time_zone=None)
name: 'time series'
nw.Series[pyarrow]
[
	2009-01-02 00:00:00
	2009-01-03 00:00:00
	2009-01-04 00:00:00
	2009-01-05 00:00:00
	2009-01-06 00:00:00
	…
	2009-12-28 00:00:00
	2009-12-29 00:00:00
	2009-12-30 00:00:00
	2009-12-31 00:00:00
	2010-01-01 00:00:00
]

shape: (30,)
dtype: UInt32
name: 'lower max rows'
nw.Series[pyarrow]
[
	0
	1
	2
	…
	27
	28
	29
]

shape: (30,)
dtype: Int16
name: 'oh pandas too???'
nw.Series[pandas]
[
	29
	28
	27
	26
	25
	24
	…
	5
	4
	3
	2
	1
	0
]

Would be nicer-er if we used the short type codes from polars

dangotbanned added enhancement New feature or request pandas-like Issue is related to pandas-like backends polars Issue is related to polars backend labels Jul 3, 2025

dangotbanned added 2 commits July 3, 2025 14:29

Merge branch 'main' into df-repr-html

04e1f51

Merge branch 'main' into df-repr-html

bf7e87a

dangotbanned commented Jul 3, 2025

View reviewed changes

dangotbanned added 4 commits July 9, 2025 14:42

Merge remote-tracking branch 'upstream/main' into df-repr-html

edb7ae4

feat: Support pl.LazyFrame._repr_html_

3769326

- `pandas` reuses the eager version - `pyarrow` doesn't support - `ibis` requires changing global config, so skipping that - `dask` does have a `_repr_html_`, but doesn't parse well

Merge remote-tracking branch 'upstream/main' into df-repr-html

3559cf8

chore: add note on testing

ea42e83

dangotbanned commented Jul 9, 2025

View reviewed changes

feat: Support pl.Series._repr_html_

3748f23

- `pandas` only supports it for `pd.DataFrame`

dangotbanned marked this pull request as ready for review July 9, 2025 16:53

dangotbanned added 9 commits July 11, 2025 20:58

Merge branch 'main' into df-repr-html

5bb73d8

Merge branch 'main' into df-repr-html

e77c8de

Merge branch 'main' into df-repr-html

d8f09a4

Merge branch 'main' into df-repr-html

4227f10

Merge branch 'main' into df-repr-html

690e6e4

Merge branch 'main' into df-repr-html

6f2ab6b

Merge branch 'main' into df-repr-html

61497d6

Merge remote-tracking branch 'upstream/main' into df-repr-html

c4dca25

Merge branch 'main' into df-repr-html

f73d564

FBruzzesi reviewed Aug 2, 2025

View reviewed changes

dangotbanned marked this pull request as draft August 3, 2025 15:22

dangotbanned added 2 commits August 3, 2025 16:05

Merge remote-tracking branch 'upstream/main' into df-repr-html

e32e139

rename header -> caption_text

57e333d

#2776 (comment)

dangotbanned added 2 commits August 3, 2025 16:50

change caption-side to top

7758265

Related #2776 (comment)

Merge remote-tracking branch 'upstream/main' into df-repr-html

dfb5f5e

dangotbanned mentioned this pull request Aug 11, 2025

feat: Add {Expr,Series}.is_close #2962

Merged

10 tasks

dangotbanned mentioned this pull request Oct 16, 2025

nw.DType, pl.DataType feature parity #3214

Open

feat: Use _repr_html_ when native supports it #2776

Are you sure you want to change the base?

feat: Use _repr_html_ when native supports it #2776

Uh oh!

Conversation

dangotbanned commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Uh oh!

dangotbanned Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Possible follow-ups

Uh oh!

dangotbanned commented Aug 2, 2025

Uh oh!

FBruzzesi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned commented Aug 3, 2025

Uh oh!

dangotbanned commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: Use `_repr_html_` when native supports it #2776

feat: Use `_repr_html_` when native supports it #2776

dangotbanned commented Jul 3, 2025 •

edited

Loading

dangotbanned Jul 3, 2025 •

edited

Loading

dangotbanned commented Jul 9, 2025 •

edited

Loading

dangotbanned Aug 2, 2025 •

edited

Loading

dangotbanned commented Aug 15, 2025 •

edited

Loading