Skip to content
Merged
Changes from 24 commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
6d36b4e
attempted outline of how things could look
ym-pett Aug 13, 2025
1062d99
attempting to read in plugins
ym-pett Aug 13, 2025
8d4cda6
linting
ym-pett Aug 13, 2025
cfd156f
trying to see an effect of plugin
ym-pett Aug 13, 2025
e3a2f3b
cleanup after pair session
ym-pett Aug 14, 2025
f65d7bf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 14, 2025
a133796
Merge branch 'narwhals-dev:main' into create_fromnative_daft
ym-pett Aug 14, 2025
eee9068
think we're managing to import the class
ym-pett Aug 14, 2025
c3a8c4d
changes mean we can read daft df, joy
ym-pett Aug 14, 2025
ca01346
Merge branch 'main' into create_fromnative_daft
ym-pett Aug 15, 2025
c4611fe
nicer error handling, not there yet
ym-pett Aug 16, 2025
90ad973
nicer error handling, not there yet
ym-pett Aug 16, 2025
76232df
added some thoughts, currently going in circles
ym-pett Aug 16, 2025
ebc1a8f
added some thoughts, currently going in circles
ym-pett Aug 16, 2025
15d9c8c
work from pair session
ym-pett Aug 18, 2025
16832f9
more explicit error handling
ym-pett Aug 18, 2025
9e11a8f
error handling now passes ruff check
ym-pett Aug 18, 2025
d8663d8
moved plugins back to end, raising general Exception
ym-pett Aug 19, 2025
34e7fb6
silencing ruff errors, deleted t.py
ym-pett Aug 19, 2025
8a2b019
Merge branch 'main' into create_fromnative_daft
ym-pett Aug 19, 2025
f92cdbf
Merge branch 'main' into create_fromnative_daft
ym-pett Aug 19, 2025
54509d2
Merge branch 'main' into create_fromnative_daft
ym-pett Aug 19, 2025
941edc4
Merge branch 'main' into create_fromnative_daft
ym-pett Aug 19, 2025
72a5df4
checking for version and preventing tests on plugin codeblock
ym-pett Aug 19, 2025
e783af7
wip implementing marco's proposal
ym-pett Aug 20, 2025
4cf2f96
new version passes tests
ym-pett Aug 20, 2025
a52a6bb
discover_plugins function and fixed pragma no cover
ym-pett Aug 20, 2025
a4e539a
added pragma, removed group argument
ym-pett Aug 22, 2025
aca8cc4
Merge branch 'main' into create_fromnative_daft
ym-pett Aug 22, 2025
02d544a
wip: add test-plugin
MarcoGorelli Aug 22, 2025
0952cd9
wip
MarcoGorelli Aug 22, 2025
de0e5ce
wip
MarcoGorelli Aug 22, 2025
9dfcd09
fixup
MarcoGorelli Aug 22, 2025
b999e9a
remove unused function, fixup type ignore
MarcoGorelli Aug 22, 2025
3460f4e
install test-plugin in CI
MarcoGorelli Aug 22, 2025
1c246dc
pass Version down
MarcoGorelli Aug 22, 2025
ab8303d
fixup, remove more defaults
MarcoGorelli Aug 22, 2025
a8501f5
coverage
MarcoGorelli Aug 22, 2025
19dc900
actually stage test file
MarcoGorelli Aug 22, 2025
42f2df8
remove daft traces
MarcoGorelli Aug 22, 2025
d6a384a
rename
MarcoGorelli Aug 22, 2025
9cf290d
install test_plugin in makefile
MarcoGorelli Aug 22, 2025
bdd71e7
coverage
MarcoGorelli Aug 22, 2025
bde288f
fix typing (for real this time)
MarcoGorelli Aug 22, 2025
7035533
aah ruff check was removing the not-really-unused import
MarcoGorelli Aug 22, 2025
4868aab
lru_cache -> cache
MarcoGorelli Aug 22, 2025
afd8620
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 1, 2025
b09d3ad
merged main into branch to resolve conflicts
ym-pett Sep 4, 2025
1a73ab8
feat(suggestion): `<3.10` support?
dangotbanned Sep 4, 2025
1481d48
fix: don't expect plugins?
dangotbanned Sep 4, 2025
59589c1
chore(typing): Ignore unimplemented
dangotbanned Sep 4, 2025
6888f78
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 8, 2025
8f22fdb
wip: hybrid approach to importing
ym-pett Sep 8, 2025
b3f2b60
wip: checkpoint
ym-pett Sep 8, 2025
422ec70
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 8, 2025
b0b524b
damn I broke the plugin tests
ym-pett Sep 8, 2025
0960d69
wip: fixed local mess in test_plugin
ym-pett Sep 8, 2025
3207de1
wip: fixed local mess in test_plugin
ym-pett Sep 8, 2025
d951220
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 8, 2025
ea28de2
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 9, 2025
b563ace
added not_implemented functions to namespace
ym-pett Sep 9, 2025
9847793
removed comment
ym-pett Sep 9, 2025
1f38e31
changed naming for plugin entrypoins
ym-pett Sep 9, 2025
5dee0bb
added not_implemented functions to dictnamespace
ym-pett Sep 9, 2025
c508ab0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2025
f2d6e57
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 9, 2025
6c65fd4
fix(typing): Add `py.typed` marker
dangotbanned Sep 9, 2025
b26d7d4
fix: Fully qualify imports (from `tests`)
dangotbanned Sep 9, 2025
22362ad
added protocol & plugin detection function
ym-pett Sep 11, 2025
d22f046
changed plugin contract
ym-pett Sep 11, 2025
028e473
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 11, 2025
87e67bf
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 12, 2025
5c16c7f
added is_native back in, adapted plugin tests
ym-pett Sep 12, 2025
e00c7c6
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 15, 2025
0c69762
wip: moving plugins
ym-pett Sep 15, 2025
f7765e8
wip: fixed imports in plugins utils
ym-pett Sep 15, 2025
e3e5ec7
refactored plugin-related utils into their own file
ym-pett Sep 15, 2025
0cd5f2f
refactor: Make `plugins` a module instead of a package
dangotbanned Sep 16, 2025
d511fc0
chore(typing): ignore `@cache` warning
dangotbanned Sep 16, 2025
6dd4d1c
refactor: Expose as `plugins.from_native`
dangotbanned Sep 16, 2025
8a96d84
fix(typing): Add the `Plugin` annotation I forgot
dangotbanned Sep 16, 2025
1944e47
docs(DRAFT): Start working on `plugins.from_native`
dangotbanned Sep 16, 2025
ce712e0
feat(typing): Add some slightly narrower typing
dangotbanned Sep 16, 2025
7bcec61
docs: Update `plugins.from_native`
dangotbanned Sep 16, 2025
9bbc0a6
Merge pull request #2 from narwhals-dev/plugin/dgb
ym-pett Sep 25, 2025
0f65066
Merge pull request #1 from ym-pett/tidy_plugin_utils
ym-pett Sep 25, 2025
8ad25ea
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 25, 2025
21aab4a
Merge branch 'main' into create_fromnative_daft
ym-pett Sep 30, 2025
0138803
fix mess in pytest.yml
ym-pett Sep 30, 2025
fa09ba6
modified pytest.yml
ym-pett Sep 30, 2025
213cc31
Update pytest.yml
ym-pett Sep 30, 2025
da2d115
Update pytest.yml
ym-pett Sep 30, 2025
3a10b68
added _with_native to plugintest
ym-pett Sep 30, 2025
2d3f034
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 8, 2025
547db41
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 8, 2025
afd4d83
added is_native not implemented to plugin tests
ym-pett Oct 8, 2025
910a31e
defined is_native properly
ym-pett Oct 8, 2025
f227c13
using not_implemented and nocover to silence plugin test failures
ym-pett Oct 8, 2025
3ef415d
removed duplicate definition
ym-pett Oct 8, 2025
793a1cd
trying to silence colums coverage error
ym-pett Oct 8, 2025
c0d9d00
seeing if no cover works in this file
ym-pett Oct 8, 2025
725ce3c
implementing _with_version to stop test failing, cleaned up unnecessa…
ym-pett Oct 10, 2025
bd95e6d
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 10, 2025
150dc52
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 12, 2025
dc82860
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 13, 2025
41a64c0
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 13, 2025
242d8dd
removing skip for lower versions as it now runs for all
ym-pett Oct 13, 2025
62ea8c9
added pytest to 39 and windows
ym-pett Oct 13, 2025
29d7b8e
fixing TYP001 guard import
ym-pett Oct 14, 2025
1b877d4
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 15, 2025
f7e5ae9
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 16, 2025
ee37848
fixing import order
ym-pett Oct 16, 2025
28ca7fd
fixing typos
ym-pett Oct 16, 2025
19d08e1
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 18, 2025
840c65f
Update docs/extending.md
ym-pett Oct 18, 2025
4550ed3
Update docs/extending.md
ym-pett Oct 18, 2025
79ce894
made suggested changes to docs
ym-pett Oct 18, 2025
a49e214
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 20, 2025
b980628
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 20, 2025
0cdb198
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 21, 2025
4a3f43b
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 24, 2025
3f06f3a
Merge branch 'main' into create_fromnative_daft
ym-pett Oct 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions narwhals/translate.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from __future__ import annotations

import datetime as dt
import sys
from decimal import Decimal
from functools import wraps
from typing import TYPE_CHECKING, Any, Callable, Literal, TypeVar, overload
Expand Down Expand Up @@ -532,6 +533,27 @@ def _from_native_impl( # noqa: C901, PLR0911, PLR0912, PLR0915
raise TypeError(msg)
return Version.V1.dataframe(InterchangeFrame(native_object), level="interchange")

from importlib.metadata import entry_points

discovered_plugins = entry_points(group="narwhals.plugins")

Copy link
Contributor Author

@ym-pett ym-pett Aug 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've sprinkled # pragma: no cover rather liberally, hoped this would prevent the tests from running.

Plus I thought the if statement (if sys.version_info >= (3, 10):) would prevent lower versions from testing for the plugin - do the test failures indicate this still runs? I.e. the references to 3.9, have I got that right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no cover prevents the lines from being reported in the coverage report, but the tests still get executed

the problematic part for python 3.9 is entry_points(group="narwhals.plugins") so you'll need to put that inside the if sys.version_info block

if sys.version_info >= (3, 10):
for plugin in discovered_plugins:
obj = plugin.load() # pragma: no cover

try:
df_compliant = obj.from_native( # pragma: no cover
native_object, eager_only=False, series_only=False
)
except Exception as e:
if "daft" in str(type(native_object)): # pragma: no cover
msg = "Hint: you might be missing the `narwhals-daft` plugin"
raise Exception(msg) from e # noqa: TRY002
# continue looping over the plugins
continue # pragma: no cover
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't say I'm keen on either β‘  catching all Exceptions or β‘‘ having plugin specific error messages at the Narwhals level.

  1. If there's some type of conversion error within the plugin during from_native then we suggest to the user that they need to install a package that is already installed.
  2. Breaks encapsulation within the plugin (e.g. if I want my plugin to have nice error messages, I need to make an upstream change).

It may be worth revisiting a registration pattern for this. If we have dynamically populated global mapping like:

MAPPING: dict[type, type] = {
    daft.DataFrame: narwhals_daft.DaftLazyFrame
    pandas.DataFrame: narwhals._pandas_like.PandasLikeDataFrame # or maybe the MAPPING only stores plugins instead of backends that exist in the main Narwhals package?
    # etc.
}

Then from_native becomes:

def from_native(obj, …):
    try:
        compliant = MAPPING[obj]
    except KeyError:
        msg = f'Narwhals does not understand type(obj)}'
        raise TypeError( nsg)
    return compliant(obj,β€ˆβ€¦)

This separates the question of "should we be able to handle this type of object?" from the "let's attempt to handle this object". The main Narwhals package would only need to create a global mapping that plugins can populate.

To handle keywords like series_only and eager_only we can embed those arguments in the key of the mapping or have a nested hierarchy. Would be happy to discuss this idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @camriddell thanks for the input, much appreciated! I'll play with it and make sure I understand, if I've got questions I'll get back to you. This certainly looks more explicit than what I've already got but I need to do a bit of work to absorb fully.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for comments!

OK, here's another idea then. Just before line 350, we do something like:

    for plugin in discovered_plugins:
        obj = plugin.load()  # pragma: no cover
        if obj.is_native_object(native_object):
            native_object = obj.from_native(native_object,version=version)
            break

Like this, if obj_from_native succeded in converting native_object, then it'll go down one of the if is_compliant_dataframe(native_object): / if is_compliant_lazyframe(native_object): / if is_compliant_series(native_object): blocks, and they already deal with all the series_only / eager_only / pass_though complexities

Regarding the error message, happy to leave them out of this PR for the time being - let's aim to get the simplest but functional change merged, and then we can iterate on improving the user experience

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try out!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MarcoGorelli, can I check: the is_native_object function should live in narwhals-daft? In which case it can only check for it being a daft-type (I still need to figure out).

or should it live in the _namespace file in narwhals with the other is_native... function, in which case I imagine it should check for a more general type (also need to figure out)?

Copy link
Contributor Author

@ym-pett ym-pett Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the moment I'm working on the first assumption but happy to change!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup exactly, is_native_object should be defined in narwhals-daft. then each plugin can have its own definition of native object

you'll just need to check if it's an instance of daft.DataFrame

else:
return df_compliant.to_narwhals() # pragma: no cover

if not pass_through:
msg = f"Expected pandas-like dataframe, Polars dataframe, or Polars lazyframe, got: {type(native_object)}"
raise TypeError(msg)
Expand Down
Loading