Skip to content

Conversation

abujabarmubarak
Copy link

What does this PR do?

This PR adds support for engine="polars" in the pandas.read_csv() function. It enables users to leverage the performance of the Polars DataFrame engine when reading CSV files in pandas.


Why is this needed?

This enhancement addresses issue #61813. Since Polars is a high-performance DataFrame library with fast CSV parsing, adding it as an engine allows pandas users to benefit from its speed while staying within the pandas API.


What changes were made?

✅ Added Polars Support in _read() Function

  • Included a conditional block inside the _read() function in pandas/io/parsers/readers.py to handle engine="polars"
  • This helps pandas use polars.read_csv() under the hood and convert the result to a pandas DataFrame using .to_pandas()

✅ Updated Engine Validation

  • Modified _refine_defaults_read() to accept "polars" as a valid engine
  • This ensures pandas doesn’t raise a ValueError when engine="polars" is passed

✅ Created a New Test File

  • Created test_read_csv_polars.py inside pandas/tests/io/parser/
  • The test verifies that using engine="polars" in read_csv() loads a simple CSV correctly
  • Ensures code coverage and prevents future regressions

How to use it?

import pandas as pd

# Requires Polars to be installed
# pip install polars

df = pd.read_csv("example.csv", engine="polars")
print(df.head())

This allows pandas users to benefit from Polars' speed and memory efficiency while still using the familiar pandas API.


Dependencies

Requires the user to have Polars installed:

pip install polars

If polars is not installed, the engine will raise an ImportError with instructions.


Related Issues

Closes #61813


Let me know if any additional tests or validations are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENH: Add Polars engine to read_csv

1 participant