Skip to content

PERF: read list of parquet files at once #5723

Description

@mvashishtha

You can provide a list of local files (but not a list of directories or s3 files) to read_parquet, as in #5698. For #5698 I will make a fix to read the files separately and concat the results. I attempted a more general solution but it required too much surgery on modin's read_parquet code, which assumes in many places that there's just one file or directory to read. One difficulty I had is that ParquetDirectory can take a list of files but not a list of directories, so we can't convert to a list of directories at the beginning.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Minor bugs or low-priority feature requestsPerformance 🚀Performance related issues and pull requests.new feature/request 💬Requests and pull requests for new features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions