Skip to content

Make it possible to exclude pyarrow dep #276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
orangetin merged 5 commits into from
Jun 2, 2025
Merged

Make it possible to exclude pyarrow dep #276

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
10a1e74
Move pyarrow to an in method import.
scosman Mar 19, 2025
59c7acb
Make pyarrow optional, and show an error if not installed when trying…
scosman Jun 2, 2025
00c6194
Merge branch 'main' into main
orangetin Jun 2, 2025
3a4a442
fix formatting issue
scosman Jun 2, 2025
d8c25a5
Ran `poetry lock`
scosman Jun 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/together/utils/files.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
from traceback import format_exc
from typing import Any, Dict, List

from pyarrow import ArrowInvalid, parquet

from together.constants import (
MAX_FILE_SIZE_GB,
Expand Down Expand Up @@ -372,6 +371,8 @@ def _check_jsonl(file: Path) -> Dict[str, Any]:


def _check_parquet(file: Path) -> Dict[str, Any]:
# in method import - this allows client to exclude the pyarrow dep if they don't need it. Saved ~80MB and more compatible with older systems.
from pyarrow import ArrowInvalid, parquet
Copy link
Member

@orangetin orangetin May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you wrap this in a try/except with details on how to install this with the dependency group? something like pip install together[parquet]

report_dict: Dict[str, Any] = {}

try:
Expand Down