Skip to content

Conversation

aabmass
Copy link
Member

@aabmass aabmass commented Sep 15, 2025

Description

Follow up to #3752. Beginning of a fsspec based uploader (optional dependency) to gen AI utils. fsspec generically supports many cloud and local backends like /path/on/disk, gs://, s3://..., etc. This PR contains the entry point, environment variable for configuring, Queueing, and the interfaces. Next PR will contain the actual implementation of writing the files.

Part of #3065, #3753

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Added tests for the entry point, optional dependency handling, and queueing.

Does This PR Require a Core Repo Change?

  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated - will add once the feature is complete
  • Unit tests have been added
  • Documentation has been updated

@aabmass aabmass closed this Sep 16, 2025
@aabmass aabmass deleted the genai-upload-fsspec-boilerplate branch September 16, 2025 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant