Skip to content

[Feature] [SC-179831] Preliminary support for ranged downloads in SDK #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[Feature] [SC-179831] Preliminary support for ranged downloads in SDK #1

wants to merge 5 commits into from

Conversation

@nickles-lee
Copy link
Owner

@nickles-lee nickles-lee commented Oct 30, 2024 •
edited
Loading

Changes

  • Adds a class that exposes a ranged download as a seekable BinaryIO stream
  • The seekable stream will re-open any closed streams, accounting for some faults and expected socket timeouts due to inactivity, etc.

Known issues

  • [DECO-23875] results in the download being performed in full before the byte array is streamed. This will cause negative performance impacts with this PR until it is fixed.
  • Integration testing is possible via an adhoc script, will follow in a separate PR
  • CI jobs are only run on the DB main fork;

Tests

  • [✅] make test run locally
  • [✅] make fmt applied
  • [🔁] relevant integration tests applied. This'll be added in a separate PR merging onto this branch for reasons

See databricks#807 for CI results

Branch Model

  • All PRs for this feature will be merged onto a common feature branch that will be merged to the main branch in one go.

@nickles-lee nickles-lee changed the title [SC-179831] Preliminary support for ranged downloads in SDK SC-179831 Preliminary support for ranged downloads in SDK Oct 30, 2024
@nickles-lee nickles-lee changed the title SC-179831 Preliminary support for ranged downloads in SDK [Feature] [SC-179831] Preliminary support for ranged downloads in SDK Oct 30, 2024
nickles-lee
nickles-lee commented Oct 30, 2024
View reviewed changes
databricks/sdk/__init__.py
from databricks.sdk import azure
from databricks.sdk.credentials_provider import CredentialsStrategy
from databricks.sdk.mixins.compute import ClustersExt
from databricks.sdk.mixins.files import DbfsExt
Copy link
Owner Author

@nickles-lee nickles-lee Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File is autogen

nickles-lee
nickles-lee commented Oct 30, 2024
View reviewed changes
databricks/sdk/service/apps.py
attempt += 1
raise TimeoutError(f'timed out after {timeout}: {status_message}')

def wait_get_app_stopped(self,
Copy link
Owner Author

@nickles-lee nickles-lee Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File is autogen

nickles-lee
nickles-lee commented Oct 30, 2024
View reviewed changes
databricks/sdk/service/compute.py
attempt += 1
raise TimeoutError(f'timed out after {timeout}: {status_message}')

def wait_command_status_command_execution_finished_or_error(
Copy link
Owner Author

@nickles-lee nickles-lee Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File is autogen

nickles-lee
nickles-lee commented Oct 30, 2024
View reviewed changes
databricks/sdk/service/pipelines.py
def __init__(self, api_client):
self._api = api_client

def wait_get_pipeline_idle(
Copy link
Owner Author

@nickles-lee nickles-lee Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File is autogen

@nickles-lee nickles-lee closed this Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nickles-lee