Skip to content

Redundant allocation and memory copy in ranged S3 reader _read_unbuffered #380

@clumsy

Description

@clumsy

s3torchconnector version

1.4.3

s3torchconnectorclient version

1.4.3

AWS Region

No response

Describe the running environment

Any environment

What happened?

chunk is PyBytes and chunk[:chunk_size] will create and copy this bytearray in https://github.com/awslabs/s3-connector-for-pytorch/blob/main/s3torchconnector/src/s3torchconnector/s3reader/ranged.py#L151

Instead we should use memoryview to use lightweight pointer arithmetic

Relevant log output

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions