Reading a TDMS file on Apache Athena

[wz2b]

Hi, I am trying to use nptdms on apache athena (which is really pyspark). I was able to install nptdms on athena with no real problems. Now, as I try to parallelize/iterate through files, I'm not sure what to do next. In Athena, you load binary files using their version of a DataFrame, where one of the rows of the dataframe is content - basically it's a bytearray.

How do I load that into nptdms? I don't see an obvious way to just feed it the bytes.

--C

[adamreeve]

Hi @wz2b, npTDMS works with any file like object. You'll probably want to create an io.BytesIO instance from your bytearray to read it into npTDMS. See https://docs.python.org/3/library/io.html#io.BytesIO

[wz2b]

Do you think there's a way to feed an s3://bucket/whatever.tdms stream right into it?

…

On Sun, May 21, 2023 at 12:44 PM Adam Reeve ***@***.***> wrote: Hi @wz2b <https://github.com/wz2b>, npTDMS works with any file like object. You'll probably want to create an io.BytesIO instance from your bytearray to read it into npTDMS. See https://docs.python.org/3/library/io.html#io.BytesIO — Reply to this email directly, view it on GitHub <#305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALTMOOSGIMRVEHPQHBKBSLXHJA5DANCNFSM6AAAAAAYJDMSHY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[adamreeve]

What type of Python object is your S3 stream?

[wz2b]

From the AWS client s3 library it would be a StreamingBody: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html#botocore.response.StreamingBody which does have a method named read() ... so it seems like that might work

…

On Sun, May 21, 2023 at 1:02 PM Adam Reeve ***@***.***> wrote: What type of Python object is your S3 stream? — Reply to this email directly, view it on GitHub <#305 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALTMOJXLDXQ4QD6NCJIKYLXHJDBTANCNFSM6AAAAAAYJDMSHY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[adamreeve]

Ah actually I don't think this will work as npTDMS needs to be able to seek back to arbitrary positions in the file, so won't work when streaming (and the StreamingBody class doesn't implement the seek method). So you will need to read the whole file into a bytearray or bytes first.