Decrypt file on the fly

[hugocatlas]

Dear smart-open community,

I'm using smart-open for a while now and I'm particularly happy with the possibility of reading a file from a SFTP and writing it directly into S3.
I now have a new client that will start sending openssl-encrypted files over the SFTP.
I would love to know if smart-open is able to handle this use case, in other words, decrypting the file while reading it from SFTP and writing it to S3.

My MVP pipeline did something like:

os.system(f"openssl enc -pbkdf2 -in {input_filepath} -out {output_filepath} -d -aes256 -k {key_filepath}")

Now that I need to industrialize it, I would love to make it streamlined: without having to download the file from the SFTP locally, decrypt it, then load it into S3.

Is smart-open able to deal with this? Else, do you have any hints on how I can achieve this?

Cheers,

Hugo

[mpenkov]

• Update your pipeline to write to stdout instead out output_filepath
• Write a utility that reads from stdin and writes to S3 (using smart_open)
• Pipe the output of the updated pipeline to your new utility

[hugocatlas]

Thank you, @mpenkov, for this hint.

Indeed, I got it working through a stream.
However, it only works if both input (encrypted file) and output (decrypted file) are locally.
If I try to do this on S3 natively (let's drop the SFTP requirement by now), I get an error like:

File "/Users/hugocatlas/.pyenv/versions/3.10.6/lib/python3.10/subprocess.py", line 1606, in _get_handles
    p2cread = stdin.fileno()
io.UnsupportedOperation: fileno

The problem is that the subprocess.Popen can accept stdin and stdout to be file objects only if they are io.BufferedReader and io.BufferedWriter instances, respectively. When dealing with S3 files, smart-open returns smart_open.s3.Reader and smart_open.s3.MultipartWriter instances, respectively, and that's what is making it incompatible with subprocess.

My temporary workaround is to transfer it locally, do my decryption thing, and upload it again.

Here is a script that tests all strategies.

import subprocess
import traceback
from smart_open import open as sopen

aws_access_key_id = <access key id>
aws_secret_access_key = <secret access key>
bucket_name = < bucket name>
directory = <directory in bucket>
filename = <encrypted filename>
decrypted_filename =  <decrypted filename>
keyfilename = <key to decrypt>

input_s3_fileobject_string = f"s3://{aws_access_key_id}:{aws_secret_access_key}@{bucket_name}/{directory}/{filename}"
output_s3_fileobject_string = f"s3://{aws_access_key_id}:{aws_secret_access_key}@{bucket_name}/{directory}/{decrypted_filename}"

def _decrypt_with_openssl(input, output):
    p = subprocess.Popen((f"openssl enc -pbkdf2 -d -aes256 -k {keyfilename}"), stdin=input, stdout=output, stderr=subprocess.STDOUT, shell=True,)
    assert p.wait() == 0

def decrypt_local():
    with sopen(filename, 'rb') as finp:
        with sopen(decrypted_filename, 'wb') as fout:
            print(type(finp)) # <class '_io.BufferedReader'>
            print(type(fout)) # <class '_io.BufferedWriter'>
            _decrypt_with_openssl(input=finp, output=fout)

def decrypt_hybrid1():
    with sopen(filename, 'rb') as finp:
        with sopen(output_s3_fileobject_string, 'wb') as fout:
            print(type(finp)) # <class '_io.BufferedReader'>
            print(type(fout)) # <class 'smart_open.s3.MultipartWriter'>
            _decrypt_with_openssl(input=finp, output=fout)

def decrypt_hybrid2():
    with sopen(input_s3_fileobject_string, 'rb') as finp:
        with sopen(decrypted_filename, 'wb') as fout:
            print(type(finp)) # <class 'smart_open.s3.Reader'>
            print(type(fout)) # <class '_io.BufferedWriter'>
            _decrypt_with_openssl(input=finp, output=fout)

def decrypt_hybrid3():
    with sopen(input_s3_fileobject_string, 'rb') as finp:
        with sopen(filename, 'wb') as fout:
            print(type(finp)) # <class 'smart_open.s3.Reader'>
            print(type(fout)) # <class '_io.BufferedWriter'>
            for content in finp:
                fout.write(content)

    decrypt_local()

    with sopen(decrypted_filename, 'rb') as finp2:
        with sopen(output_s3_fileobject_string, 'wb') as fout2:
            print(type(finp2)) # <class 'smart_open.s3.Reader'>
            print(type(fout2)) # <class '_io.BufferedWriter'>
            for content in finp2:
                fout2.write(content)


def decrypt_remote():
    with sopen(input_s3_fileobject_string, 'rb') as finp:
        with sopen(output_s3_fileobject_string, 'wb') as fout:
            print(type(finp)) # <class 'smart_open.s3.Reader'>
            print(type(fout)) # <class 'smart_open.s3.MultipartWriter'>
            _decrypt_with_openssl(input=finp, output=fout)

if __name__ == "__main__":
    strategies = {"fully local": decrypt_local, "source local, target remote": decrypt_hybrid1, "source remote, target local": decrypt_hybrid2, "copying local first": decrypt_hybrid3, "fully remote": decrypt_remote}

    for strategy, function in strategies.items():
        try:
            print(f"\nDecrypt {strategy}")
            function()
        except Exception:
            print("\n", traceback.format_exc())
        else:
            print(f"Strategy {strategy} is a success!")

I tried converting the smart-open objects into io's instance, without success.
Do you know how we can improve this strategy?

Thank you,

Hugo

[mpenkov]

Try running smart_open in its own subprocess.

You essentially want something like

openssl enc -pbkdf2 -in {input_filepath} -out {output_filepath} -d -aes256 -k {key_filepath} | python upload_to_s3.py ...

Note the | (pipe). You could possibly run everything in the same subprocess using named pipes, but I've never used them in Python, so I can't help you there.

[hugocatlas]

This is indeed the easiest solution.
The aim of using smart_open would be to not have to download the file on the pod my app is running on.
The files I'm dealing with are +10Gb and my application pods are really limited in disk storage, hence the requirement of using a stream.

I'm sticking with my workaround for now, but if by any chance the team is looking for new features, having the possibility of plugging a subprocess in between two files open by smart_open, it would be awesome!

[mpenkov]

Smart_open doesn't provide files, it provides streams that are file-like objects. That's the reason you're unable to communicate between different subprocess with smart_open. Providing file object is simply out of scope for us.