Lepton Executor fails with `exec /bin/sh: argument list too long`

[sean-smith]

Describe the bug

When using the lepton executor there's a data mover step that uploads the code prior to kicking off the job. This is limited to just 128kb because it's using busybox:1.37.0 container which is a very low ARG_MAX limit. This fails with the error:

exec /bin/sh: argument list too long

Steps/Code to reproduce bug

The data copy command) embeds the entire job directory as a base64-encoded tarball and run it as a shell command: cmd = f"echo {encoded_data} | base64 -d > {dest_path}/archive.tar.gz && tar -xzf ..." causing this to fail.

Expected behavior

This should work for arbitrary amount of files.