Enhancement: job attachments interface does not use storage profiles by default

[crowecawcaw]

Describe the bug

The S3AssetManager class which is the public interface for job attachments functionality does not use storage profiles for default.

The S3AssetManager class takes a farm_id, queue_id, and session in its constructor. Then later, the prepare_paths_for_upload method takes storage_profile as an optional parameter. It's not terrible difficult to fetch the storage profile and pass it in, but passing in storage profiles is easy to miss, possible to mess up, and should be done automatically. If S3AssetManager already has the queue ID, it can find the storage profile to use internally and automatically.

deadline-cloud/src/deadline/job_attachments/upload.py

Line 1264 in dc67589

storage_profile: Optional[StorageProfile] = None,

deadline-cloud/src/deadline/job_attachments/upload.py

Lines 861 to 868 in dc67589

	def __init__(
	self,
	farm_id: Optional[str] = None,
	queue_id: Optional[str] = None,
	job_attachment_settings: Optional[JobAttachmentS3Settings] = None,
	asset_uploader: Optional[S3AssetUploader] = None,
	session: Optional[boto3.Session] = None,
	asset_manifest_version: ManifestVersion = ManifestVersion.v2023_03_03,

More broadly, it'd be nice if there were a single create_job function that wrapped up the job inputs and job attachments together so that consumers of the libraries can simply pass in a list of paths with the job and don't need to know how job attachments, shared storage, and storage profiles all interact.

Expected Behaviour

Job attachments uses a queue's storage profile by default.

Current Behaviour

Job attachments needs to be opted in to using storage profiles.

Reproduction Steps

n/a

Environment

n/a

[mwiebe]

More broadly, it'd be nice if there were a single create_job function that wrapped up the job inputs and job attachments together so that consumers of the libraries can simply pass in a list of paths with the job and don't need to know how job attachments, shared storage, and storage profiles all interact.

The function deadline.client.api.create_job_from_job_bundle is intended to serve this purpose. You can see the implementation and docstring of it here:

deadline-cloud/src/deadline/client/api/_submit_job_bundle.py

Line 370 in 53c507e

def create_job_from_job_bundle(

You can see an example use of the function in this sample: https://github.com/aws-deadline/deadline-cloud-samples/blob/mainline/conda_recipes/submit-package-job-script.py#L463

[crowecawcaw]

The create_job_from_job_bundle is the job submitting function I was describing, so please ignore that part of the ask.

The original issue - using storage profiles by default - I think still makes sense. If the job attachments code takes farm and queue IDs, it should by default use storage profiles. I was helping someone support storage profiles, and it required them to learn about how storage profiles worked, figure out where to fetch them from, and then pass them into this function. Since storage profiles are a core part of how Deadline handles assets, it'd make sense to use them default and minimize the effort library consumers need to do to for consistent asset handling.

[jericht]

I think this is more of a feature request than a bug, it's a usability enhancement. The current behaviour is not broken, marking this as an enhancement