Decouple file upload from BE

[davidjovanovic]

When adding a new dataset through the UI, we can type in an external URL or select a file. That file will be sent thorugh the UI to the BE and uploaded to S3.

For the moment the size limitation on the BE is "unlimited" (tested with 7gb file). It is achieved by chunking the file inputStream and using these properties (not sure if all are needed):
spring.servlet.multipart.max-file-size=-1 spring.servlet.multipart.max-request-size=-1 spring.servlet.multipart.enabled=true server.tomcat.max-http-form-post-size=-1

Another possibility and improvement could be to first upload the file to S3 and create an API on BE that will send a list of files from the preffered bucket to the UI, then on the UI when creating a new Dataset we would directly select a file and "atach" it to the Dataset.

This could make the upload process faster and give the possibility of easier integration with solutions that already use S3 as a storage.

1. Upload file through minio UI:
   Pros:

• if uploading big files through UI still causes issue this could potentialy solve it
  Cons:
• forcing user to switch platform and use another UI for uploading files (bad UX, additional instructions, users not reading manuals)

2. PresignedPUT throguh UI for uploading big files to S3:
3. Bucket already contains files (choose existing file from S3 and link it with a dataset):
4. Create Artifact CRUD management page on UI for S3 (uploading using presigned PUT,DELETE):

[IgorBalog-Eng]

What we should take into consideration is that in this case, when bucekt/file is already present in S3, we would also need to have accessKey/secretKey for user/policy who can access this bucekt/object.

[davidjovanovic]

Investigate how current logic works if the file is not yet uploaded. Can you obtain the presignedURL i the upload is still ongoing? Can you start downloading if the file is still uploading? Same goes for listing all objects from S3.

[IgorBalog-Eng]

Idea might be to have once flow for dataset creation:

• bucket already contains file - uploaded via Artifact Management logic or by some other system
• on dataset, we should call get all files from bucket
• choose file from bucket
• we should also provide accessKey and secretKey attached to the bucket/file policy to avoid using root credentials for presignURL GET
• continue with dataset creation

This will result in unique dataset creation flow, to avoid this use case from:

• dataset metadata
• upload file into bucket (preferably using presignPUT)
• wait for upload to finish OR update dataset after finish in background (causing problems for using incomplete dataset)
• save dataset
  Depending on upload speed this can take some time, since it is required to have objectKey for dataset creation