Streaming processing of large file - create/update Dataset

[IgorBalog-Eng]

If there is a need that processing large file fails in some case, following code base can be used as starting point for solving this problem:

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-fileupload2-jakarta-servlet6</artifactId>
    <version>2.0.0-M3</version>
</dependency>
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-fileupload2-core</artifactId>
    <version>2.0.0-M3</version>
</dependency>
<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.14.0</version>
</dependency>

Following config file might not be required:

@Configuration
public class MultipartConfig {

    @Bean
    public MultipartResolver multipartResolver() {
        return new StandardServletMultipartResolver();
    }

    @Bean
    public JakartaServletFileUpload fileUpload() {
        return new JakartaServletFileUpload();
    }
}

```

DatasetController.saveDataset

```java
@PostMapping
public ResponseEntity<GenericApiResponse<JsonNode>> saveDataset(HttpServletRequest request) 
```

We can omit existing multipart files, since they will be processed from request into DatasetService:

```java
import org.apache.commons.fileupload2.core.DiskFileItemFactory;
import org.apache.commons.fileupload2.core.FileItemFactory;
import org.apache.commons.fileupload2.jakarta.servlet6.JakartaServletFileUpload;
import org.apache.commons.lang3.StringUtils;

public Dataset saveDataset(HttpServletRequest request) {

 try {
            if (JakartaServletFileUpload.isMultipartContent(request)) {
                FileItemFactory fileItemfactory = DiskFileItemFactory.builder().get();
                final JakartaServletFileUpload fileUpload = new JakartaServletFileUpload(fileItemfactory);
                org.apache.commons.fileupload2.core.FileItemInputIterator iterator = fileUpload.getItemIterator(request);

                String filename = null;
                String contentType = null;
                InputStream fileStream = null;

                // First pass to collect metadata
                while (iterator.hasNext()) {
                    org.apache.commons.fileupload2.core.FileItemInput item = iterator.next();
                    String fieldName = item.getFieldName();

                    if (item.isFormField() && "dataset".equals(fieldName)) {
                        // Deserialize dataset from the form field
                        dataset = CatalogSerializer.deserializePlain(org.apache.commons.io.IOUtils.toString(item.getInputStream(), StandardCharsets.UTF_8), Dataset.class);
                    } else if (item.isFormField() && "url".equals(fieldName)) {
                        externalURL = org.apache.commons.io.IOUtils.toString(item.getInputStream(), StandardCharsets.UTF_8);
                    } else if (item.isFormField() && "authorization".equals(fieldName)) {
                        authorization = org.apache.commons.io.IOUtils.toString(item.getInputStream(), StandardCharsets.UTF_8);
                    }

                    if (!item.isFormField() && "file".equals(fieldName)) {
                        filename = item.getName();
                        contentType = item.getContentType();
                        fileStream = item.getInputStream();
                    }
                }

                // Handle file upload if file part was found
                if (fileStream != null) {
                    try {
                        // Create a buffered input stream that can be reset/reused
                        InputStream bufferedStream = new BufferedInputStream(fileStream);
                        bufferedStream.mark(Integer.MAX_VALUE);

                        String etag = artifactService.uploadLargeArtifact(
                                bufferedStream,
                                filename,
                                contentType,
                                request.getContentLengthLong()
                        ).get();
                        log.info("Artifact uploaded with ID: {}", etag);

                        // Add the artifact to dataset and save
                        Artifact artifact = Artifact.Builder.newInstance()
                                .artifactType(ArtifactType.FILE)
                                .value(filename)
                                .contentType(contentType)
                                .filename(filename)
                                .build();

                        Dataset datasetWithArtifact = addArtifactToDataset(dataset, artifact);
                        savedDataSet = repository.save(datasetWithArtifact);
                        artifactService.saveArtifact(artifact);
                        log.info("File {} uploaded successfully with artifact ID {}", filename, artifact.getId());
                    } finally {
                        fileStream.close();
                    }
                } else if (StringUtils.isNotBlank(externalURL)) {
                    // Handle external URL case
                    Artifact artifact = artifactService.uploadArtifact(dataset.getId(), null, externalURL, authorization);
                    Dataset datasetWithArtifact = addArtifactToDataset(dataset, artifact);
                    savedDataSet = repository.save(datasetWithArtifact);
                } else {
                    // Save dataset without artifact
                    savedDataSet = repository.save(dataset);
                }
            } else {
                // Non-multipart request, just save the dataset
                savedDataSet = repository.save(dataset);
            }
        } catch (Exception e) {
            log.error(e.getMessage(), e);
            throw new InternalServerErrorAPIException("Dataset could not be saved");
        }

        catalogService.updateCatalogDatasetAfterSave(savedDataSet);
        log.info("Inserted Dataset with id {}", savedDataSet.getId());
        return savedDataSet;


}
```

ArtifactService

```java
public CompletableFuture<String> uploadLargeArtifact(InputStream inputStream, String filename, String contentType, long contentLength) {

        String fileId = ToolsUtil.generateUniqueId();

        ContentDisposition contentDisposition = ContentDisposition.attachment()
                .filename(filename)
                .build();

        try {
            return s3ClientService.uploadFile(inputStream,
                            s3Properties.getBucketName(),
                            filename,
                            contentType,
                            contentDisposition.toString())
                    .whenComplete((res, ex) -> {
                        ;
                        log.info("File {} uploaded successfully under id {}", filename, res);
                        try {
                            inputStream.close();
                        } catch (IOException e) {
                            log.warn("Failed to close input stream", e);
                        }
                    });
        } catch (Exception e) {
            log.error("File streaming upload failed", e);
            throw new CatalogErrorAPIException("File streaming upload failed: " + e.getLocalizedMessage());
        }
    }

```

Current codebase does not work, and we have problem that request/response streams are closed after processing request, while there is still request.multipart(file).inputStream being processed to upload into S3 bucket using batch mode.

[IgorBalog-Eng]

This one will be addressed via different task - create CRUD logic for Artifact (upload into bucket)
After that, on dataset creation/update API call will be made to fetch all available files in bucket, and choose one of them.