[DEV-12528] Remove Hadoop copy merge step from Spark downloads

[aguest-kc]

Description:

Remove the Hadoop copy merge step from Spark downloads and use the correct number of partitions instead of 1.

Technical Details:

Updated the number of partitions to be total records / max records per file instead of 1. Skipped the Hadoop copy merge step since the previous change results in the correct number of records in each row. Added a method for renaming the part files to the expected format.

Requirements for PR Merge:

1. Unit & integration tests updated
2. Data validation completed (examples listed below)
   1. Does this work well with the current frontend? Or is the frontend aware of a needed change?
   2. Is performance impacted in the changes (e.g., API, pipeline, downloads, etc.)?
   3. Is the expected data returned with the expected format?
3. Jira Ticket(s)
   1. DEV-12528

Explain N/A in above checklist:

2. API documentation updated (examples listed below)
   No API contracts need to be updated for this change.

3. Appropriate Operations ticket(s) created
   No operation tickets are needed for this change.

[sethstoudenmier]

Approving, but left some comments are possible cleanup / improvements.