VED-770-e2e-batch-rewrite

[nhsdevws]

Summary

• Routine Change
• ❗ Breaking Change
• 🤖 Operational or Infrastructure Change
• ✨ New Feature
• ⚠️ Potential issues that might be caused by this change

• Tests described by json
• Tests use different ods/Vax combination for FIFO parallel processing
• Uploaded in together before poll
• Revised validations
  = Complete cleanup of source/dest s3, audit/events table, SQS FIFOs (only purged for dev PRs)

Reviews Required

• Dev
• Test
• Tech Author
• Product Owner

Review Checklist

ℹ️ This section is to be filled in by the reviewer.

• I have reviewed the changes in this PR and they fill all or part of the acceptance criteria of the ticket, and the code is in a mergeable state.
• If there were infrastructure, operational, or build changes, I have made sure there is sufficient evidence that the changes will work.
• I have ensured the changelog has been updated by the submitter, if necessary.

[github-actions]

This branch is working on a ticket in the NHS England VED JIRA Project. Here's a handy link to the ticket:

VED-770

[dlzhry2nhs]

Nice work - looks like it's running quickly in the pipeline and it'll be great to get this back in as a CI check.

Couple of comments added mostly around readability and expressing intent.

We can also have tickets down the line to add further test cases of critical functionality e.g. invalid files, duplicates, and richer assertions on the persisted data. Happy for that to be later rather than now - this PR gets us to a better position.

Only reservation relates to the readability - while it is now performing, not convinced this is much more readable or extensible than what we had before. We can see when we attempt to add the additional required cases. If necessary could spike moving to pytest parallel - it has always been my preference regardless of whether it is integration or unit tests, for test readability i.e. clear setup, act and assert steps to be used. (Details of what we are doing in the steps can be abstracted into code but in this suite it seems like everything is tucked away into the new test classes away from the original e2e_batch test file.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[JamesW1-NHS]

just a minor point: if we do timeout, should we not log something? (e.g. the fact we timed out, and/or some information on the unprocessed files)