Working Add Data - Integrate assign entities

[pooleycodes]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

Somewhat still a work in progress, but in the sprint to get to config manager version 1 going the async add data needed a rewrite.

• The core implementations/changes here are the use of pipeline transform and assign entities function such to correctly check for found entities and assign new lookups for unrecognised transform rows.
• Some refactoring such as moving source csv and endpoint csv creation to a utils and created in the workflow.py. The pipeline.py file should only be concerned with all the steps to actually run the pipeline.

Related Tickets & Documents

• Ticket Link Make Config Manager work for simple datasets config#1902

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above. Please refer to the Digital Land Testing Guidance for more information.

• Yes
• No, and this is why: Tests do exist for all of these services, more thorough tests of the add data task is definitely needed.
• I need help with writing tests

[optional] Are there any dependencies on other PRs or Work?

1. More of an investigation is needed why the check tool has assign_entries() function, is it needed?
2. Config Manager tests are needed, especially integration tests, some publish examples have been created that would be good to use.