Fix/incorrect processing

[eveleighoj]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

Started out as a bug fix but became a lot bigger. Included refactoring of code to simplify the code base, bug fixes to stop processing incorrectly.

Refactoring:

• create a pipeline.py file to separate separate Issue and entity pipelines out. they are almost completely separate processes that don't use the same inputs.
• removed for loop as it caused confusion, this is obvious from the use of local and global variables being called.
• altered input to move away assuming where all data was stored. I think more can be done on this but for now separating the two buckets was good enough as all data comes from one of the data buckets. This also massively helps with local testing which was almost non-existent for the actual spark functions.
• created an entryPoints directory, this will help with making the functionality expandable in the future as we will want all entryPoints to be pushed up
• TODO: database location should be fed in as an input

Bug fixes

• point column was being completely removed even when point values existed. this cause errors in datasets when there was no geometry data expected, e.g. for a lot of tree data and transport access nodes

Optimisation:

• for some reason the entity transformer was being ran multiple times. I believe this is generally because of confusion of the for loop which has been refactored

Related Tickets & Documents

• Ticket Link

digital-land/config#1728

QA Instructions, Screenshots, Recordings

Please replace this line with instructions on how to test your changes, a note
on the devices and browsers this has been tested on, as well as any relevant
images for UI changes.

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above. Please refer to the Digital Land Testing Guidance for more information.

• x] Yes
• No, and this is why: please replace this line with details on why tests
  have not been included
• I need help with writing tests

[optional] Are there any post deployment tasks we need to perform?

[optional] Are there any dependencies on other PRs or Work?