Adds partial Delta Lake support

[erp12]

Adds support for Delta Lake storage format. This PR is best reviewed 1 commit at a time.

Currently, some core functionality of Delta is disabled due to a known incompatibility between Spark 3.1 and Delta 0.7+. Once these issues are addressed in the next Delta release, the commented out code in this PR should be uncommented. All commented out code has been tested on Spark 3.0.2 + Delta 0.8.

I chose to implement the Delta API as using with-dynamic-import, but I am not confident in this decision and would love feedback. My editor fails to recognize any symbol defined within with-dynamic-import which hinders navigation and refactoring.

Some things I would like to do before calling this PR done:

• Add docstrings
• Get feedback on the data-oriented interface for merge strategies. Corresponding Scala documentation here.

[dakra]

@erp12 Can you update the code for delta 1.0.0.

I use this current PR sporadically for ad-hoc analysis and works well. Thanks for it :)

[dakra]

Just want to mention that there is now already delta 1.1 which supports Spark 3.2.

[dakra]

@erp12 @anthony-khong For delta there is now version 2.1.1
which supports Spark version 3.3

It's obviously difficult do keep everything up-to-date and
compatible with each other. Especially when even minor version
increases apparently break things.
ftw, at least delta and AWS EMR support Spark 3.3 which was
released June.

Are there plans to upgrade the geni spark version to 3.3?
And if that's done, we could consider merging this PR as well.
I'm using it pretty regularly for personal queries/debugging
things and works pretty well.

Thanks,
Daniel