Config file ENV overrides are unreliable

[alastairtree]

The sharepoint upload config value had to be hardcoded because the ENV override is not working. Think due to the level of nesting > 2. This is a bad system. Need to get it sorted long term - config + ENVs should NOT BE THIS HARD.

[alastairtree]

This commit was the HACK 194242d

[mfacchinelli]

This issue is caused by the fact that the override variable should be called MAG_UPLOAD_ROOT_PATH, but we have defined underscores _ to be the delimiter for nested env var properties.

This means that Prefect iterprets MAG_UPLOAD_ROOT_PATH as overwriting a property called settings.upload.root.path, instead of settings.upload.root_path.

The least intrusive way to fix this is to use an alternative way of overwriting properties, which is:

MAG_UPLOAD='{"root_path": "my/path"}'

See https://github.com/ImperialCollegeLondon/imap-pipeline-core/tree/fix/upload-config for some WIP.