:bug: [BUG]: Load order of `EXTRA_CONF_DIRS` not respected by `COMPOSE_CONF_LIST`

[fmigneault]

Summary

Load order of EXTRA_CONF_DIRS is not respected by COMPOSE_CONF_LIST to apply items defined in a critical order.

The EXTRA_CONF_DIRS should be considered as user-critical definitions to be respected at all costs, and therefore, loaded as much as possible toward the end to preserve its overrides.

Details

In my env.local, I define the following:
(note the last daccs-env which contains custom docker-compose overrides, defined outside birdhouse-deploy)

export EXTRA_CONF_DIRS="
    ./components/monitoring
    ./optional-components/canarie-api-full-monitoring
    ./optional-components/wps-healthchecks
    ./optional-components/secure-thredds
    ./optional-components/testthredds
    ./optional-components/test-weaver
    ./optional-components/secure-data-proxy
    ./components/weaver
    ./components/cowbird
    ../../daccs-env
"

Following are the exact load order reported by COMPOSE_CONF_LIST (only reformatted line-by-line for readability)
when running pavics-compose up -d

note: using a diff only to highlight relevant items, not an actual diff
($\color{green}{green}$: items matching above extras, $\color{red}{red}$: items of interest)

docker-compose.yml
./config/proxy/docker-compose-extra.yml
./config/phoenix/docker-compose-extra.yml
./config/geoserver/docker-compose-extra.yml
./config/flyingpigeon/docker-compose-extra.yml
./config/finch/docker-compose-extra.yml
./config/raven/docker-compose-extra.yml
./config/hummingbird/docker-compose-extra.yml
./config/thredds/docker-compose-extra.yml
./config/portainer/docker-compose-extra.yml
./config/magpie/docker-compose-extra.yml
./config/twitcher/docker-compose-extra.yml
./config/jupyterhub/docker-compose-extra.yml
./config/frontend/docker-compose-extra.yml
./config/project-api/docker-compose-extra.yml
./config/catalog/docker-compose-extra.yml
./config/malleefowl/docker-compose-extra.yml
./config/solr/docker-compose-extra.yml
+ ./components/monitoring/docker-compose-extra.yml
+ ./optional-components/testthredds/docker-compose-extra.yml
+ ./optional-components/test-weaver/docker-compose-extra.yml
+ ./components/weaver/docker-compose-extra.yml
+ ./components/cowbird/docker-compose-extra.yml
+ ../../daccs-env/docker-compose-extra.yml
./config/mongodb/docker-compose-extra.yml
./config/postgres/docker-compose-extra.yml
./config/wps_outputs-volume/docker-compose-extra.yml
- ./config/data-volume/docker-compose-extra.yml  # loaded after custom env!
./config/ncwms2/docker-compose-extra.yml
./config/phoenix/config/proxy/docker-compose-extra.yml
./config/geoserver/config/magpie/docker-compose-extra.yml
./config/geoserver/config/proxy/docker-compose-extra.yml
./config/flyingpigeon/config/magpie/docker-compose-extra.yml
./config/flyingpigeon/config/wps_outputs-volume/docker-compose-extra.yml
./config/finch/config/magpie/docker-compose-extra.yml
./config/finch/config/proxy/docker-compose-extra.yml
./config/finch/config/wps_outputs-volume/docker-compose-extra.yml
./config/raven/config/magpie/docker-compose-extra.yml
./config/raven/config/proxy/docker-compose-extra.yml
./config/raven/config/wps_outputs-volume/docker-compose-extra.yml
./config/hummingbird/config/data-volume/docker-compose-extra.yml
./config/hummingbird/config/magpie/docker-compose-extra.yml
./config/hummingbird/config/wps_outputs-volume/docker-compose-extra.yml
./config/thredds/config/proxy/docker-compose-extra.yml
./config/portainer/config/proxy/docker-compose-extra.yml
./config/magpie/config/proxy/docker-compose-extra.yml
./config/twitcher/config/proxy/docker-compose-extra.yml
./config/jupyterhub/config/magpie/docker-compose-extra.yml
./config/jupyterhub/config/proxy/docker-compose-extra.yml
./config/frontend/config/proxy/docker-compose-extra.yml
./config/project-api/config/proxy/docker-compose-extra.yml
./config/catalog/config/magpie/docker-compose-extra.yml
./config/catalog/config/proxy/docker-compose-extra.yml
./config/malleefowl/config/data-volume/docker-compose-extra.yml
./config/malleefowl/config/magpie/docker-compose-extra.yml
./config/malleefowl/config/proxy/docker-compose-extra.yml
./config/malleefowl/config/wps_outputs-volume/docker-compose-extra.yml
./config/solr/config/proxy/docker-compose-extra.yml
+ ./optional-components/wps-healthchecks/config/catalog/docker-compose-extra.yml
+ ./optional-components/wps-healthchecks/config/finch/docker-compose-extra.yml
+ ./optional-components/wps-healthchecks/config/flyingpigeon/docker-compose-extra.yml
+ ./optional-components/wps-healthchecks/config/hummingbird/docker-compose-extra.yml
+ ./optional-components/wps-healthchecks/config/malleefowl/docker-compose-extra.yml
+ ./optional-components/wps-healthchecks/config/raven/docker-compose-extra.yml
+ ./optional-components/secure-thredds/config/magpie/docker-compose-extra.yml
+ ./optional-components/testthredds/config/proxy/docker-compose-extra.yml
+ ./optional-components/secure-data-proxy/config/magpie/docker-compose-extra.yml
+ ./optional-components/secure-data-proxy/config/proxy/docker-compose-extra.yml
+ ./components/weaver/config/magpie/docker-compose-extra.yml
+ ./components/weaver/config/proxy/docker-compose-extra.yml
+ ./components/weaver/config/twitcher/docker-compose-extra.yml
+ ./components/cowbird/config/geoserver/docker-compose-extra.yml
+ ./components/cowbird/config/jupyterhub/docker-compose-extra.yml
+ ./components/cowbird/config/magpie/docker-compose-extra.yml
+ ./components/cowbird/config/proxy/docker-compose-extra.yml
./config/canarie-api/config/proxy/docker-compose-extra.yml
./config/wps_outputs-volume/config/proxy/docker-compose-extra.yml
./config/ncwms2/config/magpie/docker-compose-extra.yml
./config/ncwms2/config/proxy/docker-compose-extra.yml
./config/ncwms2/config/wps_outputs-volume/docker-compose-extra.yml

As displayed above, load order seems somewhat random. The docker-compose from weaver, cowbird, etc. are loaded high up, and remaining configs from the same components are then loaded almost at the end. Meanwhile, multiple other components are loaded in between.

In this specific case, the last ../../daccs-env from EXTRA_CONF_DIRS is explicitly required at the end.
Amongst other things, it overrides the volumes.data: {} definition with an actual location on the machine.
Because ./config/data-volume/docker-compose-extra.yml is loaded after ../../daccs-env/docker-compose-extra.yml, it "overrides-back" my original override.
I end up getting the following error due to mismatching definitions with the existing volume:

ERROR: Configuration for volume data specifies "o" driver_opt None, but a volume with the same name uses a different "o" driver_opt (bind). If you wish to use the new configuration, please remove the existing volume "birdhouse_data" first:
$ docker volume rm birdhouse_data

Furthermore, the ./optional-components/canarie-api-full-monitoring item from EXTRA_CONF_DIRS is missing entirely.
Not quite sure why, but this is breaking #284 and the E2E that depends on it.

To Reproduce

Steps to reproduce the behaviour:

See description.
Load order hard to predict.

Environment

Information	Value
Server/Platform URL	n/a
Version Tag/Commit	https://github.com/bird-house/birdhouse-deploy/tree/1.24.0
Related issues/PR	#284
Related components	pavics-compose
Custom configuration	see description

Concerned Organizations

All

[mishaschwartz]

@fmigneault I've put up a PR with a fix for the load order issue.

I am unable to replicate the issue regarding the missing ./optional-components/canarie-api-full-monitoring/docker-compose-extra.yml file. That file loads fine when I test it out on my local test environment. I'm using the same EXTRA_CONF_DIRS variable that you have above and the code is currently on the master branch. If that continues to be a problem my guess is that it is unrelated to the load order issue.

The load order is not random, the docker-compose-extra.yml file defined at the root of each component's directory is loaded first. Once all of these are loaded, any overrides that are defined in config/*/docker-compose-extra.yml are loaded.

[tlvu]

My bad, I did not see this loader order problem in my previous review of #296

For context, the load order is important by design because it allows for the override capabilities describe here

birdhouse-deploy/birdhouse/env.local.example

Lines 61 to 73 in 37fe32e

	# Last dir/component in the EXTRA_CONF_DIRS list have highest override
	# precedence, example:
	#
	# * Last docker-compose volume mount to same destination win over the
	# previous docker-compose volume mount.
	#
	# * Last default.env can change the values of all previous default.env.
	#
	# * Last pre/post docker-compose-up script can potentially undo actions
	# from previous scripts.
	#
	# Suggested to keep the private-config-repo last in the list to be able to
	# override anything.

Each org private config, which is suggested to be the last component in the list, should be able to override all previous docker-compose fragments, all previous default.env values and all previous pre/post docker-compose-up scripts.

This component override capabilities, together with env.local, are basically the customization feature allowing each org to change any default behaviors.