PR210: lifetime mgnt of the cache dir

[rasmunk]

In PR #210 I have question about how you establish the cache_dir beyond provisioning it ahead of time during testing, or manually via the prime_templates call in mig/lib/templates/__main__.

That is, if the directory is removed or not established ahead of time, how does the TemplateStore ensure that the directory is present and available when the service is running.

Maybe during the initial construction before it is added to the configuration context should it ensure that it is created. So that we don't hit the disk on every render

It is created ahead of time during "provisioning" exactly to try to minimise prodding the filesystem :) Since MiG does not have a long running server process that handles requests, and instead is arranged to process every request as a new process (even WSGI requests unfortunately), directory creation at construction time is the equivalent of doing it for every request since a TemplateStore will be constructed for every request.

At present, the code treats the presence of the directory as an invariant. One cannot bring the system up, either in development or in docker, without the template cache directory having been created. If it were to be absent, an error is returned gracefully.

That said, I don't quite follow the concern about the directory being removed. How would that happen unless someone quite manually messed with the contents of the containers? If one were able to do that I think then much greater damage is possible like editing the code itself so I don't think it much of a concern.

[rasmunk]

Removal is not the big concern either for me, it would just be for resilience of the service in the long run. It is more who responsible for establishing it and at what point. If it indeed is meant to be "handled" beforehand, it would be good to have at least a procedure that handles all prerequisites for the cache_dir itself and all the required sub-directories that can be called either at build or service launch.

Removal is not the big concern either for me, it would just be for resilience of the service in the long run. It is more who responsible for establishing it and at what point. If it indeed is meant to be "handled" beforehand, it would be good to have at least a procedure that handles all prerequisites for the cache_dir itself and all the required sub-directories that can be called either at build or service launch.

Got it. Though do we not already have the procedure you are asking for in the priming procedure? It's a CLI, invoked by both the local dev environment and by the docker builds to create the cache directory and get the templates compiled to bytecode. Calling it at service start time would be trivial also. Would you like to see something more or different than that?

[rasmunk]

Also to clarify/verify, if we added a new template at runtime, we would need to manually create its own directory inside the cache_dir correct? via a call such as python3 -m mig.lib.templates prime after adding it to the configuration.

Also to clarify/verify, if we added a new template at runtime, we would need to manually create its own directory inside the cache_dir correct?

Not manually by hand, but it does need to have been triggered. Those directories are created based on the template packages that are enabled in the configuration. They are created as part of the priming process (think of this as precompilation of he templates - see the templates __main__.py and its invocation such as https://github.com/ucphhpc/migrid-sync/pull/453/changes#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R107-R109). If templates are changed then yes, the priming step needs to be invoked. The reason this step will be unlikely to need doing by hand is that it is done by the dev environment and is wired into the build :)

Though I think I see some of the confusion - the separate template support PR does not have this priming step in the Makefile. The reason for this was the original separate templates PR was only the infrastructure, and no enablement of any templates, so there is nothing to prime.

You can see the prime commad being exercised by the test suite here: https://github.com/ucphhpc/migrid-sync/pull/210/changes#diff-99d901d9150b7d1e4fd7b0d8e7631213a62ba59b3f0914deabc68654b5dc34b6R190

And for completeness, the same prime is executed by docker-migrid: https://github.com/ucphhpc/docker-migrid/pull/151/changes#diff-abe2175713c20d7ae9501a46c063ed27afc856e861dbb7ca012e1af03c5e3cecR1345-R1348

Re-edited some of my answers as I realised we had meant slightly different things by "manually". The process does need to be explicitly triggered, but we've added triggering to almost all the requisite places including the build.

[rasmunk]

Ahh okay, I didn't see any invocation of mig.lib.templates in the PR, so I was wondering where if at all this was taking place to establish the required cache_dir. But alright since it is in the docker-migrid PR and in the parent PR of this that makes more sense. Will just note on the docker-migrid PR that this is required before it can be merge.

Cool :)

Fwiw, we could fold in those line to call priming from the local Makefile into this PR if you feel it improves the clarity. It won't conflict and won't actually do much but won't hurt either.

[rasmunk]

The concern about runtime, is mostly whether we have to invoke the priming as part of a hotfix patch if we have to apply and distribute such a fix that involves any of the templates. But that will properly be down to circumstances.

[rasmunk]

Cool :)

Fwiw, we could fold in those line to call priming from the local Makefile into this PR if you feel it improves the clarity. It won't conflict and won't actually do much but won't hurt either.

I think it would be good to get the priming into this PR, since at least I consider it a crucial part of introducing it.

Cool :)
Fwiw, we could fold in those line to call priming from the local Makefile into this PR if you feel it improves the clarity. It won't conflict and won't actually do much but won't hurt either.

I think it would be good to get the priming into this PR, since at least I consider it a crucial part of introducing it.

Done.