Add preprocessor hook

[drossy]

We're investigating how we can manage our secrets external to quattor/aquilon, in something like Vault (https://www.vaultproject.io/). Is it feasible to add a preprocessor hook to ncm-ncd? Similar to the pre-hook but with the ability to update the profile (which we'd use to replace placeholder credentials in the profile with the correct ones from our secret management tool), which would then be used by all components?

[jrha]

ncm-ncd already has support for pre-hooks, but if they need to update the profile, then I suspect you may instead need support in CCM.

[jrha]

Alternatively, if the secret management tool knows what files to update then it could be triggered by the existing post_hook

[stdweird]

if you want to manipulate the profile, you can do it in pan (but them compilation hosts needs access to all secrets), ccm (when parsing the profile, it could inject data) or ncm-ncd; but none of these have anything easy builtin to do so. however, if you know some perl, it shouldn't be too hard to hack it in. to have a clean supported solution is probably more work, starting with proper design 😉

[drossy]

Alternatively, if the secret management tool knows what files to update then it could be triggered by the existing post_hook

But that has the issue that more components will always be running, even when the total change is zero. And won't it add a small potential issue of the wrong credentials being in place temporarily until the post-hook runs?

[ned21]

If you are going to modify the profile then I would think CCM is the right place to do it, some sort of post-processing after ccm-fetch. This means your secrets are all available in /var/lib/ccm but then if you are writing them to config files on disk then root can read them anyway, and it also removes the dependency on Vault being available every time ncm-ncd runs: it only has to be up during the ccm-fetch.

[jrha]

This feels similar to the functionality we removed back in 2016 in quattor/CCM#145

[gombasg]

I don't know the all the details, but the request feels wrong. You should not mix secrets with configuration. Secrets should always be stored in distinct files, and the configuration deployed by Quattor should only reference the location of those files. The exact implementation may depend on what kind of software are you configuring. Security conscious software should allow (or outright require) storing secrets in stand-alone files. In other cases, you might want to use some "include-like" functionality to ensure the configuration fragment containing the secret is distinct from the configuration shipped by Quattor.

Also, you do not want to require forcing a profile update every time you refresh a password. We've tried that, it is awful, don't go there. Changing/refreshing a password/certificate/other secret must not require ncm-ncd to run.

[drossy]

I'd agree but secrets are already mixed with configuration in quattor: https://github.com/quattor/template-library-core/search?q=password

[drossy]

I'd agree but secrets are already mixed with configuration in quattor: https://github.com/quattor/template-library-core/search?q=password

Bah, apologies, I feel that was too snarky, let me try again. I think that secrets are configuration items, though ones that need special handling. If I want to configure a service that access a remote database it normally needs 4 things, hostname, database name, username and password - without any of those items configuration of the service will fail. But at least 1 of those you will not want to be public information.
I do agree that ideally secrets would not sit in the same file as non-secret configuration items, but I've encountered many pieces of software where it does, so the secret management solution has to be able to cope with it.

So secret handling is an aspect of configuration management; I'll take your point that avoiding ncm-ncd runs for credential refresh is preferable, but if the other configuration items are changing there's a non-zero chance the secrets are also changing, so it would surely be desirable to initiate a refresh of the the secrets at the same time? Even if there is a second workflow that periodically checks only for changes to the credentials. Or perhaps its driven the other way round, the secret manager initiates a ncm-ncd run when the secrets have changed? Either way I feel that the two are linked even if loosely, and I'm fairly sure there will be many services that will not pick up changed credentials until restarted/reloaded/signalled so the secret manager can't just replace a file on disk with another and must have some way to "kick" the service, which crosses over with ncm-ncd functionality.

My feeling is that there are 3 routes to handling secrets in quattor: one effectively entails an audit of all the templates to update them to remove embedded secrets or extend their secret handling in some fashion TBD, with attendant changes to the components. Another is abandoning the components when secret and non-secret items sit alongside each other and filecopying in a template of the config file and merging in the secrets, as part of the restart section of filecopy. The first seems like the "right" solution but is also a lot of work, so wouldn't deliver soon. The second, to me, reduces the value of quattor as a configuration management system - why have a large template library if good practice is to not use certain parts of it. And finally there's what I think of as the pragmatic solution, blending the configuration items from quattor with secrets from another source, to hand a complete configuration to the existing unchanged components, which is what the original issue was aiming at.