[RFE] `locksmith`: integration with `FleetLock`

[tormath1]

Current situation

locksmith is highly bound to etcd: users who want to have a cluster reboot coordination needs to use etcd. The idea is to implement a FleetLock client into locksmith.

Impact

Flatcar users will benefit from FleetLock integration:

• more backend flexibility to orchestrate rebooting of instances inside a cluster
• remove etcd dependency from locksmith

Ideal future situation

User will configure locksmith via a HTTP FleetLock endpoint URL.

Implementation options

• FleetLock:

• develop a Go FleetLock HTTP client (https://github.com/flatcar-linux/fleetlock)

• Locksmith:

• use the Go FleetLock HTTP client instead of the etcd lock client (started in [WIP] Integrate Fleetlock client with Locksmith locksmith#14)

• Flatcar:

• ship airlock in the OS + systemd configuration ready to run a local airlock instance
• documentation: explain how users can use locksmith + airlock + etcd to emulate the current behavior + other use cases like using an external FleetLock endpoint
• upgrade locksmith into the OS

• Kola:

• implement more tests to cover actual use cases (avoid regression)
• implement a custom testing FleetLock server to create scenarios (network latency, etc.) used by the tested locksmith
• implement test with new features offered by FleetLock

Additional information

• FleetLock protocol: https://coreos.github.io/zincati/development/fleetlock/protocol/
• Locksmith: https://github.com/kinvolk/locksmith
• Airlock: https://github.com/coreos/airlock

[pothos]

About migration to etcd3: We might keep etcd2 support around in locksmith for some time besides FleetLock, or we could directly delete it and make FleetLock the new default together with an inbuilt airlock layer (part of locksmith instead of a separate binary) where we point the inbuilt Airlock layer to the configured locksmith etcd server. This will then use the etcd3 protocol. Update behavior should be ok even though etcd keeps the v2 and v3 state separate: The old clients coordinate themselves in the v2 state and the new clients have just updated and don't need to coordinate themselves for the next hours in which all old clients of the same pool should have updated.

One more idea:

• Run a public Airlock server that users can rely on for simple coordination by using secret group names for their clusters.