Discussion: optimistic lock conflicts with Redis backend

[wzihao81-ctrl]

Description

Hi, thanks for go-ipam!

When running go-ipam as a gRPC microservice with multiple instances + Redis, we see frequent optimistic lock conflicts under concurrent IP allocations/releases (~30 QPS, multiple namespaces). This leads to retries and higher latency.

From the code/behavior it seems each operation:

Uses an optimistic lock on a namespace key in Redis
Reads namespace state → modifies in memory → writes back
This works well for read-heavy loads, but becomes a bottleneck with steady concurrent writes.

For a simplified use case (mainly prefix + IP allocation/release), could we support an alternative Redis backend that:

For each {namespace, prefix}, keeps all allocatable IPs in a Redis SET
Allocation: SPOP (or SRANDMEMBER + SREM)
Release: SADD
This would rely on Redis atomic SET operations instead of namespace-level optimistic locking, potentially reducing conflicts in write-heavy scenarios.

Questions:

Would such a “SET-based” mode/backend fit the project direction (even as an optional/experimental feature)?

[majst01]

Description

Hi, thanks for go-ipam!

When running go-ipam as a gRPC microservice with multiple instances + Redis, we see frequent optimistic lock conflicts under concurrent IP allocations/releases (~30 QPS, multiple namespaces). This leads to retries and higher latency.

From the code/behavior it seems each operation:

Uses an optimistic lock on a namespace key in Redis Reads namespace state → modifies in memory → writes back This works well for read-heavy loads, but becomes a bottleneck with steady concurrent writes.

For a simplified use case (mainly prefix + IP allocation/release), could we support an alternative Redis backend that:

For each {namespace, prefix}, keeps all allocatable IPs in a Redis SET Allocation: SPOP (or SRANDMEMBER + SREM) Release: SADD This would rely on Redis atomic SET operations instead of namespace-level optimistic locking, potentially reducing conflicts in write-heavy scenarios.

Questions:

Would such a “SET-based” mode/backend fit the project direction (even as an optional/experimental feature)?

Hi,

Nice to see you are using it quite heavy. I was not the redis backend author, nor do we use this backend in our production landscape (postgres is our prefered solution). If i understand your suggestion correct, this new approach would mean you have to migrate existing redis backends to the new way of storing the prefixes.

I am always happy to see improvements to the codebase, and if the migration is also covered with the new implementation, im totally fine with that.

[wzihao81-ctrl]

Description

Hi, thanks for go-ipam!
When running go-ipam as a gRPC microservice with multiple instances + Redis, we see frequent optimistic lock conflicts under concurrent IP allocations/releases (~30 QPS, multiple namespaces). This leads to retries and higher latency.
From the code/behavior it seems each operation:
Uses an optimistic lock on a namespace key in Redis Reads namespace state → modifies in memory → writes back This works well for read-heavy loads, but becomes a bottleneck with steady concurrent writes.
For a simplified use case (mainly prefix + IP allocation/release), could we support an alternative Redis backend that:
For each {namespace, prefix}, keeps all allocatable IPs in a Redis SET Allocation: SPOP (or SRANDMEMBER + SREM) Release: SADD This would rely on Redis atomic SET operations instead of namespace-level optimistic locking, potentially reducing conflicts in write-heavy scenarios.
Questions:
Would such a “SET-based” mode/backend fit the project direction (even as an optional/experimental feature)?

Hi,

Nice to see you are using it quite heavy. I was not the redis backend author, nor do we use this backend in our production landscape (postgres is our prefered solution). If i understand your suggestion correct, this new approach would mean you have to migrate existing redis backends to the new way of storing the prefixes.

I am always happy to see improvements to the codebase, and if the migration is also covered with the new implementation, im totally fine with that.

Yes, that’s correct. In my own project I’m already using a simple Python script to handle the migration:

We first stop all traffic to the ipam service
The script reads all existing prefixes and their allocation state from Redis
Then it converts them into the new SET‑based layout (one SET per {namespace, prefix})
So in practice, the migration has been working fine for us in a controlled maintenance window.

Would it be acceptable if I first keep this SET‑based Redis backend in my own repository (including the migration script), and then open a PR to go-ipam once the approach is stable and better documented?

[majst01]

The Migration should happen with go-ipam and not force users to call some external python scripts.