fix: improve pod label patching with error handling and retry mechanism

[drivebyer]

• Added JSON pointer escaping for label keys
• Implemented retry mechanism for label patching
• Enhanced error handling and logging
• Added validation for empty labels
• Created utility function for JSON pointer escaping in util package

Description

#1217
Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• Tests have been added/modified and all tests pass.
• Functionality/bugs have been confirmed to be unchanged or fixed.
• I have performed a self-review of my own code.
• Documentation has been updated or added where necessary.

Additional Context

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 32 lines in your changes missing coverage. Please review.

Project coverage is 28.83%. Comparing base (60ef5a2) to head (69851dd).
Report is 41 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/k8sutils/pod.go	0.00%	30 Missing ⚠️
pkg/util/strings.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1231      +/-   ##
==========================================
- Coverage   30.20%   28.83%   -1.37%     
==========================================
  Files          55       58       +3     
  Lines        6294     6700     +406     
==========================================
+ Hits         1901     1932      +31     
- Misses       4200     4609     +409     
+ Partials      193      159      -34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[trynocoding]

@drivebyer Hello, has this patch fixed the #1217?
I tried to fail the master node, the sentinel has been failover, but the label of the pod on the new master node is not being updated.

[root@master redis]# kubectl get po --show-labels
NAME                        READY   STATUS    RESTARTS   AGE    LABELS
redis-replication-0         0/2     Pending   0          1s     app=redis-replication,controller-revision-hash=redis-replication-66cccc9c9c,redis_setup_type=replication,role=replication,statefulset.kubernetes.io/pod-name=redis-replication-0
redis-replication-1         2/2     Running   0          102m   app=redis-replication,controller-revision-hash=redis-replication-66cccc9c9c,redis-role=slave,redis_setup_type=replication,role=replication,statefulset.kubernetes.io/pod-name=redis-replication-1
redis-replication-2         2/2     Running   0          101m   app=redis-replication,controller-revision-hash=redis-replication-66cccc9c9c,redis-role=slave,redis_setup_type=replication,role=replication,statefulset.kubernetes.io/pod-name=redis-replication-2
redis-sentinel-sentinel-0   1/1     Running   0          97m    app=redis-sentinel-sentinel,controller-revision-hash=redis-sentinel-sentinel-6944bf47f6,redis_setup_type=sentinel,role=sentinel,statefulset.kubernetes.io/pod-name=redis-sentinel-sentinel-0
redis-sentinel-sentinel-1   1/1     Running   0          96m    app=redis-sentinel-sentinel,controller-revision-hash=redis-sentinel-sentinel-6944bf47f6,redis_setup_type=sentinel,role=sentinel,statefulset.kubernetes.io/pod-name=redis-sentinel-sentinel-1
redis-sentinel-sentinel-2   1/1     Running   0          95m    app=redis-sentinel-sentinel,controller-revision-hash=redis-sentinel-sentinel-6944bf47f6,redis_setup_type=sentinel,role=sentinel,statefulset.kubernetes.io/pod-name=redis-sentinel-sentinel-2
[root@master ~]# 
[root@master ~]# kubectl exec -it redis-replication-1 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulted container "redis-replication" out of: redis-replication, redis-exporter
redis-replication-1:/data$ redis-cli -p 6379 info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.1.246,port=6379,state=online,offset=1146098,lag=0
master_failover_state:no-failover
master_replid:fca285fdb30572800c1377bc731bb19720eb0382
master_replid2:bbf34d5e1c5cb30bd7874a0a87fe910edfed58de
master_repl_offset:1146098
second_repl_offset:37532
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:81761
repl_backlog_histlen:1064338
redis-replication-1:/data$ exit
exit
[root@master ~]# kubectl exec -it redis-replication-2 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulted container "redis-replication" out of: redis-replication, redis-exporter
^[[Aredis-replication-2:/daredis-cli -p 6379 info replicationtion
# Replication
role:slave
master_host:10.0.2.16
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:1148919
slave_repl_offset:1148919
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:fca285fdb30572800c1377bc731bb19720eb0382
master_replid2:bbf34d5e1c5cb30bd7874a0a87fe910edfed58de
master_repl_offset:1148919
second_repl_offset:37532
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:81761
repl_backlog_histlen:1067159
redis-replication-2:/data$ 

operator log

{"level":"info","ts":"2025-02-05T16:29:24+08:00","msg":"StatefulSet not ready yet, requeuing","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-replication","namespace":"default"},"namespace":"default","name":"redis-replication","reconcileID":"e71400cc-7e76-4750-b0ed-93c8bddd8086","namespace":"default","name":"redis-replication"}
{"level":"info","ts":"2025-02-05T16:29:25+08:00","msg":"StatefulSet not ready yet, requeuing","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-replication","namespace":"default"},"namespace":"default","name":"redis-replication","reconcileID":"cec8bfd8-0c6b-4985-9fbe-7cfcf770541d","namespace":"default","name":"redis-replication"}
{"level":"info","ts":"2025-02-05T16:29:26+08:00","msg":"StatefulSet not ready yet, requeuing","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-replication","namespace":"default"},"namespace":"default","name":"redis-replication","reconcileID":"816fd418-7140-4a5a-9191-011184275ebe","namespace":"default","name":"redis-replication"}
{"level":"info","ts":"2025-02-05T16:29:27+08:00","msg":"StatefulSet not ready yet, requeuing","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-replication","namespace":"default"},"namespace":"default","name":"redis-replication","reconcileID":"7c9bd3d0-6ef0-46b9-9aa5-535131e8f3c8","namespace":"default","name":"redis-replication"}

[drivebyer]

@trynocoding Thanks for your report :) This will be fixed by the next release.