[BUG] Handler 'on_create' failed permanently: RabbitMQ backup daemon is not ready

[PhBouzid]

Describe the bug
Floating bug. The error is periodically reproduced on different clouds.

while deploy, sometimes error arise:
{code}
Resource [rabbitmq-service] has no state found by path [$.status.conditions[?(@.type=='Successful')].status]
$.status.conditions[?(@.type=='Failed')] fields content:
error: ,
message: RabbitMQ backup daemon is not ready,
reason:
Contact resource owner to investigate the reason it has no status field. Also you can try to increase polling time with parameter
{code}
Operator log contains following messages:
{code}
[2024-10-28 14:17:44,267] kopf_script_0/op [INFO ] RabbitMQ pods are ready
[2024-10-28 14:17:44,726] kopf_script_0/op [INFO ] Executed command in pod successfully.
[2024-10-28 14:17:44,728] kopf_script_0/op [INFO ] Feature flags are enabled successfully
[2024-10-28 14:17:50,681] kopf.objects [ERROR ] [rabbitmq/rabbitmq-service] Handler 'on_create' failed permanently: RabbitMQ backup daemon is not ready.
[2024-10-28 14:17:50,683] kopf.objects [INFO ] [rabbitmq/rabbitmq-service] Creation is processed: 0 succeeded; 1 failed.
{code}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment:

• Application Version:
• K8S Version:

Additional context
Add any other context about the problem here.

[OleksandrIaresko]

Need to improve the current backup test for all SaQs services: consul, rabbitmq, kafka, zookeapear, opensearch

[andrewluckyguy]

@PavelYadrov, loglevel: INFO has been added in deployment parameters for the test pipeline:
https://github.com/Netcracker/qubership-rabbitmq/actions/runs/19675898995
Re-run it several times if the issue isn't reproduced

[andrewluckyguy]

Changed to loglevel: DEBUG
https://github.com/Netcracker/qubership-rabbitmq/actions/runs/19675898995/job/56439852087

[PavelYadrov]

Hi, i'm unable to reproduce this issue
https://github.com/Netcracker/qubership-rabbitmq/actions/runs/19675898995

Not sure what we should do, lest continue working on this issue once problem is reproduced

[PavelYadrov]

I finally get why this is not working, following is the detailed explanation why it fails.

Reason - we get "Warning" state from "/health" endpoint, here it is - https://github.com/Netcracker/qubership-backup-daemon/blob/ec400e76e54e267a596adb6b773276bd4e065c67/src/httpapi.py#L137

So, here's WHY we fail, take a look at this piece of code

            status = False  # False = "Warning", True = "UP"
            last = last_successful = None

            # calculate last successful
            for vault in vaults:
                if not vault.is_failed():
                    last_successful = vault.to_json()
                    status = True
                    break

            if vaults_size > 0:
                last_vault = vaults[:1][0]
                last = last_vault.to_json()
                if last_vault.is_failed():
                    status = False
            else:
                status = True

            status_string = "UP" if status else "Warning"

Our case - we get NON-ZERO vaults_size (main example is s3 storage, even for clean-install process, no-one clears it)

And there was NO last_successful backups

In that case, it'll return "Warning" state with NON_ZERO dump_count - https://github.com/Netcracker/qubership-backup-daemon/blob/ec400e76e54e267a596adb6b773276bd4e065c67/src/httpapi.py#L181

That mean we fail due to our operator handling -

qubership-rabbitmq/operator/src/backup_helper.py

Line 62 in c0129a9

elif status == 'Warning':

To summarize, this occurs not only for clean-install process, this issue may occur ANYTIME when we install RabbitMQ with Backup Daemon(BD) and storage from BD is not empty or contains only NOT SUCCESSFUL backups(expired, for different rabbit version, e.t.c)

One question remains - should we do something about it?

I suggest not, because its a great indicator if something gone wrong.

On the other hand, i don't like idea to "permamently mark BD as failed" after our operator cycle, maybe we should make it do reconcile if it fails or ask backup daemon to make snapshot (to get at least one Successful and totally applicable - this might be bad for heavy loaded envs for many reasons),.