strange crash of credentials_obfuscation

[gomoripeti]

We've noticed the following crash in a server log. It happened during or soon after a startup of RabbitMQ 3.9.14 OTP 24.1.7 (during an upgrade from RMQ 3.9.7 OTP 24.0.4) Unfortunately this was more than a month ago so we don't have much more details

I am just wondering in what scenario this could have happened?

• It is not possible that the secret was set before decrypt but after encrypt because the term is already {encypted, ...}
• Decryption does not work with secret <<"nocookie">>
• Could the secret have changed? In what cases does RabbitMQ updates the secret of credentials_obfuscation? Is it always the erlang cookie or is it configurable?

** Generic server credentials_obfuscation_svc terminating
** Last message in was {decrypt,
                           {encrypted,
                               <<"...">>}}
** When Server state == {state,true,aes_128_cbc,sha256,1,
                               <<"<secret>">>}
** Reason for termination ==
** {badarg,
       [{binary,part,
            [<<...>>,
             0,-10],
            [{error_info,#{module => erl_stdlib_errors}}]},
        {credentials_obfuscation_pbe,unpad,1,
            [{file,"src/credentials_obfuscation_pbe.erl"},{line,119}]},
        {credentials_obfuscation_svc,handle_call,3,
            [{file,"src/credentials_obfuscation_svc.erl"},{line,111}]},
        {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},
        {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
** Client <0.20885.0> stacktrace
** [{gen,do_call,4,[{file,"gen.erl"},{line,233}]},
    {gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},
    {rabbit_shovel_parameters,'-deobfuscated_uris/2-lc$^0/1-0-',1,
                              [{file,"rabbit_shovel_parameters.erl"},
                               {line,446}]},
    {rabbit_shovel_parameters,deobfuscated_uris,2,
                              [{file,"rabbit_shovel_parameters.erl"},
                               {line,446}]},
    {rabbit_shovel_parameters,parse_amqp091_source,1,
                              [{file,"rabbit_shovel_parameters.erl"},
                               {line,400}]},
    {rabbit_shovel_parameters,parse,3,
                              [{file,"rabbit_shovel_parameters.erl"},
    {rabbit_shovel_worker,init,1,
                          [{file,"rabbit_shovel_worker.erl"},{line,55}]},
    {gen_server2,init_it,6,[{file,"gen_server2.erl"},{line,565}]}]

  crasher:
    initial call: credentials_obfuscation_svc:init/1
    pid: <0.123.0>
    registered_name: credentials_obfuscation_svc
    exception error: bad argument
      in function  binary:part/3
         called as binary:part(<<...>>,
                               0,-10)
         *** argument 3: out of range
      in call from credentials_obfuscation_pbe:unpad/1 (src/credentials_obfuscation_pbe.erl, line 119)
      in call from credentials_obfuscation_svc:handle_call/3 (src/credentials_obfuscation_svc.erl, line 111)
      in call from gen_server:try_handle_call/4 (gen_server.erl, line 721)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 750)
    ancestors: [credentials_obfuscation_sup,<0.121.0>]
    message_queue_len: 0
    messages: []
    links: [<0.122.0>]
    dictionary: [{delegate,delegate_5},
                  {guid,{{649508584,4131680279,733544890,2350673331},0}},
                  {{xtype_to_module,topic},rabbit_exchange_type_topic}]
    trap_exit: false
    status: running
    heap_size: 6772
    stack_size: 29
    reductions: 69870
  neighbours: