Erlang 23 and above is not compatible with the Federal Information Processing Standards (FIPS) cluster

[fanzhang15]

Rabbitmq pod is restarting continuously.

# oc get po wd-rabbitmq-discovery-0 -w
NAME                      READY   STATUS             RESTARTS   AGE
wd-rabbitmq-discovery-0   0/1     CrashLoopBackOff   14         108m

Log

2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0> ** Stacktrace =
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0> **  [{crypto,evp_generate_key_nif,[x25519,undefined],[]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>      {ssl_cipher,generate_client_shares,2,
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>                  [{file,"ssl_cipher.erl"},{line,1394}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>      {ssl_gen_statem,initial_hello,3,[{file,"ssl_gen_statem.erl"},{line,458}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>      {gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1166}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>      {tls_connection,init,1,[{file,"tls_connection.erl"},{line,154}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0> ** Client <0.328.0> stacktrace
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0> ** [{gen,do_call,4,[{file,"gen.erl"},{line,208}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>     {gen_statem,call_dirty,4,[{file,"gen_statem.erl"},{line,671}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>     {ssl_gen_statem,call,2,[{file,"ssl_gen_statem.erl"},{line,1166}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>     {ssl_gen_statem,handshake,2,[{file,"ssl_gen_statem.erl"},{line,222}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>     {tls_gen_connection,start_fsm,8,
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>                         [{file,"tls_gen_connection.erl"},{line,89}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>     {ssl_gen_statem,connect,8,[{file,"ssl_gen_statem.erl"},{line,191}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>     {ssl,connect,4,[{file,"ssl.erl"},{line,606}]},
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>     {http_transport,connect,4,[{file,"http_transport.erl"},{line,110}]}]
2022-05-31 04:53:41.995347+00:00 [erro] <0.330.0>
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>   crasher:
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>     initial call: ssl_gen_statem:init/1
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>     pid: <0.330.0>
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>     registered_name: []
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>     exception error: bad argument
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>       in function  crypto:evp_generate_key_nif/2
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>          called as crypto:evp_generate_key_nif(x25519,undefined)
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>       in call from ssl_cipher:generate_client_shares/2 (ssl_cipher.erl, line 1394)
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>       in call from ssl_gen_statem:initial_hello/3 (ssl_gen_statem.erl, line 458)
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>       in call from gen_statem:loop_state_callback/11 (gen_statem.erl, line 1166)
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>       in call from tls_connection:init/1 (tls_connection.erl, line 154)
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>     ancestors: [tls_connection_sup,tls_sup,ssl_connection_sup,ssl_sup,
2022-05-31 04:53:42.076696+00:00 [erro] <0.330.0>                   <0.93.0>]

This doesn't happen if using Rabbitmq 3.8

The problem Is RabbitMQ 3.9 uses ERLang 23 versus the 22.3 for rabbitmq 3.8. It looks like Erlang 23 had an issue where it does not work with FIPS which may extend into the early 24 release. We also tested use RabbitMQ 3.10.5 with Erlang 25 and the issue still exists.

Please let us know if there's a way/plan to fix this.

[lukebakken]

I believe everything you need to know is covered in the discussion to which you linked: link.

FIPS can't be supported until the underlying OpenSSL and Erlang do - erlang/otp#4818 (comment)

See the comment by @HansN:

OpenSSL 3.0 will have a FIPS module they say, but 3.0 is not supported by OTP yet. We have it on the story list, but it is not prioritized and I can't promise when it will be.

@michaelklishin provides a "workaround" of sorts that may suffice - only use TLS 1.3.

[michaelklishin]

Our team does not develop or maintain Erlang/OTP. We do have a couple of updates from the trenches, though:

• Erlang 25 may get FIPS support later this year (no promises)
• Erlang 26 (or maybe 25 by 25.3 or so) may get OpenSSL 3.0 support, which is orthogonal to FIPS but is a step in the right direction for security hardening
• Erlang 25 supports TLS 1.3 today. It's an all or nothing proposition but you can use it with RabbitMQ today.

[michaelklishin]

It looks like Erlang will gain FIPS mode compatibility with OpenSSL 3.x as of 26.1 or 26.2.

[michaelklishin]

There is: it will be available in the next version of the commercial RabbitMQ edition (as an option, not forced down anyone's throat).

Our team won't package it for everyone to use for free. Any open source user can build their own Erlang with FIPS, which takes a certain amount of effort but these days is significantly easier than it was just a couple of years ago. And OpenSSL 3.x-compatible.

[ryan-morris]

@michaelklishin since everything looks like it has now been released with support for FIPS, are you saying that unless we build from source (with FIPS flags), it's only available in the commercial version? Is FIPS support currently available in the commercial offering?

It took me a bit of digging to find this message, from what I could tell everything should be good to go but am still getting the low_entropy error with everything up to date (from official rmq apt repos) and fips enabled, so just looking for a little bit of clarification.

[lukebakken]

are you saying that unless we build from source (with FIPS flags), it's only available in the commercial version?

Yes.

Is FIPS support currently available in the commercial offering?

Yes.

[ryan-morris]

@lukebakken lightening fast response, much appreciated. I've put in a contact request for more info on the commercial version. Not to derail the conversation here too much, but is that a fairly easy swap over for an existing cluster? Just change out the repos and update?

[michaelklishin]

Correct, that's what I meant.

The migration path depends on how you provision RabbitMQ, the commercial edition has three installation options right now. Debian and RPM packages are not available, for example, but on Kubernetes it should be comparable to "swapping the repository and the image."

There is an intermediate option of buying support for OSS RabbitMQ but that does not currently include "standalone" (as in, not a part of an OCI image or an OVA) Erlang packages with FIPS.