rabbitmq fails to start with a crypto "low entropy" error on FIPS compliant server

[reuvenstr]

Describe the bug

Hi,
After enabling FIPS mode on Ubuntu 22.04 it fails to start rabbit-server (3.12.12-1) with erlang (1:26.2.1-1)

{application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{low_entropy,[{crypto,strong_rand_bytes,1,[{file,\"crypto.erl\"},{line,1155}]},{credentials_obfuscation_svc,check,3,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,197}]},{credentials_obfuscation_svc,init_state,0,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,169}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,980}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,935}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,241}]}]}}},{credentials_obfuscation_app,start,[normal,[]]}}}

Crash dump is being written to: erl_crash.dump...done

Is there any solution without upgrading rabbitmq/erlang versions?
Thanks for your time 😄

Reproduction steps

1. Install ubuntu 22.04, rabbitmq-server. Check everything is fine
2. Enable FIPS mode and restart server
3. rabbitmq server is failing to start
   ...

Expected behavior

rabbitmq server is working fine on ubuntu 22.04 with FIPS mode

Additional context

Any additional information will be provided upon request

[lukebakken]

More than likely the OpenSSL being used by Erlang 26.2.1 is not FIPS compliant, or you haven't enabled FIPS in OpenSSL correctly.

RabbitMQ and Erlang do support FIPS, but Team RabbitMQ only supports that combination for users with commercial contracts (https://www.rabbitmq.com/commercial-offerings)

[reuvenstr]

Looks like you are right, rabbitmq required access to fipsmodule.cnf file, but it was only accessible to root user..

[reuvenstr]

Is it possible to make the error more related to the actual issue, instead of some random explanation?

[lukebakken]

Considering I don't know what the actual issue is in this case, I can't say. I don't know anything about how you've configured RabbitMQ or OpenSSL, nor the OpenSSL version in your environment, nor even how you installed RabbitMQ and Erlang. I don't have the time to guess.

I'm somewhat surprised that a file access issue would cause a low_entropy error.

RabbitMQ is fully open-source, and a contribution would be appreciated.

[michaelklishin]

@reuvenstr I'm not sure what you mean by "random explanation" but you are running a version that is out of community support so you should be grateful that two core team engineers have responded at all.

Note the low_entropy part of the error. It comes directly from the crypto library when the library itself or the OS defaults are too low for it to produce a secure random value (e.g. a random number), which is what credentials_obfuscation does on node boot.

Googling for "OpenSSL low entropy" and "OpenSSL low entropy Erlang" will produce all kinds of discussions without a universal solution except for explicit seeding. Which IIRC we have
done at some point.

There is a dedicated OpenSSL wiki page on the topic.

We haven't seen this error in a few months, possibly credentials_obfuscation was adapted to see the generator or some Erlang's TLS implementation or OpenSSL's crypto functions now have better defaults (in the Erlang 26.x on OpenSSL 3.x era, that is).

[michaelklishin]

Here is the function in question that calls crypto:strong_rand_bytes/1 with the argument of 128, which produces a "low entropy" exception from your OpenSSL.

We don't have enough details to determine how to reproduce the error but any contributions (including investigating the problem, various resolution options and their pros/cons) are welcome. Writing the code is not the hard part here, determining the root cause and a safe workaround is.

[reuvenstr]

@reuvenstr I'm not sure what you mean by "random explanation" but you are running a version that is out of community support so you should be grateful that two core team engineers have responded at all.

Note the low_entropy part of the error. It comes directly from the crypto library when the library itself or the OS defaults are too low for it to produce a secure random value (e.g. a random number), which is what credentials_obfuscation does on node boot.

Googling for "OpenSSL low entropy" and "OpenSSL low entropy Erlang" will produce all kinds of discussions without a universal solution except for explicit seeding. Which IIRC we have done at some point.

There is a dedicated OpenSSL wiki page on the topic.

We haven't seen this error in a few months, possibly credentials_obfuscation was adapted to see the generator or some Erlang's TLS implementation or OpenSSL's crypto functions now have better defaults (in the Erlang 26.x on OpenSSL 3.x era, that is).

Missed that community support already ended. For sure I'm grateful for such a fast response.
And I will be more than happy to help with the investigation of it. Let me try new versions of rabbitmq and erlang, and I will update if the issue was mitigated already, or still exists.

As I meant in my first response, that issue was solved with read permissions to fips conf file, that's why it's a bit confusing with error that I received

[reuvenstr]

I took links from docs and installed the newest versions of rabbitmq and erlang
and the error still exists

ii rabbitmq-server 3.13.4-1 all Multi-protocol messaging and streaming broker
ii erlang-asn1 1:26.2.5.1-1 amd64 Erlang/OTP modules for ASN.1 support
ii erlang-base 1:26.2.5.1-1 amd64 Erlang/OTP virtual machine and base applications
ii erlang-crypto 1:26.2.5.1-1 amd64 Erlang/OTP cryptographic modules
ii erlang-eldap 1:26.2.5.1-1 amd64 Erlang/OTP LDAP library
ii erlang-ftp 1:26.2.5.1-1 amd64 Erlang/OTP FTP client
ii erlang-inets 1:26.2.5.1-1 amd64 Erlang/OTP Internet clients and servers
ii erlang-mnesia 1:26.2.5.1-1 amd64 Erlang/OTP distributed relational/object hybrid database
ii erlang-os-mon 1:26.2.5.1-1 amd64 Erlang/OTP operating system monitor
ii erlang-parsetools 1:26.2.5.1-1 amd64 Erlang/OTP parsing tools
ii erlang-public-key 1:26.2.5.1-1 amd64 Erlang/OTP public key infrastructure
ii erlang-runtime-tools 1:26.2.5.1-1 amd64 Erlang/OTP runtime tracing/debugging tools
ii erlang-snmp 1:26.2.5.1-1 amd64 Erlang/OTP SNMP applications
ii erlang-ssl 1:26.2.5.1-1 amd64 Erlang/OTP implementation of SSL
ii erlang-syntax-tools 1:26.2.5.1-1 amd64 Erlang/OTP modules for handling abstract Erlang syntax trees
ii erlang-tftp 1:26.2.5.1-1 amd64 Erlang/OTP TFTP client and server
ii erlang-tools 1:26.2.5.1-1 amd64 Erlang/OTP various tools
ii erlang-xmerl 1:26.2.5.1-1 amd64 Erlang/OTP XML tools

Openssl version:

OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
built on: Mon Mar 4 13:43:22 2024 UTC
platform: debian-amd64
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-aDFub7/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-3"
MODULESDIR: "/usr/lib/x86_64-linux-gnu/ossl-modules-3"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0xfefa32034f8bffff:0x2c1

What was done

1. Install ubuntu 22.04 and enable ubuntu pro in order to activate fips mode
2. Generate fips config with command (openssl fipsinstall -out /etc/ssl/fipsmodule.cnf -module /usr/lib/x86_64-linux-gnu/ossl-modules-3/fips.so)
3. Configure /ets/ssl/openssl.cnf to use generated config and FIPS enforce mode
   Parts of the file were changed

.include /etc/ssl/fipsmodule.cnf

[provider_sect]
default = default_sect
# The fips section name should match the section name inside the
# included fipsmodule.cnf.
fips = fips_sect

[default_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes

4. Restart rabbitmq and it fails with error from below
   2]: =CRASH REPORT==== 4-Jul-2024::05:38:46.243385 === : crasher: : initial call: credentials_obfuscation_svc:init/1 : pid: <0.149.0> : registered_name: [] : exception error: low_entropy : in function crypto:strong_rand_bytes/1 (crypto.erl, line 1155) : in call from credentials_obfuscation_svc:check/3 (src/credentials_obfuscation_svc.erl, line 197) : in call from credentials_obfuscation_svc:init_state/0 (src/credentials_obfuscation_svc.erl, line 169) : in call from gen_server:init_it/2 (gen_server.erl, line 980) : in call from gen_server:init_it/6 (gen_server.erl, line 935) : ancestors: [credentials_obfuscation_sup,<0.147.0>] : message_queue_len: 0 : messages: [] : links: [<0.148.0>] : dictionary: [] : trap_exit: false : status: running : heap_size: 610 : stack_size: 28 : reductions: 288 : neighbours: : =SUPERVISOR REPORT==== 4-Jul-2024::05:38:46.249993 === : supervisor: {local,credentials_obfuscation_sup} : errorContext: start_error : reason: {low_entropy, : [{crypto,strong_rand_bytes,1, : [{file,"crypto.erl"},{line,1155}]}, : {credentials_obfuscation_svc,check,3, : [{file,"src/credentials_obfuscation_svc.erl"}, : {line,197}]}, : {credentials_obfuscation_svc,init_state,0, : [{file,"src/credentials_obfuscation_svc.erl"}, : {line,169}]}, : {gen_server,init_it,2,[{file,"gen_server.erl"},{line,980}]}, : {gen_server,init_it,6,[{file,"gen_server.erl"},{line,935}]}, : {proc_lib,init_p_do_apply,3, : [{file,"proc_lib.erl"},{line,241}]}]} : offender: [{pid,undefined}, : {id,credentials_obfuscaton_svc}, : {mfargs,{credentials_obfuscation_svc,start_link,[]}}, : {restart_type,permanent}, : {significant,false}, : {shutdown,5000}, : {child_type,worker}] : =CRASH REPORT==== 4-Jul-2024::05:38:46.250240 === : crasher: : initial call: application_master:init/4 : pid: <0.146.0> : registered_name: [] : exception exit: {{shutdown, : {failed_to_start_child,credentials_obfuscaton_svc, : {low_entropy, : [{crypto,strong_rand_bytes,1, : [{file,"crypto.erl"},{line,1155}]}, : {credentials_obfuscation_svc,check,3, : [{file,"src/credentials_obfuscation_svc.erl"}, : {line,197}]}, : {credentials_obfuscation_svc,init_state,0, : [{file,"src/credentials_obfuscation_svc.erl"}, : {line,169}]}, : {gen_server,init_it,2, : [{file,"gen_server.erl"},{line,980}]}, : {gen_server,init_it,6, : [{file,"gen_server.erl"},{line,935}]}, : {proc_lib,init_p_do_apply,3, : [{file,"proc_lib.erl"},{line,241}]}]}}}, : {credentials_obfuscation_app,start,[normal,[]]}} : in function application_master:init/4 (application_master.erl, line 142) : ancestors: [<0.145.0>] : message_queue_len: 1 : messages: [{'EXIT',<0.147.0>,normal}] : links: [<0.145.0>,<0.44.0>] : dictionary: [] : trap_exit: true : status: running : heap_size: 987 : stack_size: 28 : reductions: 239 : neighbours: : =INFO REPORT==== 4-Jul-2024::05:38:46.250605 === : application: credentials_obfuscation : exited: {{shutdown, : {failed_to_start_child,credentials_obfuscaton_svc, : {low_entropy, : [{crypto,strong_rand_bytes,1, : [{file,"crypto.erl"},{line,1155}]}, : {credentials_obfuscation_svc,check,3, : [{file,"src/credentials_obfuscation_svc.erl"}, : {line,197}]}, : {credentials_obfuscation_svc,init_state,0, : [{file,"src/credentials_obfuscation_svc.erl"}, : {line,169}]}, : {gen_server,init_it,2, : [{file,"gen_server.erl"},{line,980}]}, : {gen_server,init_it,6, : [{file,"gen_server.erl"},{line,935}]}, : {proc_lib,init_p_do_apply,3, : [{file,"proc_lib.erl"},{line,241}]}]}}}, : {credentials_obfuscation_app,start,[normal,[]]}} : type: transient : Runtime terminating during boot (terminating) : : Crash dump is being written to: erl_crash.dump...done : [os_mon] cpu supervisor port (cpu_sup): Erlang has closed : [os_mon] memory supervisor port (memsup): Erlang has closed
5. Change permissions of /etc/ssl/fipsmodule.cnf to 0600, and restart, everything is ok

Any additional info that you need?

There is no additional configuration were performed on rabbitmq or openssl besides from above

[lukebakken]

Change permissions of /etc/ssl/fipsmodule.cnf to 0600, and restart, everything is ok

Thanks for the information. We'll see if we can make the error more meaningful, but it will be low priority.

[michaelklishin]

It's not the versions of RabbitMQ and Erlang that produce this error, it is something in your OS and the OpenSSL version's expectation of what constitutes a "sufficient level of entropy."

credentials_obfuscation can catch the exception and log something more specific but it cannot point you to the root cause. There are no known common root causes and the answer likely lies in the OpenSSL codebase (the parts responsible for random value generation).