[Proposal] FIPS-Capable default distribution: Shipping BCFIPS in the Default OpenSearch Distribution

[cwperks]

1. Context

OpenSearch currently ships Bouncy Castle FIPS (BCFIPS) jars only in the FIPS distribution (built with -Pcrypto.standard=FIPS-140-3). The default distribution does not include these jars. Because plugins inherit jars from lib/, this creates complexity for any plugin aiming to support FIPS:

• Plugins like Security must maintain separate builds to manage whether BCFIPS jars must be excluded or bundled.
• When both core and a plugin supply BCFIPS jars, installation fails due to jar-hell.

---

2. Decision

We will make the default OpenSearch distribution FIPS-capable by shipping BCFIPS jars in lib/ and exposing them via api scope.

To support this:

• Plugins will be expected to treat BCFIPS as a provided dependency rather than bundling their own.

---

3. Rationale

Benefits

• First-party and third-party plugins no longer need custom FIPS builds.
• The default distro provides the same providers and algorithms used in the FIPS build.
• Plugins stop bundling their own BC/BCFIPS jars.

Considerations

• Community plugins using BC/BCFIPS may need minor adaptation.

• Default distro will include additional crypto jars.

  • FIPS behavior remains opt-in; provider ordering remains unchanged unless FIPS mode is explicitly enabled.

---

4. Alternatives Considered

A. Maintain FIPS-only distribution for BCFIPS (status quo)

• Pros: No change for community plugin authors;
• Cons: Continues requiring dual builds for first-party plugins;

B. Broader jar-hell relaxation without shipping BCFIPS by default

For this alternative:

• Plugin install logic will be updated to ignore jar-hell only when the conflicting jar is identical (matching GAV + checksum).

• Pros: Helps with conflict handling.

• Cons: Does not address the core problem: plugins still lack a consistent FIPS-capable baseline.

---

5. Outcome

• The default distribution becomes FIPS-capable, simplifying plugin development and unifying the runtime crypto environment.

[cwperks]

@reta this was raised again and I'd like to revisit it. Having a single distribution is simpler as we only have one set of artifacts to publish, but wanted to raise this up in an effort to get a definitive decision made on this matter. If we have a separate distribution then we'll need some support from @peterzhuamazon and the release team to publish official artifacts built with -Pcrypto.standard=FIPS-140-3.

[peterzhuamazon]

@cwperks @reta Having separate versions instead of one means all distributions would need to have a secondary copy, effectively raising the 8 (?) total distributions to 14 (?).

However, having a single version would mean the crypto jars to be included for users that do not need FIPS.

I am open to have a single version, but having a param to enable FIPS if needed. Is that doable? Thanks.

[reta]

Thanks @cwperks @peterzhuamazon.

I am open to have a single version, but having a param to enable FIPS if needed. Is that doable? Thanks.

I believe early on during FIPS integration we made a decision to have these dependencies non-optional (basically, we only depend on bc-fips libraries), in this respect having a single distribution with FIPS support would make sense (apparently, we may also need -Dorg.bouncycastle.fips.approved_only=true but that is a detail).

If we have a separate distribution then we'll need some support from @peterzhuamazon and the release team to publish official artifacts built with -Pcrypto.standard=FIPS=140-3.

To me personally the full consequences of taking this path are not clear. I suspect the OpenSearch ecosystem (plugins primarily) may rely on non-FIPS ciphers / algorithms / crypto primitives, so it may be a blocker for them (or at least, breaking change). From the other side, having too many distribution is also a problem - high maintenance costs.

I am wondering if we have good examples to follow, do other open source projects (that are not libraries) offer dual FIPS/non-FIPS modes?

[cwperks]

@reta early on we did not add the BCFIPS deps directly to server (or a lib the server directly or indirectly depends on). I believe that discussion started with opensearch-project/OpenSearch#14912 (comment) and continued on from opensearch-project/OpenSearch#14912 (comment).

That led to the current support which is a separate distribution that operators would have to create with -Pcrypto.standard=FIPS-140-3 since the project does not official public artifacts using that param.

With FIPS-capable default distribution an operator could choose whether to run with FIPS Compliance or not. When running with FIPS compliance, a cluster would only have approved algorithms available to it.

In order to make the default distribution FIPS-Capable it would be necessary to always bundle the BCFIPS deps and provide them in lib/ (whether directly declared in server/build.gradle or transitively from a lib the server depends on)

This is where the BCFIPS jars are copied to lib/ when creating a distro like ./gradlew localDistro -Pcrypto.standard=FIPS-140-3): https://github.com/opensearch-project/OpenSearch/blob/main/distribution/build.gradle#L350-L352

[reta]

@reta early on we did not add the BCFIPS deps directly to server (or a lib the server directly or indirectly depends on). I believe that discussion started with opensearch-project/OpenSearch#14912 (comment) and continued on from opensearch-project/OpenSearch#14912 (comment).

Correct @cwperks , what I meant is that we only refer to BC-FIPS dependencies (and not plain BC), some projects like plugin-cli have hard dependencies [1], many others likely not yet.

In order to make the default distribution FIPS-Capable it would be necessary to always bundle the BCFIPS deps and provide them in lib/ (whether directly declared in server/build.gradle or transitively from a lib the server depends on)

Right, as of today, some tools and some plugins come with FIPS related dependencies, from this perspective lifting that to become the default at the distribution level would make sense. So how about providing a script with default distribution (for example, enforce-fips-compliance.sh, as we do with other things) which will:

1. Bring FIPS dependencies where necessary
2. Change jvm.options to enforce FIPS mode
3. Convert keystores etc to be FIPS compliant
4. ...

[1] https://github.com/opensearch-project/OpenSearch/blob/main/distribution/tools/plugin-cli/build.gradle

[cwperks]

@reta There is a FIPS CLI in the core codebase and documentation is being added shortly here: opensearch-project/documentation-website#11412

That CLI helps prepare a cluster to run in FIPS-approved mode.

So, it sounds like there is consensus that we can bring the BCFIPS dependencies to server/build.gradle at api level so that all components can rely on these deps at compile time and runtime?

[reta]

@reta There is a FIPS CLI in the core codebase and documentation is being added shortly here: opensearch-project/documentation-website#11412

👍

So, it sounds like there is consensus that we can bring the BCFIPS dependencies to server/build.gradle at api level so that all components can rely on these deps at compile time and runtime?

Yes, from my perspective, we should not make things more complicated than they should be. However, this could be considered a major change, I would suggest to roll the CLI tooling out first.

[beanuwave]

BC states that certification is valid only for the tested hardware + software combination. For bc-fips-2.1.0, all tests were conducted on Ubuntu 22.04 + Intel + Java 8/11/17/21 + w/wo PAA (Processor Algorithm Acceleration), which reduces our distribution choice to Ubuntu 22.04 with Java 21.

For the avoidance of doubt, it is hereby stated that the CMVP makes no statement as to the correct operation of the module or the security strengths of the generated keys when so ported if the specific operational environment is not listed on the validation certificate.

https://downloads.bouncycastle.org/fips-java/docs/BC-FJA-SecurityPolicy-2.1.0.pdf

@reta @cwperks How to you feel about reusable cluster configuration across all plugins that require FIPS testing?
This ensures any plugin can be properly tested under FIPS constraints without duplicating configuration.

As already mentioned above, FIPS integration requires several configuration changes:

• -Dorg.bouncycastle.fips.approved_only=true JVM parameter
• Password-protected internal key/value store with at least 14-character passwords
• Replacement of the JDK's default SSL truststore
• Replacement of, or addition to, the JDK's default security.properties file
• Installation of required libraries: bctls-fips, bc-fips ...

[reta]

Thanks @beanuwave

@reta @cwperks How to you feel about reusable cluster configuration across all plugins that require FIPS testing?
This ensures any plugin can be properly tested under FIPS constraints without duplicating configuration.

I think this would be great and significantly reduce the amount of work the plugin developers should do. Could we bundle that as part of FIPS CLI tooling / opensearch-project/documentation-website#11412 ? Or you have a different take on that?

[beanuwave]

@reta I'm considering OpenSearch core’s buildSrc as a valid option: https://github.com/opensearch-project/OpenSearch/pull/20166/files

[mikkeljohnsen]

I have installed opensearch 3.4.0 on RHEL10 with FIPS. But it is not working.

Trying to run this:

/usr/share/opensearch/jdk/bin/keytool   -importcert   -keystore /etc/opensearch/certs/truststore.bcfks   -storetype BCFKS   -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider   -providerpath /usr/share/opensearch/plugins/opensearch-security/bc-fips-2.1.2.jar   -storepass changeit   -alias opensearch-ca   -file /etc/opensearch/certs/ca.crt

But get

WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::load has been called by org.bouncycastle.crypto.fips.NativeLoader$1 in an unnamed module (file:/usr/share/opensearch/plugins/opensearch-security/bc-fips-2.1.2.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

keytool error: java.lang.Exception: Provider "org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider" not found

Have also added "-Dorg.bouncycastle.fips.approved_only=true" this to "/etc/opensearch/jvm.options"

Is FIPS not working ?

[beanuwave]

@mikkeljohnsen Have you tried running keytool with the following options?

        -providername BCFIPS \
        -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
        -providerpath $LIB_PATH/bc-fips-2.1.2.jar

As a suggestion, the fips-demo-installer-cli might make this a bit easier.

[mikkeljohnsen]

@beanuwave It is not working either and opensearch-fips-demo-installer is not working.

/usr/share/opensearch/bin/opensearch-fips-demo-installer generated -p "Q3I5J#reTHGpQ1ljDhwO3"

Generate new BCFKS trust store from system defaults? [y/N] y
Generating BCFKS trust store...
Loading system truststore from: /usr/share/opensearch/jdk/lib/security/cacerts
Loaded 146 certificates from system truststore
Successfully loaded cacerts as PKCS12 format
Converting to BCFKS format: /etc/opensearch/opensearch-fips-truststore.bcfks

Error: java.security.NoSuchProviderException: no such provider: BCFIPS

Gives the same error: Error: java.security.NoSuchProviderException: no such provider: BCFIPS