Add build-tooling to run in the FIPS environment

[beanuwave]

Description

Provides additional build tooling to support builds in FIPS env, including a CLI trust-store installer to override $JAVA_HOME/lib/security/cacerts

• add demo/test CLI configurator with the ability to:
  -- migrate JVM's default SSL trust store to a BCFKS-formatted one
  -- use an existing PKCS#11 trust store
  -- display installed 'KeyStore' providers
  -- show help
  -- execute above commands interactively or in script mode
• add BC libs to standalone REST tests.
• print out 'java.security.properties' for reproducibility information

Related Issues

Resolves RFC

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for f656bd4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 20a5611: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 2241009: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 1829731: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

❌ Patch coverage is 61.00478% with 163 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.07%. Comparing base (252cff8) to head (4368d76).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...ools/cli/fips/truststore/CreateFipsTrustStore.java	22.22%	53 Missing and 3 partials ⚠️
.../opensearch/bootstrap/FipsTrustStoreValidator.java	45.76%	30 Missing and 2 partials ⚠️
...ols/cli/fips/truststore/FipsTrustStoreCommand.java	36.00%	16 Missing ⚠️
...h/tools/cli/fips/truststore/TrustStoreService.java	71.42%	12 Missing and 2 partials ⚠️
...ls/cli/fips/truststore/UserInteractionService.java	81.03%	10 Missing and 1 partial ⚠️
.../org/opensearch/gradle/test/rest/RestTestUtil.java	0.00%	9 Missing ⚠️
.../tools/cli/fips/truststore/SecureRandomHolder.java	27.27%	8 Missing ⚠️
.../cli/fips/truststore/ProviderSelectionService.java	84.61%	4 Missing and 2 partials ⚠️
...search/gradle/test/StandaloneRestTestPlugin.groovy	0.00%	3 Missing ⚠️
...li/fips/truststore/GeneratedTrustStoreCommand.java	25.00%	3 Missing ⚠️
... and 5 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18921      +/-   ##
============================================
- Coverage     73.11%   73.07%   -0.04%     
- Complexity    70661    70692      +31     
============================================
  Files          5724     5741      +17     
  Lines        323498   323915     +417     
  Branches      46852    46900      +48     
============================================
+ Hits         236518   236706     +188     
- Misses        67846    68108     +262     
+ Partials      19134    19101      -33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cwperks]

Patch coverage is showing as 0% because the new tests are guarded with

@BeforeClass
public static void beforeClass() throws Exception {
   assumeTrue("Test should run in FIPS JVM", FipsMode.CHECK.isFipsEnabled());
}

The actual patch coverage is much higher, but the gradle check of this repo does not run with FIPS enabled.

[github-actions]

❌ Gradle check result for 974cec3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 974cec3: SUCCESS

[beanuwave]

@andrross @reta @cwperks Just pushed an update to introduce the new cluster-settings - do you think it's evolve into the right direction?

[github-actions]

❌ Gradle check result for 69d46e3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

Why do we need addSunJceProvider() here? It is already executed for each test case https://github.com/opensearch-project/OpenSearch/pull/18921/files#diff-2cf06f036bcc6d4f14a98424445a8fa2d5c6db18edfb88ee6f046a7fcc4754a3R38, right?

[beanuwave]

To undo the JCE provider removal, we need to add it back both before each test and after the test class completes. This ensures that other test units, which run in random order after SecurityProviderManagerTests and rely on JCE won't fail.

[cwperks]

We should find a way to run more of the gradle check against a FIPS compliant cluster in the future (perhaps the tests in the Rest High-Level client?) which could help to find issues early.

[reta]

To undo the JCE provider removal, we need to add it back both before each test and after the test class completes.

If we do exactly that - why do we need after class hook? The test order will not have any impact since before / after will be called for each test every time.

[beanuwave]

This fixes a race condition I encountered locally after committing this test class. NoSuchProviderException was thrown in unexpected places, suggesting that some test class isn't properly cleaning up after execution. I don't mind withdraw this change and do a proper bug-fix commit if this is more suitable, or the issue simply does not exist any longer.

[reta]

I think we are missing @After (not @AfterClass) to restore the original providers

[beanuwave]

We already have @Before method to restore the original providers for us. This setup/teardown is probably not what you expect:

@Before
@Override
public void setUp() throws Exception {
    super.setUp();
    addSunJceProvider();
}

@After
public void tearDown() throws Exception {
    if (!inFipsJvm()) {
        addSunJceProvider();
    }
}

@AfterClass
// restore the same state as before running the tests.
public static void afterClass() throws Exception {
    if (inFipsJvm()) {
        SecurityProviderManager.removeNonCompliantFipsProviders();
    }
}

[github-actions]

✅ Gradle check result for a5b935f: SUCCESS

[cwperks]

outside of running in fips compliance mode, these will be blank?

[beanuwave]

Yes that's the purpose. Docker warns you that this parameter is not set when running w/o FIPS. E.g.

WARN[0000] The "FIPS_GENERATE_TRUSTSTORE" variable is not set. Defaulting to a blank string.

Inside docker-entrypoint the check ensures the value is true otherwise no action is taken.

[cwperks]

@beanuwave thank you for updating this to a CLI and removing from the bootstrap flow. Overall this looks like the right direction for this change and having the separate CLI can log out to a cluster administrator what its doing to prepare the cluster to run in FIPS approved mode. I don't have any concerns adding picocli and actually think that would be a good replacement for other CLIs in this repo in the future as picocli is very widely used in the Java community.

[github-actions]

❕ Gradle check result for 4368d76: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[reta]

Should the TRUST_STORE_PASSWORD be passed as an argument or entered from the command line?

[beanuwave]

Atm it's auto-generated or entered via command line - but I think we can and should extend the CLI with an option for password.

[beanuwave]

I've updated CLI with new --password option, please have a look.

[reta]

@reta @cwperks @andrross I believe PR is ready for another round of reviews - would you like to take a look?

Thanks @beanuwave for picking it up, did first pass, definitely missed something (the change is large) but we are getting there, thank you.

[beanuwave]

@reta @cwperks Thank you both for taking the time and doing a deep dive on this. I’ll address the remaining comments next week.

[github-actions]

❌ Gradle check result for f826d03: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 7cfeb72: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?