Troubleshooting BouncyCastle FIPS-related Issues with Quarkus Unit Tests and it possibly related to quarkus's class loading strategy

[cuichenli]

I apologize for the somewhat ambiguous title of this message. The issue we're encountering is intricate, but I'll do my best to clarify it for you. We've observed that when incorporating bouncycastle fips into our Quarkus application, it's causing unit test failures, which I suspect are related to the manner in which dependencies are managed by Quarkus.

For illustration, consider the repository at https://github.com/cuichenli/quarkus-fips. Running the tests on the latest commit should result in success; however, if we remove the line in application.properties or if we remove the java file src/main, the test fails at: https://github.com/cuichenli/quarkus-fips/blob/master/app/src/test/java/org/example/AppTest.java#L43.

To provide further details, here's how we encountered the issue, as I want to ensure we're addressing it correctly:

Initially, we had a basic setup with an empty src directory and some elementary test code. You can view this initial setup at https://github.com/cuichenli/quarkus-fips/tree/init. At this point, the unit test was failing. Our investigation led us to discover that the value of a ThreadLocal was being altered during the keyGenerator.init(256); call, which caused the test to fail. Although I don't have access to the BouncyCastle source code to show the direct cause, the change in the ThreadLocal value was observed without any thread switching—the CryptoServicesRegistrar.threadFipsMode changed from true to null within the function call.

To circumvent this, we examined decompiled BouncyCastle code and found that setting a JVM argument could enforce a default value across all threads. This adjustment marks our second stage of troubleshooting, the details of which can be found at https://github.com/cuichenli/quarkus-fips/tree/use-jvm-args. The test no longer failed at the previous line, but the expected exception was not being caught. After some refinement, by altering the caught exception type, the test passed as seen at https://github.com/cuichenli/quarkus-fips/tree/catch-assertionerror.

The behavior seemed unusual, after some investigation, we noticed it could be a potential classloader issue. We then configured the classloader to prioritize parent classes and reverted some earlier changes, as at https://github.com/cuichenli/quarkus-fips/tree/use-parent-first, but the tests began to fail again.

Interestingly, creating a dummy file in the src directory allowed the tests to pass once more, and now we are on the latest code.

I've shared the full context of the issue above and would be immensely grateful for any insights or assistance in understanding what exactly is happening on the Quarkus side. Your guidance will also help in verifying that we're approaching the resolution correctly.

Thank you for your time and support.

[quarkus-bot[bot]]

/cc @Karm (securepipeline), @jerboaa (securepipeline)

[sberyozkin]

Thanks @cuichenli for this analysis.

Can you check if registering BC FIPS providers as shown in https://quarkus.io/guides/security-customization#bouncy-castle-fips or https://quarkus.io/guides/security-customization#bouncy-castle-jsse-fips works ?

[cuichenli]

Thanks @sberyozkin . But unfortunately, we have tried it and it did not work, you can refer to this code: https://github.com/cuichenli/quarkus-fips/tree/try-security-extension

[cuichenli]

I have an update following my discussion with Copilot, although I'm not entirely certain about its accuracy.

It appears the issue stems from the use of multiple class loaders. Specifically, the ThreadLocal variable within the fips library is loaded by the AppClassLoader. However, when this ThreadLocal is set during our unit tests, it is done so under the Quarkus class loader. As a result, the value of ThreadLocal is established within the scope of Quarkus. Nonetheless, when the fips library executes its logic, it checks for the ThreadLocal variable within the AppClassLoader's scope. This discrepancy leads to a null value being encountered, which in turn triggers the error.

Additionally, I noticed that the previous fix was ineffective when the src/main directory lacked any files. It seems that Quarkus does not load the application.properties file from src/main/resources during unit tests if there is no source code present. However, by placing the property in src/test/resources/application.properties, the tests now successfully pass even when there is no source code.

[sberyozkin]

@cuichenli I've added a test showing how to confirm BC FipsMode is working as expected, see #40211.
Note BCFIPS must be passed as a provider to Java crypto providers like KeyGenerator, the BC guide and all the examples on the web do have BCFIPS added as a last provider, and therefore require specifying a provider name.

[cuichenli]

@sberyozkin thanks for your help, i have updated my repo to follow the test you made but still, the test is failing: https://github.com/cuichenli/quarkus-fips/tree/follow-example

one main difference i can notice is that you set up the BC related stuff in src/main while our code is doing it in src/test, so i still feel it could be something wrong on how the code / classes are loaded from quarkus side.

[sberyozkin]

@cuichenli I think it is important to confirm that the Quarkus endpoint is FIPS enabled, so I'm not sure what you'd like to prove with putting that code in the actual unit test class. IMHO we won't be able to reach any conclusion with your setup

[cuichenli]

Yes, I completely agree with you. I am posting those out of curiosity, as I would like to understand what is actually happening.

[cuichenli]

We have got yet another issue, when FIPS and kubernetes dev services used together, the unit test would not event start. Example: https://github.com/cuichenli/quarkus-fips/tree/kubernetes-client

To reproduce the issue, set your KUBECONFIG environment variable to be some dummy file so we can ensure the dev services will be started, then run the test. Example:

export KUBECONFIG=/dafsa/f/saf/
./gradlew test --tests "org.example.KubernetesTest" -i

Once the dev service is created, the kubernetes-client extension will wait forever to connect to the kubernetes api server. By checking the source code of https://github.com/dajudge/kindcontainer, i noticed that it will run Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider()); https://github.com/dajudge/kindcontainer/blob/master/src/main/java/com/dajudge/kindcontainer/client/ssl/SslUtil.java#L34 . and if i use a custom built jar with this line commented out, the tests will pass without any issue.

[sberyozkin]

@cuichenli What happens if you only add

implementation("org.bouncycastle:bc-fips:1.0.2.4")
implementation("io.quarkus:quarkus-security")

?

[sberyozkin]

@cuichenli
Unfortunately, Quarkus can't control this code, https://github.com/dajudge/kindcontainer/blob/master/src/main/java/com/dajudge/kindcontainer/client/ssl/SslUtil.java#L34.

I think you need to raise an enhancement request in that project instead, to make sure it can run with BCFIPS or BCFIPSTLS mode

[cuichenli]

What happens if you only add

the tests will hang forever on

2024-04-29 20:39:18,850 INFO  [com.daj.kin.ApiServerContainer] (build-14) Waiting for API server...

I think you need to raise an enhancement request in that project instead, to make sure it can run with BCFIPS or BCFIPSTLS mode

If we can confirm there is nothing we can do on quarkus side, then surely i will try to raise an issue on kindcontainer side. but i still do not understand why this happened, i tried to use debugger to step into the code in kindcontainer, and it reported issue on classloader. something like the following

loader constraint violation: when resolving method 'void org.bouncycastle.crypto.CryptoServicesRegistrar.checkConstraints(org.bouncycastle.crypto.CryptoServiceProperties)' the class loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @404f52c3 of the current class, org/bouncycastle/jce/provider/BouncyCastleProvider, and the class loader 'app' for the method's defining class, org/bouncycastle/crypto/CryptoServicesRegistrar, have different Class objects for the type org/bouncycastle/crypto/CryptoServiceProperties used in the signature (org.bouncycastle.jce.provider.BouncyCastleProvider is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @404f52c3, parent loader 'app'; org.bouncycastle.crypto.CryptoServicesRegistrar is in unnamed module of loader 'app')

[sberyozkin]

@cuichenli Can you remove the test code trying to use this BC Crypto code in the test class ? Simply listing those 2 dependencies should not force Quarkus to try to load it

[cuichenli]

yes i was running the test with this test loading BC Crypto removed. sorry i did not update the git repo, the change remains on my local machine.