Make FIPS test less shaky

Signed-off-by: Matthias Büchse <[email protected]>

Author: mbuechse

Standards/scs-0101-w1-entropy-implementation-testing.md

@@ -36,7 +36,7 @@ reported as an error:
 - the service `rngd` is not running,
 - the special file `/proc/sys/kernel/random/entropy_avail` does not contain
   the value 256 (pinned since kernel 5.18),
-- the number of FIPS 140-2 failures exceeds 3 out of 1000 blocks
+- the number of FIPS 140-2 failures exceeds 5 out of 1000 blocks
   tested, as determined by `cat /dev/random | rngtest -c 1000` .
 
 Note: The latter two items act as surrogates for the following item, which
@@ -50,6 +50,8 @@ The following items MUST be detected and reported as a warning:
 
 - any flavor missing the attribute `hw_rng:allowed=True`,
 - any image missing the attribute `hw_rng_model: virtio`,
+- the number of FIPS 140-2 failures exceeds 3 out of 1000 blocks
+  tested (compare with errors).
 
 Note that the requirement regarding the kernel patch level will not be
 checked, because of two reasons: (a) we already check the file `entropy_avail`

Tests/iaas/entropy/entropy-check.py

@@ -145,10 +145,16 @@ def check_fips_test(lines, image_name):
         if failure_re:
             fips_failures = failure_re.string[failure_re.regs[0][0]:failure_re.regs[0][1]].split(" ")[1]
             if int(fips_failures) <= 3:
-                return True  # this is the single 'successful' code path
+                return True  # strict test passed
+            logger.warning(
+                f"VM '{image_name}' didn't pass the strict FIPS 140-2 testing. "
+                f"Expected a maximum of 3 failures, got {fips_failures}."
+            )
+            if int(fips_failures) <= 5:
+                return True  # lenient test passed
             logger.error(
                 f"VM '{image_name}' didn't pass the FIPS 140-2 testing. "
-                f"Expected a maximum of 3 failures, got {fips_failures}."
+                f"Expected a maximum of 5 failures, got {fips_failures}."
             )
         else:
             logger.error(f"VM '{image_name}': failed to determine fips failures")