Implement structured evidence in rules and results

[rhmdnd]

Compliance scanning results often provide a simple "yes/no" or "pass/fail" answer, which isn't always helpful for governance, risk management, and operational response. We need to evolve our Rule and CheckResult structs to include evidence that explains why a check passed or failed, transforming these results into actionable intelligence.

This will significantly benefit both technical and non-technical users. Auditors and compliance officers will gain context, while system administrators will be able to quickly zero in on problems.

Two-Part Solution

To provide a complete picture, the evidence should be delivered in two formats:

1. Human-Readable Evidence: A concise, clear explanation of the check's outcome. This is especially useful for auditors and non-technical stakeholders who need to understand the "why" behind a failure. For example, a human-readable message could be "Some pods are missing required CPU resource limits."

2. Structured Evidence: This is machine-readable data, such as a Kubernetes manifest, that provides the exact details needed for remediation. This is crucial for a technical audience, like system or cluster administrators, who need to identify and fix the specific issue. This structured data can document either compliant or non-compliant resources.

Use Cases and Examples

We can support two primary use cases by providing evidence for both passing and failing checks.

Case 1: Failure Remediation

For checks that fail, the structured evidence will identify the specific non-compliant resources. This allows administrators to quickly find and fix the problem. The human-readable failureReason gives the context, while the nonCompliantResources provides the technical details.

id: pods-must-have-cpu-limits
status: FAIL
failureReason: "Some pods are missing required CPU resource limits."
structuredEvidence:
  nonCompliantResources:
    - apiVersion: v1
      kind: Pod
      metadata:
        name: frontend
        namespace: production
      spec:
        containers:
          - name: frontend
            image: nginx:1.21
            resources: {} 

Case 2: Proving Compliance

For checks that pass, the structured evidence can document the compliant resources. This is valuable for auditors and compliance officers who need to verify completeness and prove that an environment meets a certain standard. The compliantResources field would list all resources that passed the check.

id: pods-must-have-cpu-limits
status: PASS
structuredEvidence:
  compliantResources:
    - apiVersion: v1
      kind: Pod
      metadata:
        name: frontend
        namespace: production
      spec:
        containers:
          - name: frontend
            image: nginx:1.21
            resources:
              limits:
                cpu: 500m
    - apiVersion: v1
      kind: Pod
      metadata:
        name: backend
        namespace: production
      spec:
        containers:
          - name: backend
            image: postgres:16
            resources:
              limits:
                cpu: 4000m

Proposed Field Structure

We should update the Rule and CheckResult structs to include two new fields:

• failureReason: A human-readable message explaining why a check failed.
• structuredEvidence: A machine-readable field containing either compliantResources or nonCompliantResources.

While keeping errorMessage reserved specifically for technical issues (e.g., a connection error during scanning or permission issues).

By implementing this, we can ensure that every scan result provides the necessary context and technical data for security teams to create actionable tickets, enabling administrators to easily remediate issues and maintain a compliant state.

[rhmdnd]

It would be a good idea to let content authors (users) define the structuredEvidence when they create rules. This would allow them to specify whether a rule should fetch compliantResources or nonCompliantResources.

When a user then invokes the scanner, they can choose if they want to see all of this information. Returning all compliant resources for a check like CPU limits on pods could be incredibly noisy. This approach means the rule itself doesn't need to be modified for different use cases. It can be written once and then configured at scan time to meet specific evidence requirements.

Here's an idea of what that could look like:

By default, if a rule fails, the compliance-sdk would only set the failureReason. We could then add two fields to the ScanConfig struct:

type ScanConfig struct {
    ShowCompliantEvidence    bool
    ShowNonCompliantEvidence bool
}

This would give users the flexibility to generate evidence based on their specific use case. For example, a user running a quick scan to check their compliance posture might not need all the details, but a user in the middle of an audit would need to supply all the compliant resources to prove their compliance posture.

[rhmdnd]

Something else that would be important to potentially include as part of the structured evidence is the expression. That way users know the resource, and exactly what they were evaluated against to be in, or out, of compliance.

This could be important especially as rules change over time.