Split test_psa_compliance.py #209

gilles-peskine-arm · 2025-09-11T20:26:58Z

Split the PSA API compliance script test_psa_compliance.py into an engine plus a pre-branch runner, like with many similar scripts. Now each branch has the knowledge of which version of psa-arch-tests it runs against and how to adjust the results if needed.

Also do the bare minimum to allow TF-PSA-Crypto to declare SPAKE2+ key types and algorithms in Mbed-TLS/TF-PSA-Crypto#453.

Fixes #158.

PR checklist

Please remove the segment/s on either side of the | symbol as appropriate, and add any relevant link/s to the end of the line.
If the provided content is part of the present PR remove the # symbol.

changelog not required because: test only
development PR not required because: crypto only
TF-PSA-Crypto PR Split test_psa_compliance.py and update compliance version to 1.2 TF-PSA-Crypto#453
framework PR here
3.6 PR provided Backport 3.6: Split test_psa_compliance.py mbedtls#10400
tests provided

Signed-off-by: Gilles Peskine <[email protected]>

This will allow consuming branches to each have their executable entry point. Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

…thms Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

…odgy) Signed-off-by: Gilles Peskine <[email protected]>

When looking for constructors, do complain if we see an unusual macro with a parameter: there's a significant chance that it's something new that will require specific handling. As a consequence, we need to explicitly skip more things that are known not to be constructors. Keep ignoring macros without parameters that don't look like constructors for the types we care about. Those probably don't matter. Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

Having separate patch files has several benefits: * They're available for integrators who wouldn't use our script to test compliance. * We keep them separate so they're easier for us to keep track of, and apply separately if needed. * No need to cheat with unchanged empty lines (normally represented by a line containing a single space in a patch file) to keep `check_files.py` happy. Signed-off-by: Gilles Peskine <[email protected]>

…same name mypy can't deal with two modules with the same basename on its command line. We don't normally want modules with the same name in different directories, to avoid confusion, but it can happen occasionally while moving files across repositories. Signed-off-by: Gilles Peskine <[email protected]>

bensze01

Initial feedback for the patch file handling, still need to go through the rest of the PR.

scripts/mbedtls_framework/psa_compliance.py

An unchanged blank line results in a line containing a single space in the diff. Signed-off-by: Gilles Peskine <[email protected]>

Signed-off-by: Gilles Peskine <[email protected]>

mpg

Looking pretty good to me! I just have a question that I'd like answered before I can approve - my other point is completely optional.

mpg · 2025-09-15T09:24:51Z

scripts/mbedtls_framework/psa_compliance.py

        subprocess.check_call(['git', 'checkout', '--force', 'FETCH_HEAD'])

-        if patch:
+        if patch_files:


I don't understand the logic behind running git reset only if we have a patch to apply. It seems to me that git reset is needed if we had a patch to apply the last time we ran this script (and we are re-using the repo), which might not be the same as we have one now. Also, I don't think calling git reset when not strictly needed does any harm. So, I'd be inclined to just call it unconditionally. Am I missing something?

I wanted to minimize changes in 3.6 which doesn't need any of the new fancy stuff. Also, arguably, it makes things worse for local debugging where you might want to edit the cached psa-arch-test tree until you get it right. On the other hand, I agree with you that I would have made git reset unconditional if I was doing this from scratch. I'll defer to Bence's preference on that since he know this script's history a lot better than we do.

In my local copy, I've now changed this to be controlled by the psa-arch-tests ref specification:

With no argument or --tests-ref=REFSPEC, check that out, reset, and apply patches.

With --tests-ref=, reuse whatever is in the working tree: no git checkout, no reset, no patches. (To do reset+patch but not checkout, you can use --tests-ref=HEAD.)

Does that seem sensible?

Sounds good!

scripts/mbedtls_framework/psa_compliance.py

mpg

LGTM, thanks for answering my questions.

bensze01

Just one nit, but we can file that as an improvement for after the release

bensze01 · 2025-09-12T21:14:49Z

scripts/mbedtls_framework/psa_compliance.py

+        patch_file_glob = os.path.join(args.patch_directory, '*.patch')
+        patch_files = sorted(glob.glob(patch_file_glob))


Nit: This allows users to pass a patch_directory containing literal glob characters, which might lead to unexpected results.

glob.glob also silently ignores non-existent patch directories

Something like

patch_files = sorted(pathlib.Path(patch_directory).glob('*.patch'))

Would allow us to separate the glob from the user input

gilles-peskine-arm · 2025-10-04T06:38:25Z

I've addressed the non-blocking review comments from this PR and its consumers in #221 and its consumers.

gilles-peskine-arm added 6 commits September 11, 2025 17:15

Put the command line entry point in a function

3008a4a

Signed-off-by: Gilles Peskine <[email protected]>

Move the bulk of test_psa_compliance.py to a module

26cb1a1

This will allow consuming branches to each have their executable entry point. Signed-off-by: Gilles Peskine <[email protected]>

Move branch-dependent defaults to the calling script

43c6cbc

Signed-off-by: Gilles Peskine <[email protected]>

Support applying a patch to the compliance suite

4ff6a49

Signed-off-by: Gilles Peskine <[email protected]>

Teach crypto_knowledge the categories of SP800_100 and SPAKE2P algori…

af6b189

…thms Signed-off-by: Gilles Peskine <[email protected]>

Skip all knowledge of SPAKE2+ keys for now

565b4c3

Signed-off-by: Gilles Peskine <[email protected]>

gilles-peskine-arm added the needs-ci Needs to pass CI tests label Sep 11, 2025

gilles-peskine-arm added this to Roadmap pull requests (new board) Sep 11, 2025

gilles-peskine-arm added size-s Estimated task size: small (~2d) priority-high High priority - will be reviewed soon labels Sep 11, 2025

github-project-automation bot moved this to In Development in Roadmap pull requests (new board) Sep 11, 2025

This was referenced Sep 11, 2025

Backport 3.6: Split test_psa_compliance.py Mbed-TLS/mbedtls#10400

Open

Split test_psa_compliance.py and update compliance version to 1.2 Mbed-TLS/TF-PSA-Crypto#453

Merged

Implement SPAKE2+ public key import and export Mbed-TLS/TF-PSA-Crypto#448

Draft

gilles-peskine-arm added 3 commits September 12, 2025 10:44

Fix [] used as a default argument value (mutable default values are d…

b4035db

…odgy) Signed-off-by: Gilles Peskine <[email protected]>

Treat xxx_HAS_xxx as not a constructor, like xxx_IS_xxx etc.

ca98305

Signed-off-by: Gilles Peskine <[email protected]>

gilles-peskine-arm force-pushed the compliance-split-framework branch from 7557dd0 to ca98305 Compare September 12, 2025 15:02

gilles-peskine-arm added 2 commits September 12, 2025 17:46

bensze01 requested changes Sep 12, 2025

View reviewed changes

scripts/mbedtls_framework/psa_compliance.py Outdated Show resolved Hide resolved

scripts/mbedtls_framework/psa_compliance.py Outdated Show resolved Hide resolved

scripts/mbedtls_framework/psa_compliance.py Outdated Show resolved Hide resolved

gilles-peskine-arm added 3 commits September 12, 2025 21:59

Allow patch files to have trailing whitespace

581da21

An unchanged blank line results in a line containing a single space in the diff. Signed-off-by: Gilles Peskine <[email protected]>

Sort patch files for reproducibility

aebb1fa

Signed-off-by: Gilles Peskine <[email protected]>

Simplify passing a file to subprocess stdin

44ea713

Signed-off-by: Gilles Peskine <[email protected]>

gilles-peskine-arm added needs-review Every commit must be reviewed by at least two team members, needs-reviewer This PR needs someone to pick it up for review and removed needs-ci Needs to pass CI tests labels Sep 13, 2025

mpg reviewed Sep 15, 2025

View reviewed changes

gilles-peskine-arm requested review from bensze01 and mpg September 17, 2025 19:10

mpg approved these changes Sep 29, 2025

View reviewed changes

bensze01 approved these changes Sep 30, 2025

View reviewed changes

github-project-automation bot moved this from In Development to Has Approval in Roadmap pull requests (new board) Sep 30, 2025

bensze01 merged commit d80c4f9 into Mbed-TLS:main Sep 30, 2025
4 checks passed

github-project-automation bot moved this from Has Approval to Done in Roadmap pull requests (new board) Sep 30, 2025

bensze01 added approved Design and code approved - may be waiting for CI or backports and removed needs-review Every commit must be reviewed by at least two team members, needs-reviewer This PR needs someone to pick it up for review labels Sep 30, 2025

This was referenced Oct 3, 2025

test_psa_compliance.py doesn't detect if psa-arch-tests crashes #220

Open

Cleanups in psa_compliance.py #221

Open

		patch_file_glob = os.path.join(args.patch_directory, '*.patch')
		patch_files = sorted(glob.glob(patch_file_glob))

Split test_psa_compliance.py #209

Split test_psa_compliance.py #209

Uh oh!

Conversation

gilles-peskine-arm commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

bensze01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mpg left a comment

Choose a reason for hiding this comment

Uh oh!

mpg Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

gilles-peskine-arm Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

gilles-peskine-arm Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpg Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mpg left a comment

Choose a reason for hiding this comment

Uh oh!

bensze01 left a comment

Choose a reason for hiding this comment

Uh oh!

bensze01 Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gilles-peskine-arm commented Oct 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gilles-peskine-arm commented Sep 11, 2025 •

edited

Loading

gilles-peskine-arm Sep 30, 2025 •

edited

Loading

bensze01 Sep 12, 2025 •

edited

Loading