Skip to content

chore(python): create custom Makefile commands for fuzzing test vectors #1634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: fuzzTestingInTestVectors
Choose a base branch
from
Open

chore(python): create custom Makefile commands for fuzzing test vectors #1634

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d202f77
chore(python): create custom Makefile commands for fuzzing test vectors
ShreyesS Jul 23, 2025
678e6cd
Update TestVectorsAwsCryptographicMaterialProviders/Makefile
ShreyesS Jul 24, 2025
15ee691
added TODOs
ShreyesS Jul 24, 2025
c5108dd
centralize generation of test vectors, easy distribution to runtimes
ShreyesS Jul 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions TestVectorsAwsCryptographicMaterialProviders/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,3 +254,31 @@ test_decrypt_encrypt_vectors_rust:

test_decrypt_encrypt_vectors_go:
go -C runtimes/go/ImplementationFromDafny-go run ImplementationFromDafny.go decrypt --manifest-path=..

# Fuzz Testing Commands

# Generate fuzzed test vectors in the Python runtime directory (50 vectors)
test_generate_fuzz_vectors:
cp dafny/TestVectorsAwsCryptographicMaterialProviders/test/keys.json runtimes/python/
cd runtimes/python && python3 fuzz_generator.py --num-vectors 50
Copy link
Member

@rishav-karanjit rishav-karanjit Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

50 is too less. Given the current interop generates 1000+.

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can put fuzz_generator.py in a common directory where we put dafny code because we use this for every runtime.

Copy link
Contributor Author

@ShreyesS ShreyesS Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose 50 as a starting point. With the final CI, I'll ensure we can run tens of thousands of tests.

Also, I'll make a TODO for the common directory. I would like to discuss the logic of the workflow with you.

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose 50 as a starting point. With the final CI, I'll ensure we can run tens of thousands of tests.

Have a TODO for this. I want to set the variable such that it runs for atleast 6 hours

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I'll make a TODO for the common directory. I would like to discuss the logic of the workflow with you.

Sure but don't create TODO for every thing that it gets very difficult for you to merge to main in the end. Currently, all the stuffs in runtimes/<language> is used by that language itself like Go does not need anything in runtimes/python.

@echo "fuzzed test vectors generated in runtimes/python/"
Copy link
Member

@rishav-karanjit rishav-karanjit Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This echo will only add noise. When this command succeeds its clear it has generated

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is blocking too. We either need to remove this if it does not add any value. If it adds value then add it to all the existing make file commands.

Copy link
Contributor Author

@ShreyesS ShreyesS Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two commands build off of the existing commands, and they already have enough output to show if a procedure has executed successfully or not. However, I was building the CI for the first time, and when I ran into errors I needed to add those debugging statements to figure out where in each make command I was failing.

So I was thinking I add a TODO to remove before we merge with main, but for a few days, it would be helpful for me to track. Does that work?

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sure. If you have a TODO its fine but don't let all the TODO drown you. You goal is to have this running in main not in any other branch


# Run fuzz interoperability test from one language to another
# Usage: make test_fuzz_interop ENCRYPT_LANG=go DECRYPT_LANG=python
test_fuzz_interop:
Copy link
Member

@rishav-karanjit rishav-karanjit Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interop test does not need this why do we need this for interoperability for fuzzed test?

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is blocking because we will be have to maintain two different process where we can have just one.

Copy link
Contributor Author

@ShreyesS ShreyesS Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name might be misleading, but essentially I need the setup and runtime implementations that are done in interop to run cross-runtime testing.

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this command, its hard to re-use the existing CI workflow.

Copy link
Member

@rishav-karanjit rishav-karanjit Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interop does generate with dafny code, encrypt and decrypt
Fuzz test does generate with code you wrote, encrypt and decrypt

But, Interop does not have make command like this but fuzz test does. So, either we need to have this makefile command and change how interop test works or change this to what interop test does unless introducing a new process has a reason.

@echo "Testing interoperability from $(ENCRYPT_LANG) to $(DECRYPT_LANG)"
$(MAKE) test_generate_fuzz_vectors
@if [ "$(ENCRYPT_LANG)" != "python" ]; then \
echo "Copying test vectors to $(ENCRYPT_LANG) runtime"; \
cp runtimes/python/manifest.json runtimes/$(ENCRYPT_LANG)/; \
cp runtimes/python/keys.json runtimes/$(ENCRYPT_LANG)/; \
fi
@echo "Encrypting with $(ENCRYPT_LANG) runtime"
$(MAKE) test_encrypt_vectors_$(ENCRYPT_LANG)
@if [ "$(ENCRYPT_LANG)" != "$(DECRYPT_LANG)" ]; then \
echo "Copying encrypted manifest and keys to $(DECRYPT_LANG) runtime"; \
cp runtimes/$(ENCRYPT_LANG)/manifest.json runtimes/$(DECRYPT_LANG)/; \
cp runtimes/$(ENCRYPT_LANG)/keys.json runtimes/$(DECRYPT_LANG)/; \
fi
@echo "Decrypting with $(DECRYPT_LANG) runtime"
$(MAKE) test_decrypt_encrypt_vectors_$(DECRYPT_LANG)
Loading