chore(python): create custom Makefile commands for fuzzing test vectors #1634
Conversation
|
|
||
| # Run fuzz interoperability test from one language to another | ||
| # Usage: make test_fuzz_interop ENCRYPT_LANG=go DECRYPT_LANG=python | ||
| test_fuzz_interop: |
There was a problem hiding this comment.
Interop test does not need this why do we need this for interoperability for fuzzed test?
There was a problem hiding this comment.
This is blocking because we will be have to maintain two different process where we can have just one.
There was a problem hiding this comment.
The name might be misleading, but essentially I need the setup and runtime implementations that are done in interop to run cross-runtime testing.
There was a problem hiding this comment.
With this command, its hard to re-use the existing CI workflow.
There was a problem hiding this comment.
Interop does generate with dafny code, encrypt and decrypt
Fuzz test does generate with code you wrote, encrypt and decrypt
But, Interop does not have make command like this but fuzz test does. So, either we need to have this makefile command and change how interop test works or change this to what interop test does unless introducing a new process has a reason.
| test_generate_fuzz_vectors: | ||
| cp dafny/TestVectorsAwsCryptographicMaterialProviders/test/keys.json runtimes/python/ | ||
| cd runtimes/python && python3 fuzz_generator.py --num-vectors 50 | ||
| @echo "fuzzed test vectors generated in runtimes/python/" |
There was a problem hiding this comment.
This echo will only add noise. When this command succeeds its clear it has generated
There was a problem hiding this comment.
This is blocking too. We either need to remove this if it does not add any value. If it adds value then add it to all the existing make file commands.
There was a problem hiding this comment.
These two commands build off of the existing commands, and they already have enough output to show if a procedure has executed successfully or not. However, I was building the CI for the first time, and when I ran into errors I needed to add those debugging statements to figure out where in each make command I was failing.
So I was thinking I add a TODO to remove before we merge with main, but for a few days, it would be helpful for me to track. Does that work?
There was a problem hiding this comment.
Yeah sure. If you have a TODO its fine but don't let all the TODO drown you. You goal is to have this running in main not in any other branch
| # Generate fuzzed test vectors in the Python runtime directory (50 vectors) | ||
| test_generate_fuzz_vectors: | ||
| cp dafny/TestVectorsAwsCryptographicMaterialProviders/test/keys.json runtimes/python/ | ||
| cd runtimes/python && python3 fuzz_generator.py --num-vectors 50 |
There was a problem hiding this comment.
50 is too less. Given the current interop generates 1000+.
There was a problem hiding this comment.
We can put fuzz_generator.py in a common directory where we put dafny code because we use this for every runtime.
There was a problem hiding this comment.
I chose 50 as a starting point. With the final CI, I'll ensure we can run tens of thousands of tests.
Also, I'll make a TODO for the common directory. I would like to discuss the logic of the workflow with you.
There was a problem hiding this comment.
I chose 50 as a starting point. With the final CI, I'll ensure we can run tens of thousands of tests.
Have a TODO for this. I want to set the variable such that it runs for atleast 6 hours
There was a problem hiding this comment.
Also, I'll make a TODO for the common directory. I would like to discuss the logic of the workflow with you.
Sure but don't create TODO for every thing that it gets very difficult for you to merge to main in the end. Currently, all the stuffs in runtimes/<language> is used by that language itself like Go does not need anything in runtimes/python.
Co-authored-by: Rishav karanjit <[email protected]>
2 participants
Changes
WIP
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.