-
Notifications
You must be signed in to change notification settings - Fork 1
Description
What
The Python guest Justfile has an optimization that avoids downloading the CPython WASM build and the WASI SDK again if they are already downloaded:
datafusion-udf-wasm/guests/python/Justfile
Lines 33 to 37 in a5b9a1d
| # skip if already downloaded | |
| if [ -d python-sdk ]; then | |
| echo "python sdk already present" | |
| exit 0 | |
| fi |
datafusion-udf-wasm/guests/python/Justfile
Lines 90 to 94 in a5b9a1d
| # skip if already downloaded | |
| if [ -d wasi-sysroot ]; then | |
| echo "wasi sdk already present" | |
| exit 0 | |
| fi |
That should not only check presence but also the checksum of the already-downloaded files and should re-download if the checksums mismatch.
Why
Whenever we upgrade the CPython version or the WASI SDK, developers will locally still have an outdated copy and without just clean they will not use the new versions. This isn't an issue for our CI though.
Note that this is NOT a security issue since the checksum was validated when the data was originally downloaded, it's just that it doesn't automatically upgrade the version of a local development setup.
How
In the if-branch, do something similar to these checks
datafusion-udf-wasm/guests/python/Justfile
Lines 59 to 60 in a5b9a1d
| echo "{{SHA256_PYTHON_SDK}} python-sdk.zip" | sha256sum -c | |
| echo "{{SHA256_PYTHON_SDK_BUILD}} build-python-sdk.zip" | sha256sum -c |
datafusion-udf-wasm/guests/python/Justfile
Line 106 in a5b9a1d
| echo "{{SHA256_WASI_SDK_SYSROOT}} wasi-sysroot.tar.gz" | sha256sum -c |
and if they fail, wipe the downloaded data and re-download.