bug(tooling): wrong solc version might be used for static tests

[felix314159]

What is the context of this problem?

When trying to fill certain static tests using e.g. uv run fill -k "MCOPY_memory_expansion_costFiller" --fill-static-tests --evm-bin=/home/$USER/Documents/evmone/build/bin/evmone-t8n -vv -s a different solc version might be used than intended. This matters because I can fill these tests with solc 0.8.24 but it fails to fill with solc 0.8.29 (there are good reasons for this but this is not the scope of this issue).

How to reproduce the issue?

Put the solc 0.8.29 binary to .venv/bin/solc and ensure it has +x permission. Now ensure your config is to use 0.8.24 with uv run solc-select use 0.8.24 (you might have to install it first if you have not done so already). If you run uv run solc-select versions it should tell you: 0.8.24 (current, ...). Now run the fill command shown above. It will log that solc 0.8.24 is used (both in terminal and in html report) but in fact it uses the 0.8.29 binary, this can be detected e.g. by modifying './src/ethereum_test_specs/static_state/common/compile_yul.py' with sth like

    result2 = subprocess.run(
        ["solc", "--version"],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        check=True,
    )

    print(f"\nsolc version check: {result2.stdout}")

What should the fix be?

solc-select must be enforced, it should not silently choose the solc binary from .venv/bin/. The logging must be consistent with what is actually being used.

[danceratopz]

This must have been very annoying! But this seems quite an obscure case, if I understand correctly? As you manually have to overwrite the solc binary in the venv? (Btw, the user can instead use the --solc-bin flag to specify a different version).

I'm surprised that solc-select simply assumes the binary it finds there is the correct version. How about, we add our own check to the end of this block to verify that the binary is actually at the version reported by solc-select and exit with error if there is an unexpected version there?

execution-spec-tests/src/pytest_plugins/solc/solc.py

Lines 58 to 76 in 624f60b

	else:
	# if no solc binary is specified, use solc-select
	solc_version = solc_version or DEFAULT_SOLC_VERSION
	try:
	version, _ = solc_select.current_version()
	except ArgumentTypeError:
	version = None
	if version != solc_version:
	if config.getoption("verbose") > 0:
	print(f"Setting solc version {solc_version} via solc-select...")
	try:
	solc_select.switch_global_version(solc_version, always_install=True)
	except Exception as e:
	message = f"Failed to install solc version {solc_version}: {e}. "
	if isinstance(e, ArgumentTypeError):
	message += "\nList available versions using `uv run solc-select install`."
	pytest.exit(message, returncode=pytest.ExitCode.USAGE_ERROR)
	solc_version_semver = Version.parse(solc_version)
	config.option.solc_bin = which("solc") # save for fixture

[felix314159]

This must have been very annoying! But this seems quite an obscure case, if I understand correctly? As you manually have to overwrite the solc binary in the venv? (Btw, the user can instead use the --solc-bin flag to specify a different version).

I'm surprised that solc-select simply assumes the binary it finds there is the correct version. How about, we add our own check to the end of this block to verify that the binary is actually at the version reported by solc-select and exit with error if there is an unexpected version there?

execution-spec-tests/src/pytest_plugins/solc/solc.py

Lines 58 to 76 in 624f60b
else:
# if no solc binary is specified, use solc-select
solc_version = solc_version or DEFAULT_SOLC_VERSION
try:
version, _ = solc_select.current_version()
except ArgumentTypeError:
version = None
if version != solc_version:
if config.getoption("verbose") > 0:
print(f"Setting solc version {solc_version} via solc-select...")
try:
solc_select.switch_global_version(solc_version, always_install=True)
except Exception as e:
message = f"Failed to install solc version {solc_version}: {e}. "
if isinstance(e, ArgumentTypeError):
message += "\nList available versions using uv run solc-select install."
pytest.exit(message, returncode=pytest.ExitCode.USAGE_ERROR)
solc_version_semver = Version.parse(solc_version)
config.option.solc_bin = which("solc") # save for fixture

That's a great idea. Things might be working as expected btw, I just learned that the file .venv/bin/solc is not even a solc binary but instead a Python script without a file extension that contains:

import sys
from solc_select.__main__ import solc
if __name__ == "__main__":
    if sys.argv[0].endswith("-script.pyw"):
        sys.argv[0] = sys.argv[0][:-11]
    elif sys.argv[0].endswith(".exe"):
        sys.argv[0] = sys.argv[0][:-4]
    sys.exit(solc())

So having an additional check can't hurt but the problem was on my end - thinking I can replace this with another binary. And for completeness: The only reason I tried to replace the solc file with an actual solc binary in the first place was an unrelated ARM linux issue with solc-select.

[felix314159]

With these commits I am happy how things are: solc-select will not try to install x86 solc binaries on arm linux and it will also refuse to fill if actual solc version does not match specified solc version

[danceratopz]

Thanks for this, added a couple of comments to f180ce6.

[danceratopz]

@felix314159 feel free to make a PR :)