Skip to content

refactor(all): modernize codebase with python 3.11 features #1812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jul 1, 2025

Conversation

danceratopz
Copy link
Member

🗒️ Description

This PR modernizes the EEST codebase to leverage Python 3.11+ language features (and some neglected 3.10 features) for improved performance, readability, type safety, and security.

🔧 Python 3.11+ Specific Features

1. TypeVar → Self Pattern (7 instances)

Benefits: Eliminates boilerplate TypeVar declarations, improves type safety, and makes method signatures cleaner and more intuitive.

Replace TypeVar bound patterns with Self type for cleaner method signatures:

# Before
T = TypeVar("T", bound="BaseTest")
def from_test(cls: Type[T], **kwargs) -> T:

# After  
def from_test(cls: Type[Self], **kwargs) -> Self:

2. String removeprefix/removesuffix (6 instances)

Benefits: More readable and explicit about intent, slightly more performant (avoids len() calculations), and reduces potential off-by-one errors in manual slicing.

Replace manual string slicing with explicit prefix/suffix removal:

# Before
if pattern.startswith("collectonly:"):
    return pattern[len("collectonly:"):]

# After
if pattern.startswith("collectonly:"):
    return pattern.removeprefix("collectonly:")

3. dataclass slots=True Optimization (3 instances)

Benefits: Reduces memory footprint (up to 40% less memory per instance), faster attribute access, and prevents accidental dynamic attribute assignment.

Add memory-efficient slots to high-frequency dataclasses:

# Before
@dataclass
class Trie(Generic[K, V]):

# After
@dataclass(slots=True) 
class Trie(Generic[K, V]):

4. Enum → StrEnum (4 enum classes)

Benefits: Better type safety for string-based enums, cleaner string conversion without .value, and more explicit semantics for enums that represent string constants.

Convert string-based enums for better type safety:

# Before
class ExecutionContext(Enum):
    BLOCK = "Block"

# After
class ExecutionContext(StrEnum):
    BLOCK = "Block"

5. LiteralString for Security (3 critical areas)

Benefits: Prevents command injection vulnerabilities, enhances static analysis for security-sensitive code, and provides compile-time validation of subprocess arguments.

Secure subprocess command construction with LiteralString validation:

# Before
check_cmd = ["docker", "exec", container_id, "test", "-f", container_path]
result = subprocess.run(check_cmd, capture_output=True)

# After
def safe_docker_exec(container_id: str, command: LiteralString, *args: LiteralString) -> list[str]:
    return ["docker", "exec", container_id, command] + list(args)

check_cmd = safe_docker_exec(container_id, "test", "-f", container_path)
result = subprocess.run(check_cmd, capture_output=True)

🔄 Other Modern Python Improvements Applied

isinstance → match-case (2 functions, Python 3.10+)

Benefits: More readable pattern matching, better performance for complex type checking, and enables exhaustiveness checking for better maintainability.

Convert type checking chains to pattern matching:

# Before
if isinstance(node, LeafNode):
    # handle LeafNode
elif isinstance(node, ExtensionNode):
    # handle ExtensionNode

# After
match node:
    case LeafNode():
        # handle LeafNode
    case ExtensionNode():
        # handle ExtensionNode

Union type syntax modernization (25+ instances, Python 3.10+)

Benefits: More concise and readable type annotations, reduced import dependencies, and more intuitive union syntax.

Convert Union types to modern pipe syntax:

# Before
from typing import Union
BlockNumberType = Union[int, Literal["latest", "earliest", "pending"]]

# After
BlockNumberType = int | Literal["latest", "earliest", "pending"]

📊 Summary Stats

  • Files Modified: 12+ files across the codebase
  • Type Annotations: 30+ modern type hints applied
  • Security Patterns: 3 critical subprocess areas secured (Docker, Solc, T8N)
  • Performance Optimizations: 3 dataclass slots applied
  • String Operations: 6 removeprefix/removesuffix modernizations
  • Pattern Matching: 2 isinstance chains converted to match-case
  • Enum Improvements: 4 string-based enums converted to StrEnum
  • Union Syntax: 25+ Union type annotations modernized

🔗 Related Issues or PRs

Follow-up to:

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

@danceratopz danceratopz added scope:tests Scope: Changes EL client test cases in `./tests` type:refactor Type: Refactor scope:fw Scope: Framework (evm|tools|forks|pytest) labels Jun 27, 2025
Replace  bound patterns with Python 3.11+  type for cleaner,
more idiomatic type hints in method returns. Keeps
for Generic classes that need type constraints. This improves type safety
and reduces boilerplate code.
Replace manual string slicing patterns with Python 3.9+ removeprefix/removesuffix
methods for cleaner, more readable code that explicitly shows intent to remove
specific prefixes or suffixes.
Add slots to dataclasses that don't have class variable access conflicts.
Optimizes memory usage and attribute access speed for Trie, Case, and TestInfo
classes that are frequently instantiated during test execution.
Replace isinstance chains with Python 3.10+ match-case syntax for better
readability and more idiomatic pattern matching in trie encoding functions.
Replace Enum classes with string values with Python 3.11+ StrEnum for better
type safety, cleaner string conversion, and more explicit string enumeration
semantics in test configuration and execution contexts.
Replace `Union[X, Y]` with `X | Y` syntax for cleaner type annotations.
Applied to type aliases, function signatures, and complex type definitions
for improved readability and reduced import dependencies.

Python 3.10+ feature - more concise and intuitive union syntax.
Additional conversions of `Union[X, Y]` to `X  < /dev/null |  Y` in logging
and test modules for consistent modern type annotation syntax.
- Add `safe_docker_exec` wrapper for Docker command construction
- Implement `safe_solc_command` for compiler argument validation
- Add `safe_t8n_args` for transition tool parameter security
- Use `LiteralString` type hints to prevent command injection
- Validate file paths, EVM versions, and numeric parameters
@marioevz marioevz force-pushed the refactor/modernize-with-python311 branch from 337e02e to 6b24bcb Compare June 27, 2025 19:33
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks!

I've added the coverage_missed and rebased, so CI should pass now.

@marioevz marioevz force-pushed the refactor/modernize-with-python311 branch from 6b24bcb to 62b8269 Compare June 27, 2025 19:38
Remove safe_docker_exec and safe_docker_command functions that added
verbosity without providing meaningful security validation. These
wrappers were flagged in PR review as unnecessary noise.

Revert to inline Docker command construction for better readability
while preserving valuable security wrappers in other modules.
@danceratopz danceratopz merged commit c9b54e3 into main Jul 1, 2025
14 checks passed
@danceratopz danceratopz deleted the refactor/modernize-with-python311 branch July 1, 2025 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:fw Scope: Framework (evm|tools|forks|pytest) scope:tests Scope: Changes EL client test cases in `./tests` type:refactor Type: Refactor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants