GitHub Actions for Linting and Compiling Bytecode

[DavesCodeMusings]

GitHub Actions for Linting and Compiling Bytecode

I recently figured out how to use GitHub Actions to automatically lint my code and compile it to a bytecode (.mpy) file. If you want to do something similar in your repository, here it is in a nutshell. An explanation as to why and how follows the example.

name: Lint and compile bytecode
on: push
env:
  SRC_FILES: '*.py'
permissions:
  contents: write
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
      - name: Install ruff and mpy-cross
        run: pip3 install mpy-cross ruff
      - name: Check lint and format
        run: python3 -m ruff check $SRC_FILES
      - name: Compile bytecode
        run: python3 -m mpy_cross $SRC_FILES
      - name: Commit changes
        run: |
          if [ $(git diff | wc -l) -gt 0 ]; then
            git config --global user.name "$GITHUB_REPOSITORY_OWNER"
            git config --global user.email "[email protected]"
            git commit -am "mpy_cross $SRC_FILES"
            git push
          fi

Figure 1: Contents of my repository's .github/workflows/main.yml

Why?

Previously, I was doing all my MicroPython code linting using the usual VS Code plug-ins for Python. When I wanted to pre-compile into bytecode, I used the mpy-cross Python module at the command-line.

These steps had to be done every time before pushing code to my repository. It works, but after about the eighteenth time, it gets rather tedious. I knew other folks were using GitHub Actions to do this automatically, but I had a tough time finding the relevant information to do what I wanted in a simple, direct way.

How?

I'm rather new to GitHub Actions, so I approached the problem with the goal of proving I could push something, anything, to a repository using a workflow. I started with an example I found on StackOverflow.

name: Push commit
on: push
permissions:
  contents: write
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Create test file
        run: date +%s > test.txt
      - name: Commit test file
        run: |
          git config --global user.name 'YourName'
          git config --global user.email '[email protected]'
          git commit -am "Test file"
          git push

Figure 2: Example of generating a test file and pushing to a repository.

The first error I encountered was git complaining the file test.txt needed to be added before it could be commited and pushed. Not a big deal, because I knew the .mpy file I eventually wanted to create already existed in my repo. I just created an empty test.txt alongside it and moved on with my life.

The workflow was successful after that and I saw my test.txt with an integer date-time stamp in it.

Linting with Ruff

I recently added the Ruff plug-in to my VS Code and I've been very happy wth it. So I decided a good next step would be trying do code linting in my workflow. It wasn't too difficult. The workflow is shown below.

name: Lint and format
on: push
env:
  SRC_FILES: '*.py'
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
      - name: Install ruff
        run: pip3 install ruff
      - name: Check lint and format
        run: python3 -m ruff check $SRC_FILES

Figure 3: Linting with Ruff

A couple things to notice here are:

1. I added the environment variable SRC_FILES.
2. I removed write permissions.

The SRC_FILES variable makes it easier to configure which files are getting linted. I could have used *.py in the last line, python3 -m ruff check, but the flexibility of the variable will come in handy as more workflow steps are added.

The write permissions are not needed, because I'm not creating or altering any files with this workflow. All I'm doing is checking the code and letting the workflow fail if problems are found.

There's also a ruff fix command to correct the problems it finds, but I'm already linting with Ruff in VS Code and I'd rather fix the problems there.

Compiling Bytecode

Now that I know I can lint Python code, and I can generate a test.txt file and push it to my repository, the next step is to put everything together while generating an .mpy file instead of test.txt. Here's how:

name: Lint and compile bytecode
on: push
env:
  SRC_FILE: mymodule.py
permissions:
  contents: write
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
      - name: Install ruff and mpy-cross
        run: pip3 install mpy-cross ruff
      - name: Check lint and format
        run: python3 -m ruff check $SRC_FILE
      - name: Compile bytecode
        run: python3 -m mpy_cross $SRC_FILE
      - name: Commit changes
        run: |
          if [ $(git diff | wc -l) -gt 0 ]; then
            git config --global user.name "$GITHUB_REPOSITORY_OWNER"
            git config --global user.email "[email protected]"
            git commit -am "mpy_cross $SRC_FILE"
            git push
          fi

Figure 4: Linting and compiling Bytecode

The big changes here are:

1. Replacing the date example with the command to compile bytecode.
2. Checking for changes before committing and pushing.
3. Using a variable instead of YourName in the git config.

The command to compile bytecode is not really any different than what I had been doing manually before. Though in the workflow, I'm using the SRC_FILE variable instead of a hardcoded filename.

Checking for changes before committing is accomplished with the conditional if [ $(git diff | wc -l) -gt 0 ];

This is needed because the workflow will fail when git complains there are no changed files. This error will happen whenever a file other than $SRC_FILE is pushed. The workflow will trigger because of the push action, but there's no change in the bytecode if the pushed file is not the python file indicated by SRC_FILE. No changes causes the error and git diff is used to list changes.

The last bit is minor. Instead of YourName for the git config user.name, the built-in workflow variable GITHUB_REPOSITORY_OWNER will substitute the repository namespace instead of requiring a hardcoded value. It's not super important, but it makes things easier for other people to borrow the code without a lot of search and replace customizing.

Next Steps

My goal was a simple workflow. You can get fancier if you want. Some good references are:

• https://docs.github.com/en/actions
• https://docs.astral.sh/ruff/
• https://dev.to/ken_mwaura1/automate-python-linting-and-code-style-enforcement-with-ruff-and-github-actions-2kk1

[Josverl]

AFAIK Linting with ruff assumes Cpython code.
I'm not sure if ruff can take type stubs such as the micropython-stubs into account. If not you can add pyright which certainly does handle that to help improve quality further.

[andrewleech]

@Josverl I use ruff with your stubs quite effectively... although these are the same slightly modified copy I'm working on for mypy support.

pip install -U micropython-stm32-stubs --target ./tools/typings --no-user

This is my pyproject.toml

[tool.ruff]
# Enable pycodestyle (`E`) and Pyflakes (`F`) codes by default.
select = ["E", "F"]
ignore = []

# Allow autofix for all enabled rules (when `--fix`) is provided.
fixable = ["A", "B", "C", "D", "E", "F", "G", "I", "N", "Q", "S", "T", "W", "ANN", "ARG", "BLE", "COM", "DJ", "DTZ", "EM", "ERA", "EXE", "FBT", "ICN", "INP", "ISC", "NPY", "PD", "PGH", "PIE", "PL", "PT", "PTH", "PYI", "RET", "RSE", "RUF", "SIM", "SLF", "TCH", "TID", "TRY", "UP", "YTT"]
unfixable = []

builtins = ["const", "ptr8"]

# Exclude a variety of commonly ignored directories.
exclude = [
    ".bzr",
    ".direnv",
    ".eggs",
    ".git",
    ".hg",
    ".mypy_cache",
    ".nox",
    ".pants.d",
    ".pytype",
    ".ruff_cache",
    ".svn",
    ".tox",
    ".venv",
    "__pypackages__",
    "_build",
    "buck-out",
    "build",
    "dist",
    "node_modules",
    "venv",
    "tools/typings",
]

# Same as our configuration of Black.
line-length = 99

# Allow unused variables when underscore-prefixed.
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"

# Assume Python 3.10.
target-version = "py310"

# Avoid automatically removing unused imports in __init__.py files.
ignore-init-module-imports = true

typing-modules = ["micropython-stm32-stubs"]

[tool.ruff.mccabe]
# Unlike Flake8, default to a complexity level of 10.
max-complexity = 10

[tool.ruff.per-file-ignores]
"manifest.py" = ["F821"]  # "Undefined name" for eg. freeze, include, etc.

[tool.ruff.format]
exclude = ["src/lib", "src/micropython"]

and .pre-commit-config.yaml

files: "^src/firmware"

repos:
  - repo: https://github.com/charliermarsh/ruff-pre-commit
    # Ruff version.
    rev: "v0.1.2"
    hooks:
      - id: ruff
        args: ["--fix"]
        additional_dependencies:
          ["types-PyYAML", "micropython-stm32-stubs"]
      - id: ruff-format

[DavesCodeMusings]

I think I learned about Ruff from this group. There was a post a while back about converting to Ruff from something else (maybe Black?)