Skip to content

Conversation

ArneTR
Copy link
Member

@ArneTR ArneTR commented Aug 7, 2025

Micro Phases are a concept in GMT to dynamically create phases from stdout of running processes.

Example: You have such a usage scenario:

...
flow:
    - name: Run Playwright Tests
       container: playwright
       commands:
       - type: console
          command: node playwright.js
...

The Playwright browser might run multiple tests but in GMT they would only show up as one phase.
However by reading and parsing the stdout alongside timestamps set there GMT can now expand them to separate phases.

This works by using a regex on the stdout and capturing three groups:

  • phase_name
  • start_time
  • end_time

A working stdout can for instance look something like this:

# node playwright.js
...
Starting (Test No. 1) at 179874192798921 ;
Ending (Test No. 1) at  179874192798929 ;
Starting (Test No. 2) ....
....

The flow would then updated with a regex like so:

```yaml
...
flow:
    - name: Run Playwright Tests
       container: playwright
       commands:
       - type: console
          command: node playwright.js
          log-stdout: true
          sub-phase-expansion-pattern: '^\s*Starting \((?P<phase_name>.+)\) at (?P<start_time>\d+) ;$\n^\s*Ending \([^\)]+\) at (?P<end_time>\d+) ;$'
...

The implementation atm requires strong regex knowledge and, if not already present, instrumentation of the stdout to contain the needed timestamps and event names that shall be used as phase names.

Example of how phase names are then later expanded

Screenshot 2025-08-21 at 8 37 23 AM

Open Problems

  • The newly created phases are sub-phases of exisiting runtime-sub-phases. So technically sub-sub-phases (calling them Micro Phases for now). Which means they must be removed from accounting in phase_stats.py as currently the totals are off (double-counting)
  • The display can get quickly garbled if more than 10 sub-sub-phases are expanded. Maybe a new separate tab should be explored here
  • The Micro Phases are not started and ended by GMT directly but just mapped. Which means they cannot make use of the phase-padding feature. They will have hard cut-offs and suffer from possible undersampling issues
  • Micro Phases can overlap as the parsed stdout is currently not checked against that. It shall be reasoned if that is wanted behaviour or shall be an error

@ArneTR ArneTR changed the base branch from main to log-parsing-on-full-data August 7, 2025 05:29
Base automatically changed from log-parsing-on-full-data to main August 11, 2025 05:09
ArneTR added 3 commits August 13, 2025 09:25
* main:
  Bump actions/create-github-app-token in /.github/workflows (#1284)
  Bump pylint from 3.3.7 to 3.3.8 (#1285)
  Updated echarts vor v.6.0 (#1282)
  Bump python from 3.13.5-slim-bookworm to 3.13.6-slim-bookworm in /docker (#1286)
  Bump deepdiff from 8.5.0 to 8.6.0 (#1283)
  Log parsing on full data (#1276)
  Bump redis from 6.3.0 to 6.4.0 (#1281)
  Added Gemini PR Review
  measurement_flow_process_duration is set in any case. Supplying a timeout of None will lead to no timeout
  Detect systemd cgroup path using cgroup name (instead of container id) (#1270)
  Bump redis from 6.2.0 to 6.3.0 (#1275)
* main:
  Bump requests from 2.32.4 to 2.32.5 (#1298)
  Fix output of container logs ("print-logs" flag) +  Reuse ScenarioRunner object for all files/iterations (#1290)
  Bump actions/checkout from 4 to 5 in /.github/workflows (#1295)
  Bump python from 3.13.6-slim-bookworm to 3.13.7-slim-bookworm in /docker (#1293)
  Bump actions/create-github-app-token in /.github/workflows (#1294)
  Updated Gemini Actions Workflow to latest version from upstream
  Update gemini-pr-review.yml - Only run when actively triggered via comment
  Bump orjson from 3.11.1 to 3.11.2 (#1288)
@ArneTR
Copy link
Member Author

ArneTR commented Aug 21, 2025

This PR was also used to evaluate integrating GitHub Actions into GMT. The corresponding repo describing why we believe this is currently only possible with a very high effort integration is at : https://github.com/green-coding-solutions/eco-ci-runner-inside-gmt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant