Check could not be completed or gitStream automation is stuck forever, rules are occasionally processes

[artem-shorokhov]

Hi!

We have 3 static and 362 dynamic gitStream rules generated in the following way:

gitStream configuration

# -*- mode: yaml -*-

manifest:
  version: 1.0

automations:
  add_private_topics_label:
    if:
      - {{ files | match(regex=r/private-topics/) | some }}
    run:
      - action: add-label@v1
        args:
          label: "private-topics"
          color: "E94637"

  files_created:
    if:
      - {{ source.diff.files | match(attr='original_file', regex=r/^$/) | some }}
    run:
      - action: add-label@v1
        args:
          label: 'files-created'
          color: '#36A853'
      - action: add-reviewers@v1
        args:
          team_reviewers: [team]
      - action: require-reviewers@v1
        args:
          reviewers: [org/team]

  files_deleted:
    if:
      - {{ source.diff.files | match(attr='new_file', term='/dev/null') | some }}
    run:
      - action: add-label@v1
        args:
          label: 'files-deleted'
          color: '#EB3324'
      - action: add-reviewers@v1
        args:
          team_reviewers: [team]
      - action: require-reviewers@v1
        args:
          reviewers: [org/team]

  {% for client in clients %}
  approve_owned_nonprod_changes_{{ client.name | lower }}:
    if:
      - {{ ( files | match(regex=client.regex) | every ) and ( files | match(regex=r/\/production\//) | nope ) and ( pr.author_teams | intersection(list=client.teams) | some ) }}
    run:
      - action: add-label@v1
        args:
          label: {{ client.name }}
      - action: approve@v1
      - action: add-label@v1
        args:
          label: "auto-approved"
          color: "36A853"

  require_review_{{ client.name | lower }}:
    if:
      - {{ not ( ( files | match(regex=client.regex) | every ) and ( files | match(regex=r/\/production\//) | nope ) and ( pr.author_teams | intersection(list=client.teams) | some ) ) }}
      - {{ files | match(regex=client.regex) | some }}
    run:
      - action: add-label@v1
        args:
          label: {{ client.name }}
      - action: add-reviewers@v1
        args:
          team_reviewers: [{{ client.teams | join(",") | lower | replace(" - ", "-") | replace(" ", "-") }}]
      - action: require-reviewers@v1
        args:
          reviewers: [{{ "org/" ~ (client.teams | join(",org/")) | lower | replace(" - ", "-") | replace(" ", "-") }}]
  {% endfor %}

clients:
- name: "TEAM001"
  regex: "r/^TEAM001\//"
  teams:
    - "Team 001"

...

- name: "TEAM181"
  regex: "r/^TEAM181\//"
  teams:
    - "Team 181"

Rules are being successfully evaluated within ~20s.

But rule processing is stuck forever. However, sometimes rules are being processed.
This started to happen like 1-2 months ago.

I do not think that we are hitting automation limits here, as there was no warning in PR comments and total number of PRs is around 150 per month.

[vim-zz]

Hi, can you provide PR URL? or your GitHub organization/owner name? if its sensitive, you can email it to product at linearb.io

[artem-shorokhov]

Hi, can you provide PR URL? or your GitHub organization/owner name? if its sensitive, you can email it to product at linearb.io

Hi! Sure. Here it is.
https://github.com/ingka-group-digital/event-mesh-clients/pull/8099

[vim-zz]

Thanks for the details. Based on our analysis, the timeout happens when a single run produces too many effects from the automation rules. We’re going to address this. In the meantime, please reduce the number of actions/effects (or scope them down) and add them back gradually until it’s stable.

[vim-zz]

In the mean time, we have increased the timeout limit for you so you wont hit it that often. We plan to address this issue properly later.

[artem-shorokhov]

Thanks for the details. Based on our analysis, the timeout happens when a single run produces too many effects from the automation rules. We’re going to address this. In the meantime, please reduce the number of actions/effects (or scope them down) and add them back gradually until it’s stable.

We have 2 rules per tenant. Was able to offboard 4 inactive tenants today, thus -8 rules. Will try to clean up further.

[artem-shorokhov]

In the mean time, we have increased the timeout limit for you so you wont hit it that often. We plan to address this issue properly later.

Thanks a lot! Today, around 90% PRs had no issues.