Store workflow/step dependencies in database and update API

[lafriks]

Needed for frontend to be able to draw dependency graph for pipeline (workflows and steps)

I wanted to split frontend PR from backend as it still needs to be discussed on how to better implement UI for this but that should not be blocking to merge backend part

[codecov]

Codecov Report

❌ Patch coverage is 64.44444% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 31.66%. Comparing base (853f7a5) to head (dfd1b8e).

Files with missing lines	Patch %	Lines
server/store/datastore/workflow.go	52.94%	12 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6130      +/-   ##
==========================================
+ Coverage   31.65%   31.66%   +0.01%     
==========================================
  Files         421      421              
  Lines       28430    28464      +34     
==========================================
+ Hits         8999     9014      +15     
- Misses      18609    18626      +17     
- Partials      822      824       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[6543]

why do we need it in backend?

[lafriks]

because it goes through compiler to backend types and then from backend types it's assigned to model type

[6543]

what?!

[6543]

why not name it DependsOn like the rest of the codebase?!?

[lafriks]

Because you need to resolve string DependsOn (step names) to IDs but they are only known when all records are inserted, model has both DopendsOnNames -> string array and DependsOn -> int64 array that will be populated after saving all workflows and steps to database from DependsOnNames and then we can update depends_on column with depends_on step ID's

[6543]

can we not do it DependsOn -> string array and DependsOnIDs -> int64 array to be consistent with the rest?

[anbraten]

In my previous tests of showing a DAG in the UI, I have used the yaml configs (already exposed via api) to generate dags.

[lafriks]

In my previous tests of showing a DAG in the UI, I have used the yaml configs (already exposed via api) to generate dags.

Yes but it has many problems:

• There can be matrix pipelines with matrix values in step names

matrix:
  ENV:
    - staging
    - production

steps:
  - name: validate-config-${ENV}
    image: alpine
    commands:
      - echo "validate-config-${ENV}"
    depends_on: []

  - name: fetch-secrets-${ENV}
    image: alpine
    commands:
      - echo "fetch-secrets-${ENV}"
    depends_on: []

  - name: build-manifest-${ENV}
    image: alpine
    commands:
      - echo "build-manifest-${ENV}"
    depends_on:
      - validate-config-${ENV}
      - fetch-secrets-${ENV}

so in fronted you would need to expand them and guess workflow matrix axis

• Also similar problem would be with dependencies declared as yaml variables

Saving them to database after compiler is already done correct computations is imho best way to do this

[6543]

because it goes through compiler to backend types and then from backend types it's assigned to model type

this is a open todo to be remove by #3967 ... so we should not introduce a new issue