CI for mesa #18130

dkijania · 2025-11-18T13:29:42Z

Adding CI for mesa :

building noble and bookwom for devnet and lightnet profile for amd64 and arm64 architectures
tagged applicable jobs with Mesa tag - here my goal was to not run all other building jobs like we do for devnet/mainnet. Currently Mesa is at bugs exploration path, so we don't need to support every configuration. This is not needed now. But i want to use this branch as new method of filtering jobs per mainline branch. We need to bring back those deleted jobs anyhow
regenerated archive static test data for mesa
cherry picked [CI][Test] add wait for archive node bootstrap #18159 to stability

dkijania · 2025-11-18T13:30:07Z

!ci-build-me

dkijania · 2025-11-18T13:42:44Z

!ci-build-me

dkijania · 2025-11-19T13:25:05Z

!ci-build-me

dkijania · 2025-11-21T22:09:12Z

!ci-build-me

dkijania · 2025-11-23T12:51:20Z

!ci-build-me

dkijania · 2025-11-23T16:41:52Z

!ci-build-me

dkijania · 2025-11-23T21:19:17Z

!ci-build-me

dkijania · 2025-11-24T17:06:40Z

!ci-build-me

dkijania · 2025-11-24T18:31:42Z

!ci-build-me

dkijania · 2025-11-24T19:22:20Z

Nightly: https://buildkite.com/o-1-labs-2/mina-end-to-end-nightlies/builds/4016

dkijania · 2025-11-24T22:47:51Z

!ci-docker-me codename=bookworm arch=amd64 profile=devnet

dkijania · 2025-11-24T22:48:20Z

!ci-docker-me codename=bookworm arch=amd64 profile=lightnet

dkijania · 2025-11-24T22:48:30Z

!ci-docker-me codename=bookworm arch=arm64 profile=devnet

dkijania · 2025-11-24T22:48:36Z

!ci-docker-me codename=bookworm arch=amd64 profile=lightnet

dkijania · 2025-11-24T23:22:17Z

!ci-docker-me codename=bookworm arch=amd64 profile=lightnet

dkijania · 2025-11-24T23:22:54Z

!ci-docker-me codename=bookworm arch=armd4 profile=lightnet

dkijania · 2025-11-24T23:23:01Z

!ci-docker-me codename=bookworm arch=amd64 profile=devnet

dkijania · 2025-11-24T23:23:08Z

!ci-docker-me codename=bookworm arch=arm64 profile=lightnet

glyh · 2025-11-25T06:40:41Z

We will merge mesa back to develop in the end I assume. There's some destructive changes in this repo. What's the plan to deal with them in the future?

dkijania · 2025-11-25T06:51:59Z

@glyh the plan is to work on dhall filtering and make it common for all branches,before merging to develop. So this branch should only contains ports from compatible for patching issues we are having right now.

I choose that strategy as issues on mesa CI are not directly connected with branch but with limitations of CI which were exposed here

glyh · 2025-11-25T06:57:49Z

So the files that would be removed when merging back to develop is expected, or it's a WIP?

glyh · 2025-11-25T06:59:22Z

buildkite/src/Jobs/Lint/ArchiveUpgrade.dhall

        , path = "Lint"
        , name = "ArchiveUpgrade"
-        , tags = [ PipelineTag.Type.Fast, PipelineTag.Type.Lint ]
+        , tags = [] : List PipelineTag.Type


Why is the tag cleaned here?

This test is failing, as develop branch has discrepancies between mesa. This is the quickest way to exclude this test from running

glyh · 2025-11-25T07:01:30Z

buildkite/src/Jobs/Lint/Xrefcheck.dhall

          [ PipelineTag.Type.Fast
          , PipelineTag.Type.Lint
          , PipelineTag.Type.Stable
+          , PipelineTag.Type.Mesa


I'm not a fan tagging all CI jobs with mesa here.

instead. We should probably have another kind of filtering rule that only applies for specific jobs base on branch name.

That filtering should be by default allow all tests, and for mainnet/devnet/mesa specific jobs we configure which branch should it run on with this additional filtering

Again, I'd advocate using a proper programming language instead of dhall because I can only see this mechanism grow more and more complicated and out of hand one day.

Generally, I don't mind using another language, (I wonder what proper language means :) ), but first we need to do some deep analysis which one and what we gain/loose. Such investment will be big, ROI a bit small IMHO and we don't have time to launch such campaign,now. It is easiest to say, "we need to rewrite it", but people are not realizing the cost and maintenance burden while doing the transitions. Also keep in mind that we are not using latest dhall version and they added some comparision/equality methods which should make it easier. This is still a PR review and i don't see how i can fix it here :) Let's move such discussion to issues maybe

What mechanism grow more complicated ? Adding another tag is utilizing mechanism which is already there and we are not complicating it

I'm not a fan tagging all CI jobs with mesa here.

instead. We should probably have another kind of filtering rule that only applies for specific jobs base on branch name.

That filtering should be by default allow all tests, and for mainnet/devnet/mesa specific jobs we configure which branch > should it run on with this additional filtering

This is my idea as well, but we need to add such changed from master up. However, i can see some issues with it anyhow. First, we would need to introduce mapping between branch and jobs. We can do it in couple ways.
a) Introducing another field which will defined branch
b) using tags as I'm preparing it above
c) using buildkite filters like .branch == compatible etc (if field)

The main issue though is how to determine what is the mainline branch (on scheduled nightly we should look at BUILDKITE_BRANCH field, in PR it should be BUILDKITE_PULL_REQUEST_BASE, if we are merging not to mainline branch, then we don't know and this is my main concern.

glyh · 2025-11-25T07:03:10Z

buildkite/src/Jobs/Release/MinaArtifactNobleMesaDevnet.dhall

-            , PipelineTag.Type.Noble
-            ]
+          , network = Network.Type.Mesa
+          , tags = [] : List PipelineTag.Type


why cleaned all tags here and couple of other places?

to disable such jobs.

glyh · 2025-11-25T07:05:26Z

buildkite/src/Pipeline/Tag.dhall

      | Noble
      | Focal
      | Jammy
+      | Mesa


Probably this should be replaced by some kind of

BranchsIn and BranchsExclude that carries a list of branches.

I see there's problem converting toNatural though, why do we even want to convert tags to natural?

we are using the old dhall version where equality is supported only between naturals, that's why we converting them

glyh · 2025-11-25T07:07:16Z

src/test/archive/sample_db/archive_db.sql

@@ -2,8 +2,8 @@
 -- PostgreSQL database dump


We're changing this in place, meaning it's going to be a disaster when we try to have both version work together.

Why? I had to regenerate sample data as it is so different from develop/compatible that all archive related tests are failing. If we merge mesa to develop this data should be merged, replacing the one in develop

glyh

I'm not a fan of the change in code in place.
When merging into develop, we'll need to remove Mesa tags and keep it so that develops see everything as mainnet/devnet, I assume?

dkijania · 2025-11-25T10:17:09Z

!ci-docker-me codename=bookworm arch=arm64 profile=lightnet

dkijania · 2025-11-25T10:18:09Z

So the files that would be removed when merging back to develop is expected, or it's a WIP?

I thought we want this branch quickly to be run in CI. Thus my hacky approach. If not, then i can sort it out longterm

dkijania · 2025-11-25T16:22:52Z

!ci-docker-me codename=bookworm arch=arm64 profile=devnet

dkijania · 2025-11-25T21:49:55Z

!ci-docker-me codename=bookworm arch=amd64 profile=devnet

dkijania · 2025-11-25T21:50:07Z

!ci-docker-me codename=bookworm arch=amd64 profile=lightnet

dkijania · 2025-11-26T17:41:21Z

!ci-docker-me codename=bookworm arch=amd64 profile=lightnet

glyh · 2025-11-30T09:19:53Z

I assume this is not yet ready for a review? If so could you draft?

dkijania self-assigned this Nov 18, 2025

dkijania changed the base branch from compatible to mesa November 18, 2025 13:29

dkijania mentioned this pull request Nov 19, 2025

update create schema #18143

Closed

dkijania force-pushed the ci_for_mesa branch from 90cf20a to ac5503d Compare November 24, 2025 18:31

dkijania marked this pull request as ready for review November 24, 2025 19:19

dkijania requested review from a team as code owners November 24, 2025 19:19

glyh reviewed Nov 25, 2025

View reviewed changes

dkijania added 6 commits November 27, 2025 11:23

add mesa related jobs

6455b12

add mesa related constants

7e38212

added mesa genesis ledger

e6cdaca

Port fix for archive node test

cbf7a2b

regenerate data for sample db

a799e6e

add mesa debians

e5cc63f

dkijania force-pushed the ci_for_mesa branch from 21bd284 to e5cc63f Compare November 27, 2025 10:28

add missing jobs

aef069e

CI for mesa #18130

Are you sure you want to change the base?

CI for mesa #18130

Uh oh!

Conversation

dkijania commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkijania commented Nov 18, 2025

Uh oh!

dkijania commented Nov 18, 2025

Uh oh!

dkijania commented Nov 19, 2025

Uh oh!

dkijania commented Nov 21, 2025

Uh oh!

dkijania commented Nov 23, 2025

Uh oh!

dkijania commented Nov 23, 2025

Uh oh!

dkijania commented Nov 23, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

dkijania commented Nov 24, 2025

Uh oh!

glyh commented Nov 25, 2025

Uh oh!

dkijania commented Nov 25, 2025

Uh oh!

glyh commented Nov 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glyh Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glyh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkijania commented Nov 18, 2025 •

edited

Loading

glyh Nov 25, 2025 •

edited

Loading

glyh left a comment •

edited

Loading