TA4: Enclave restart test #215

QuentinI · 2025-08-25T12:27:09Z

Closes #<ISSUE_NUMBER>

This PR:

Adds support for running batcher in TEE to the devnet test helpers
Uses it to support testing restarting TEE batcher in the batcher restart tests

This PR does not:

Key places to review:

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1210592590143369

dailinsubjam

Looks good.

philippecamacho · 2025-09-09T14:49:48Z

Why is it possible to run the test locally even without a TEE?

dailinsubjam · 2025-09-10T00:05:27Z

@philippecamacho If you're not on a enclave-enabled instance, you'll hit this line and it will automatically skip enclave-related operations.

dailinsubjam · 2025-09-10T04:39:55Z

espresso/devnet-tests/batcher_restart_test.go

+	testRestart(t, false)
+}


I just realized to restart op-batcher-tee we not only need profile to be tee but also need things like restarting the service op-batcher-tee specifically.

Oh, that's a big catch for sure, thanks. No idea how it passed for me in the first place.

philippecamacho · 2025-09-10T13:49:04Z

@philippecamacho If you're not on a enclave-enabled instance, you'll hit this line and it will automatically skip enclave-related operations.

I see, but when this test runs in CI, is the batcher executed inside the TEE?

QuentinI · 2025-09-10T16:09:32Z

I see, but when this test runs in CI, is the batcher executed inside the TEE?

No, it isn't. Only non-tee test runs.

QuentinI · 2025-10-15T19:12:18Z

@dailinsubjam @philippecamacho this works now, I've added a devnet-enclave-tests target to justfile which you can run on an AWS Nitro instance to verify (don't forget you need to be in nix shell).

Not set up to run in CI unfortunately, building dockers on AWS machine takes too long and GitHub kills the action. To run this in CI, we need to set up building dockers in one action and re-using in subsequent ones, including uploading them to the AWS Nitro instance, which I'd argue out of scope for this PR, LMK if you disagree.

dailinsubjam

TestEnclaveRestart pass for me.
But docker ps after the test, I can see

docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS          PORTS     NAMES
224fe2454142   op-proposer:espresso     "/bin/entrypoint.sh …"   13 minutes ago   Up 10 minutes             espresso-op-proposer-1
78fefd376a3b   op-challenger:espresso   "/bin/entrypoint.sh …"   13 minutes ago   Up 10 minutes             espresso-op-challenger-1
099c18b6b26a   op-batcher:espresso      "op-batcher --espres…"   13 minutes ago   Up 10 minutes             espresso-op-batcher-1

remains running.
We can open a separate ticket for cleaning them up.

dailinsubjam · 2025-10-15T19:46:24Z

espresso/devnet-tests/devnet_tools.go.orig

I see we’ve added a backup of the backup of the original… bold move 😎

Oh heck, this is from mergiraf, I'll remove it stat 🤣

dailinsubjam · 2025-10-15T20:59:37Z

espresso/devnet-tests/devnet_tools.go

+func (d *Devnet) ServiceDown(service Service) error {
+	serviceName := d.getServiceName(service)
+	log.Info("shutting down service", "service", serviceName)
 	cmd := exec.CommandContext(
 		d.ctx,
-		"docker", "compose", "down", service,
+		"docker", "compose", "--profile", d.getProfile(), "down", serviceName,
 	)
 	return cmd.Run()
 }


This reminds me of one thing: op-batcher-tee container spawns an enclave container (batcher-enclaver-xxx) that doesn’t stop automatically when op-batcher-tee stops. In Docker Compose I’ve been cleaning it up manually via espresso/scripts/shutdown.sh, but that’s probably not the best approach.
I'm thinking we may see the same issue here, even if op-batcher-tee exits, batcher-enclaver-xxx may keep running.
One possible way is to add a cleanup hook when shutting downop-batcher-tee (not sure whether it's supported) or also add a manual shutdown here. WDYT?

Oh interesting. I'll see what we can do here

dailinsubjam

Oops, I just found tee is not triggered automatically though I'm on a AWS Nitro, will take a further look.

tee is not triggered in my run for the test

QuentinI · 2025-10-15T22:24:12Z

@dailinsubjam I've mixed up order of operations during rebase, my tee flag was set after the devnet was spun up 🤦
Fixed it, going to re-run the thing now and see if it fixes the leftover containers (it should 🤞 )

dailinsubjam · 2025-10-16T17:08:28Z

I got this

op-geth-sequencer-1  | INFO [10-16|15:47:27.716] Persisted trie from memory database      nodes=1542 size=170.63KiB time=3.279752ms    gcnodes=5765 gcsiz
e=1.63MiB gctime=11.571326ms livenodes=3207 livesize=978.01KiB
op-geth-sequencer-1  | INFO [10-16|15:47:27.717] Writing cached state to disk             block=366 hash=aaa89c..41bd2d root=5e5706..bcf590
op-geth-sequencer-1  | INFO [10-16|15:47:27.717] Persisted trie from memory database      nodes=25   size=7.76KiB   time="116.902µs"   gcnodes=0    gcsiz
e=0.00B   gctime=0s          livenodes=3182 livesize=970.25KiB
op-geth-sequencer-1  | INFO [10-16|15:47:27.717] Writing cached state to disk             block=240 hash=b04582..df2713 root=67d352..32720e
op-geth-sequencer-1  | INFO [10-16|15:47:27.717] Persisted trie from memory database      nodes=269  size=42.88KiB  time="734.222µs"   gcnodes=0    gcsiz
e=0.00B   gctime=0s          livenodes=2913 livesize=927.37KiB
op-geth-sequencer-1  | INFO [10-16|15:47:27.717] Writing snapshot state to disk           root=1751bc..130f87
op-geth-sequencer-1  | INFO [10-16|15:47:27.717] Persisted trie from memory database      nodes=0    size=0.00B     time="2.68µs"      gcnodes=0    gcsiz
e=0.00B   gctime=0s          livenodes=2913 livesize=927.37KiB
op-geth-sequencer-1  | INFO [10-16|15:47:27.721] Blockchain stopped
l1-geth-1            | INFO [10-16|15:47:28.004] Starting work on payload                 id=0x03b6dac60f0c1813
l1-geth-1            | INFO [10-16|15:47:28.004] Updated payload                          id=0x03b6dac60f0c1813 number=246 hash=e17a7f..15b198 txs=0 with
drawals=0 gas=0          fees=0           root=8c906e..654623 elapsed="132.132µs"
espresso-dev-node-1 exited with code 143
l1-validator-1 exited with code 0
l1-beacon-1          | Oct 16 15:47:28.614 INFO  Shutting down..                               reason: Success("Received SIGTERM") 
l1-beacon-1          | Oct 16 15:47:28.614 INFO  Saved DHT state                              service: "network"
l1-beacon-1          | Oct 16 15:47:28.614 INFO  Network service shutdown                     service: "network"
l1-beacon-1          | Oct 16 15:47:28.618 INFO  Saved beacon chain to disk                   service: "network"
op-geth-sequencer-1 exited with code 0 
l1-beacon-1 exited with code 0
l1-geth-1 exited with code 137
--- FAIL: TestEnclaveRestart (778.84s) 
FAIL
FAIL    github.com/ethereum-optimism/optimism/espresso/devnet-tests     788.666s
FAIL
error: Recipe `devnet-enclave-tests` failed on line 47 with exit code 1

Started in tmux so I didn't fetch the full log TAT I can fetch the complete logs with a new run later

Base automatically changed from sishan/devnet-batcher-tee to celo-integration-rebase-13.2 August 29, 2025 22:19

QuentinI force-pushed the ag/enclave-restart-test branch from 1ce717f to 9051371 Compare September 3, 2025 09:10

QuentinI changed the title ~~[WIP] enclave restart test~~ TA4: Enclave restart test Sep 3, 2025

QuentinI marked this pull request as ready for review September 4, 2025 13:32

dailinsubjam approved these changes Sep 5, 2025

View reviewed changes

dailinsubjam requested changes Sep 10, 2025

View reviewed changes

QuentinI force-pushed the ag/enclave-restart-test branch 2 times, most recently from 4bf678e to 3a01ee2 Compare September 29, 2025 19:29

QuentinI force-pushed the ag/enclave-restart-test branch 3 times, most recently from 74b5abf to 5475f6d Compare October 15, 2025 17:15

dailinsubjam mentioned this pull request Oct 15, 2025

Add devnet smoke test with TEE #234

Closed

dailinsubjam previously approved these changes Oct 15, 2025

View reviewed changes

dailinsubjam reviewed Oct 15, 2025

View reviewed changes

dailinsubjam self-requested a review October 16, 2025 13:09

QuentinI added 7 commits October 24, 2025 19:10

Test batcher restart in enclave

3be1bd2

Support differing container names in profiles

61766cc

Run enclave devnet tests in CI

ab312ae

Don't run enclave tests in CI

1b9e42e

Add build-devnet dependency to devnet-enclave-tests

31b6271

Remove mergiraf artifact

523723d

Shutdown tee on exit

081d914

QuentinI added 3 commits October 24, 2025 19:10

Use profile to query devnet status

076c8b6

Fix order of operations setting tee flag

88186fd

Wait for devent to start up

f961472

QuentinI force-pushed the ag/enclave-restart-test branch from e76deaa to f961472 Compare October 24, 2025 17:10

TA4: Enclave restart test #215

Are you sure you want to change the base?

TA4: Enclave restart test #215

Conversation

QuentinI commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR:

This PR does not:

Key places to review:

Uh oh!

dailinsubjam left a comment

Choose a reason for hiding this comment

Uh oh!

philippecamacho commented Sep 9, 2025

Uh oh!

dailinsubjam commented Sep 10, 2025

Uh oh!

dailinsubjam Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

QuentinI Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

philippecamacho commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

QuentinI commented Sep 10, 2025

Uh oh!

QuentinI commented Oct 15, 2025

Uh oh!

dailinsubjam left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dailinsubjam Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

QuentinI Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dailinsubjam Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

QuentinI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

dailinsubjam left a comment

Choose a reason for hiding this comment

Uh oh!

QuentinI commented Oct 15, 2025

Uh oh!

dailinsubjam commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

QuentinI commented Aug 25, 2025 •

edited

Loading

philippecamacho commented Sep 10, 2025 •

edited

Loading

dailinsubjam left a comment •

edited

Loading

QuentinI Oct 15, 2025 •

edited

Loading