Pre 0.4.11 cbor auxdata #2277

kuzdogan · 2025-08-05T12:10:26Z

It wasn't possible to verify contracts that have a cborAuxdata and solc <=0.4.11 because the .auxdata field in legacyAssembly was just introduced on 0.4.12. This handles the cases when the compiler is <=0.4.11 and also <0.4.7 when metadata was first introduced.

Also handles when contractIdentifier path was missing before solc 0.4.9 and when there was no metadata 09692c0

Adds tests for the cases

manuelwedler · 2025-08-05T12:26:26Z

packages/lib-sourcify/src/Compilation/SolidityCompilation.ts

@@ -202,7 +266,7 @@ export class SolidityCompilation extends AbstractCompilation {
  public async compile(forceEmscripten = false) {
    const contract =
      await this.compileAndReturnCompilationTarget(forceEmscripten);
-    this._metadata = JSON.parse(contract.metadata.trim());
+    this._metadata = JSON.parse(contract.metadata || '{}'.trim());


We have one place (maybe even more?) in the server where we expect metadata to be present:
https://github.com/ethereum/sourcify/blob/fb6387a9b948f0fa44fd13c2c71a4abc53bc8ea4/services/server/src/server/services/storageServices/SourcifyDatabaseService.ts#L344
We should also adapt this.

Besides this, you need braces here:

Suggested change

this._metadata = JSON.parse(contract.metadata || '{}'.trim());

this._metadata = JSON.parse((contract.metadata || '{}').trim());

manuelwedler

I think it would be nice to also have tests that show that versions 0.4.9 and 0.4.0 verify correctly. Best would be integration tests in the server.

manuelwedler · 2025-08-06T10:52:38Z

packages/lib-sourcify/src/Compilation/SolidityCompilation.ts

@@ -202,7 +267,7 @@ export class SolidityCompilation extends AbstractCompilation {
  public async compile(forceEmscripten = false) {
    const contract =
      await this.compileAndReturnCompilationTarget(forceEmscripten);
-    this._metadata = JSON.parse(contract.metadata.trim());
+    this._metadata = JSON.parse((contract.metadata || '{}').trim());


This still needs to be fixed (as in my comment earlier):

https://github.com/ethereum/sourcify/blob/fb6387a9b948f0fa44fd13c2c71a4abc53bc8ea4/services/server/src/server/services/storageServices/SourcifyDatabaseService.ts#L344

manuelwedler · 2025-08-06T11:05:55Z

packages/lib-sourcify/src/Compilation/SolidityCompilation.ts

@@ -41,6 +42,11 @@ export class SolidityCompilation extends AbstractCompilation {
    public compilationTarget: CompilationTarget,
  ) {
    super(jsonInput);
+    // https://github.com/ethereum/solidity/releases/tag/v0.4.9
+    // pre-0.4.9 doesn't have "filename" or "path" in the compiler output
+    if (semver.lt(this.compilerVersion, '0.4.9')) {


Can we have a test that proves that these versions verify correctly?

Our getter functions for contract output in AbstractCompilation depend on the assumption that there is a path:

https://github.com/ethereum/sourcify/blob/235acef09f5d05fed2466549884bdd45894d0067/packages/lib-sourcify/src/Compilation/AbstractCompilation.ts#L132-L134

My guess is that the output is structured differently for these Solidity versions.

kuzdogan · 2025-08-07T12:57:59Z

I was trying to add tests for different compiler versions but somehow solcjs is not working on my setup. Pushing it here to see how it behaves on CI. Can you also check if it works on Linux?

My compilations get stuck here
https://github.com/ethereum/sourcify/blob/fb3e01d3cec2f5894abced039c9c6f3276f5921d/packages/compilers/src/lib/solidityCompiler.ts#L267-L268

Note: I have a describe.only on purpose

manuelwedler · 2025-08-07T16:03:02Z

I didn't look too deeply to understand what's going on, but it's not working on my machine either. I cannot tell if it is indeed stuck on the line you referenced above. I get the following logs:

    5) should verify SimpleStorage compiled with Solidity 0.4.0+commit.acd334c9
(node:63406) V8: /home/manuel/.solc-bin/soljson/soljson-v0.4.0+commit.acd334c9.js:5413 Invalid asm.js: Invalid member of stdlib
(Use `node --trace-warnings ...` to show where the warning was created)
2025-08-07T15:42:21.650Z [error] [Compilers] Error getting solc executable - error={"message":"Failed fetching solc 0.4.0+commit.acd334c9 for platform linux-amd64. Please check if the version is valid.","stack":"Error: Failed fetching solc 0.4.0+commit.acd334c9 for platform linux-amd64. Please check if the version is valid.\n    at fetchAndSaveSolc (/home/manuel/Projects/sourcify/sourcify/packages/compilers/build/main/lib/solidityCompiler.js:225:15)\n    at processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async getSolcExecutable (/home/manuel/Projects/sourcify/sourcify/packages/compilers/build/main/lib/solidityCompiler.js:165:5)\n    at async useSolidityCompiler (/home/manuel/Projects/sourcify/sourcify/packages/compilers/build/main/lib/solidityCompiler.js:89:24)\n    at async SolcLocal.compile (/home/manuel/Projects/sourcify/sourcify/services/server/src/server/services/compiler/local/SolcLocal.ts:19:12)\n    at async SolidityCompilation.compileAndReturnCompilationTarget (/home/manuel/Projects/sourcify/sourcify/packages/lib-sourcify/build/main/Compilation/AbstractCompilation.js:21:35)\n    at async SolidityCompilation.compile (/home/manuel/Projects/sourcify/sourcify/packages/lib-sourcify/build/main/Compilation/SolidityCompilation.js:185:26)\n    at async Verification.verify (/home/manuel/Projects/sourcify/sourcify/packages/lib-sourcify/build/main/Verification/Verification.js:48:9)\n    at async _verifyFromJsonInput (/home/manuel/Projects/sourcify/sourcify/services/server/src/server/services/workers/verificationWorker.ts:152:5)\n    at async onMessage (/home/manuel/Projects/sourcify/sourcify/node_modules/piscina/src/worker.ts:184:18)","name":"Error"} | solcPlatform=linux-amd64 | version=0.4.0+commit.acd334c9 | solcRepoPath=/home/manuel/.solc-bin/linux-amd64 | solJsonRepoPath=/home/manuel/.solc-bin/soljson - [traceId=cd04fc5b-aa98-41fe-8126-172a3649d620]
2025-08-07T15:42:21.650Z [info] [Compilers] Compiling with solc-js - version=0.4.0+commit.acd334c9 - [traceId=cd04fc5b-aa98-41fe-8126-172a3649d620]
2025-08-07T15:45:30.242Z [info] [Compilers] Local compiler - Compilation done - compiler=solidity | timeInMs=190 - [traceId=cd04fc5b-aa98-41fe-8126-172a3649d620]

Note the "invalid asm.js" log, and note that the log "Local compiler - Compilation done " is 3 minutes after the one before but it says timeInMs=190.

At the end the tests fail without properly terminating node due to reaching the heap limit (port is still bound when trying to run again):

2025-08-07T15:45:31.258Z [info] [Compilers] Saved solc - version=v0.4.6+commit.2dabbdf0 | platform=bin | githubSolcURI=https://binaries.soliditylang.org/bin/soljson-v0.4.6%2Bcommit.2dabbdf0.js | solcPath=/home/manuel/.solc-bin/soljson/soljson-v0.4.6+commit.2dabbdf0.js - [traceId=dbd25a13-c89e-465c-9b8d-ae808de9dd99]

<--- Last few GCs --->

[63406:0x75d194001000]   289257 ms: Mark-Compact 4055.2 (4136.6) -> 4053.9 (4136.6) MB, pooled: 0 MB, 332.04 / 0.00 ms  (average mu = 0.964, current mu = 0.338) allocation failure; scavenge might not succeed
[63406:0x75d194001000]   290181 ms: Mark-Compact 4061.7 (4136.6) -> 4060.4 (4166.6) MB, pooled: 2 MB, 862.86 / 0.00 ms  (average mu = 0.849, current mu = 0.066) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xe36196 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0x123f4a0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0x123f777 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x146d1a5  [node]
 5: 0x1486a19 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 6: 0x145b0e8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 7: 0x145c015 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0x1434cee v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
 9: 0x18962ec v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x75d18feac476 
Aborted (core dumped)

I think there might be an issue with the solcjs version. If the usual solc executable is not available on https://binaries.soliditylang.org/, we should maybe add a different source for downloading it (GitHub?).

kuzdogan · 2025-08-11T10:10:44Z

Since this is looking more complicated than it seems, I'd remove the latest tests (rebase HEAD~1 to fb3e01d) and open a separate new issue for pre v0.4.11 verification and all issues regarding solcjs failing in server context but working in lib-sourcify. lmk if this sounds ok

Regarding your comment: https://github.com/ethereum/sourcify/pull/2277/files#r2256775854 I'm not sure if I get it but I feel we should make the metadata field nullable and handle accordingly?

manuelwedler · 2025-08-11T10:52:38Z

Since this is looking more complicated than it seems, I'd remove the latest tests (rebase HEAD~1 to fb3e01d) and open a separate new issue for pre v0.4.11 verification and all issues regarding solcjs failing in server context but working in lib-sourcify. lmk if this sounds ok

If I got it right, these problems can crash the server. Can we try to identify which version is the earliest working, and disable verification for any version lower than this, e.g. by just throwing an error from SolidityCompilation?

Regarding your comment: https://github.com/ethereum/sourcify/pull/2277/files#r2256775854 I'm not sure if I get it but I feel we should make the metadata field nullable and handle accordingly?

I agree that we should make it nullable. You could also add a schema change for this here. The problem in my comment is that the referenced line in SourcifyDatabase assumes that metadata is not null and assumes a compiler property. So this line can throw unexpectedly with the changes in this PR. I was asking to adapt that line to make it compatible with empty / null metadata.

kuzdogan · 2025-08-12T10:36:21Z

Ok this has gotten reeaaaaly complicated. While trying to figure out which version actually works for verification I noticed that contracts with 0.4.10 also can't be verified because they don't have userdoc and devdoc fields, as std-json was introduced in 0.4.11.

This seems to contradict our database schema. But I actually couldn't point which fields cause this. The only difference between a 0.4.11 and 0.4.10 contracts compiler artifacts (because it fails with new row for relation \"compiled_contracts\" violates check constraint \"compilation_artifacts_json_schema\) is the latter has userdoc and devdoc null. So I first thought that's the reason. But looking at the sql validation I can't find anything that wouldn't allow them to be null. Plus they are validated the same as the storageLayout and storageLayout is null in both cases. https://github.com/verifier-alliance/database-specs/blob/8aafd9858217c890c942bd3b61786d0e1302ad38/database.sql#L371-L385 We have a lot of contracts with storageLayout: null but only two contracts with devdoc: null (also v0.4.10). I don't really know how these got in.

Query for contracts with empty abi or userdoc or devdoc compiler versions

-- Query for contracts with empty abi or userdoc or devdoc compiler versions
SELECT 
  cd.chain_id,
  ENCODE(cd.address, 'hex') AS address,
  cc.compilation_artifacts,
  cc.compiler,
  cc.version
FROM compiled_contracts cc
JOIN verified_contracts vc ON cc.id = vc.compilation_id
JOIN contract_deployments cd ON vc.deployment_id = cd.id
WHERE cc.compilation_artifacts->'userdoc' = 'null' -- change here
limit 5;

Also somehow we verified a few contracts with 0.4.10. Here are the two:
4 33633363139F53F96A8FFCFF42400F3E866121F4
56 BA5AD0203F6BE95AEF9E5BFE192D7271DDD76217

Query for contracts with earlier compiler versions

-- Contracts with earlier compiler versions
SELECT 
  cd.chain_id,
  cd.address,
  cc.compiler,
  cc.compilation_artifacts->'userdoc' as userdoc,
  cc.compilation_artifacts->'devdoc' as devdoc,
  cc.compilation_artifacts->'storageLayout' as storageLayout,
  cc.compilation_artifacts->'abi' as abi,
  cc.version
FROM compiled_contracts cc
JOIN verified_contracts vc ON cc.id = vc.compilation_id
JOIN contract_deployments cd ON vc.deployment_id = cd.id
WHERE cc.version ilike '%0.4.10%'
limit 100;

So what am I doing?

In this PR I'm fixing only the underlying issue of legacyAssembly not having .auxdata fields. Just handle this of <0.4.12 contracts
Anything less than 0.4.11 throw and don't accept. We will be able to verify 0.4.11 as it has metadata and std-json
Postpone everything related to verifying earlier versions to later as documented in Verification of pre 0.4.11 contracts #2296 including handling of the metadata. For now we'll assume we indeed have the metadata.

kuzdogan · 2025-08-12T12:46:47Z

Ok this is the final version, can you please review @manuelwedler

The added test with 0.4.11 is still failing on my local environment but runs on the CI. Somehow there's an issue with running the solc-json specifically on the import() step. I keep the test with a comment at the top about this. I don't know if this is a good idea, or if we should open an issue for this

manuelwedler

Looks good to me :)

The test runs fine on my machine, let's keep it with the comment.
Also thanks for the type fixes. I like it when we include smaller fixes like this in PRs.

kuzdogan · 2025-08-13T14:53:28Z

Merged staging to resolve conflicts

github-project-automation bot added this to Sourcify Public [Archived] Aug 5, 2025

github-project-automation bot moved this to Triage in Sourcify Public [Archived] Aug 5, 2025

kuzdogan moved this from Triage to Sprint - Needs Review in Sourcify Public [Archived] Aug 5, 2025

manuelwedler reviewed Aug 5, 2025

View reviewed changes

kuzdogan force-pushed the pre-0.4.11-cborAuxdata branch from d1558ee to 838a275 Compare August 5, 2025 12:36

manuelwedler mentioned this pull request Aug 5, 2025

Server crashes when a compiled contract does not have metadata in the output #2207

Closed

manuelwedler requested changes Aug 6, 2025

View reviewed changes

kuzdogan mentioned this pull request Aug 12, 2025

Verification of pre 0.4.11 contracts #2296

Open

kuzdogan force-pushed the pre-0.4.11-cborAuxdata branch 4 times, most recently from 878898c to f354b9d Compare August 12, 2025 12:43

kuzdogan added 5 commits August 12, 2025 14:44

Handle legacyAssembly without .auxdata for solc <=0.4.11

21942c1

Add tests for pre-0.4.11 solc cborAuxdata handling

14683cb

Use consistent ethers version and JsonFragment instead of abitype

1992bc7

Throw on pre-0.4.11 Solidity versions in lib-sourcify

06c8c86

Add server integration tests for 0.4.11 (pass) and 0.4.10 (fail)

e974f39

kuzdogan force-pushed the pre-0.4.11-cborAuxdata branch from f354b9d to e974f39 Compare August 12, 2025 12:44

kuzdogan requested a review from manuelwedler August 12, 2025 12:46

manuelwedler approved these changes Aug 13, 2025

View reviewed changes

Merge branch 'staging' into pre-0.4.11-cborAuxdata

8e8e7fa

kuzdogan merged commit 2714dad into staging Aug 13, 2025
6 checks passed

github-project-automation bot moved this from Sprint - Needs Review to Sprint - Done in Sourcify Public [Archived] Aug 13, 2025

kuzdogan deleted the pre-0.4.11-cborAuxdata branch August 13, 2025 15:11

kuzdogan moved this from Sprint - Done to COMPLETED in Sourcify Public [Archived] Aug 14, 2025

stackenbotten3000 added this to Sourcify Public Aug 27, 2025

stackenbotten3000 moved this to COMPLETED in Sourcify Public Aug 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pre 0.4.11 cbor auxdata #2277

Pre 0.4.11 cbor auxdata #2277

Uh oh!

kuzdogan commented Aug 5, 2025 •

edited

Loading

Uh oh!

manuelwedler Aug 5, 2025

Uh oh!

manuelwedler Aug 5, 2025

Uh oh!

manuelwedler left a comment

Uh oh!

manuelwedler Aug 6, 2025

Uh oh!

manuelwedler Aug 6, 2025 •

edited

Loading

Uh oh!

kuzdogan commented Aug 7, 2025

Uh oh!

manuelwedler commented Aug 7, 2025

Uh oh!

kuzdogan commented Aug 11, 2025 •

edited

Loading

Uh oh!

manuelwedler commented Aug 11, 2025

Uh oh!

kuzdogan commented Aug 12, 2025 •

edited

Loading

Uh oh!

kuzdogan commented Aug 12, 2025

Uh oh!

manuelwedler left a comment

Uh oh!

kuzdogan commented Aug 13, 2025

Uh oh!

Uh oh!

Uh oh!

	this._metadata = JSON.parse(contract.metadata \|\| '{}'.trim());
	this._metadata = JSON.parse((contract.metadata \|\| '{}').trim());

Pre 0.4.11 cbor auxdata #2277

Pre 0.4.11 cbor auxdata #2277

Uh oh!

Conversation

kuzdogan commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manuelwedler Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

manuelwedler Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

manuelwedler left a comment

Choose a reason for hiding this comment

Uh oh!

manuelwedler Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

manuelwedler Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kuzdogan commented Aug 7, 2025

Uh oh!

manuelwedler commented Aug 7, 2025

Uh oh!

kuzdogan commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manuelwedler commented Aug 11, 2025

Uh oh!

kuzdogan commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuzdogan commented Aug 12, 2025

Uh oh!

manuelwedler left a comment

Choose a reason for hiding this comment

Uh oh!

kuzdogan commented Aug 13, 2025

Uh oh!

Uh oh!

Uh oh!

kuzdogan commented Aug 5, 2025 •

edited

Loading

manuelwedler Aug 6, 2025 •

edited

Loading

kuzdogan commented Aug 11, 2025 •

edited

Loading

kuzdogan commented Aug 12, 2025 •

edited

Loading