design compiler standard output format

[serapath]

@todo

• write down all fields which exist
• what are devdoc, userdoc, functionHashes, etc... in metadata about and should they be kept?
• check if all of the fields exists in all compiler versions and list the ones which do not have some
• check if the new compiler (> v0.5) offers some additional data
• decide and specify output format
• refactor compiler to produce output format
• update existing modules to consume the new output format

---

now

var compilation = {
      name: name.substring(1),
      abi: abi,
      sources: sources,
      compiler: compile,
      assembly: { assembly, opcodes },
      binary: {
        bytecodes: { bytecode, runtimeBytecode },
        sourcemap: { srcmap, srcmapRuntime },
      },
      metadata: {
        ast: data.sources[''].AST,
        devdoc: metadata.output.devdoc,
        userdoc: metadata.output.userdoc,
        functionHashes,
        gasEstimates,
        analysis: {
            warnings: [],
            others: []
        }
      }
    };

---

https://solidity.readthedocs.io/en/v0.5.1/using-the-compiler.html?highlight=outputSelection

    // The available output types are as follows:
    //   abi - ABI
    //   ast - AST of all source files
    //   legacyAST - legacy AST of all source files
    //   devdoc - Developer documentation (natspec)
    //   userdoc - User documentation (natspec)
    //   metadata - Metadata
    //   ir - New assembly format before desugaring
    //   evm.assembly - New assembly format after desugaring
    //   evm.legacyAssembly - Old-style assembly format in JSON
    //   evm.bytecode.object - Bytecode object
    //   evm.bytecode.opcodes - Opcodes list
    //   evm.bytecode.sourceMap - Source mapping (useful for debugging)
    //   evm.bytecode.linkReferences - Link references (if unlinked object)
    //   evm.deployedBytecode* - Deployed bytecode (has the same options as evm.bytecode)
    //   evm.methodIdentifiers - The list of function hashes
    //   evm.gasEstimates - Function gas estimates
    //   ewasm.wast - eWASM S-expressions format (not supported atm)
    //   ewasm.wasm - eWASM binary format (not supported atm)

{ NewContract: 
    { abi: [[Object]],
     devdoc: { methods: {}},
     evm: { bytecode: [Object]},
     metadata: '{
      "compiler": {
        "version": "0.5.1+commit.c8a2cb62"
      },
      "language": "Solidity",
      "output": {
        "abi": [...],
        "devdoc": {
          "methods": {}
        },
        "userdoc": {
          "methods": {}
        }
      },
      "settings": {
        "compilationTarget": {
          "MyContract": "NewContract"
        },
        "evmVersion": "byzantium",
        "libraries": {},
        "optimizer": {
          "enabled": true,
          "runs": 200
        },
        "remappings": []
      },
      "sources": {
        "MyContract": {
          "content": "...",
          "keccak256": "0x7523c81a65abb96ce8e0947d26b8b8a55cf3cc527bf9038ece9cf293478f9428"
        }
      },
      "version": 1
    }',
     userdoc: { methods: {}}
    },
    OldLibrary: { abi: [[Object]],
      devdoc: { methods: {}},
      evm: { bytecode: [Object]},
      metadata: '',
      userdoc: { methods: {}}
    }
}

---

version 0.5.0 and 0.5.1 don't have

functionHashes
~binary.bytecodes.runtimeBytecode~
~binary.sourcemap.srcmapRuntime~
sources.sourcecode.urls
sources.sourcecode.sourcelist

the ast value is missing.
not sure what is problems yet.

---

We identified the following sections in the documentation that talk about the compiler input and output and we would probably change the VERSION in the documentation to check older compiler version for their corresponding information, but we are concerned about the docs talking about the cli and not solcjs and wonder if there are more sources we could study to answer our remaining questions:

1. https://solidity.readthedocs.io/en/v0.5.1/metadata.html
   2 https://solidity.readthedocs.io/en/v0.5.1/abi-spec.html
2. https://solidity.readthedocs.io/en/v0.5.1/using-the-compiler.html
3. https://solidity.readthedocs.io/en/v0.5.1/metadata.html

Questions

1. In what way does this input spec apply to solcjs?

   • e.g. sources['XXX.sol'].urls
   • e.g. settings.remappings

2. How do we best get all imports a contract uses so i can fetch missing contract data from url's, cache them and pass them to the compiler when using solcjs?

3. How do I get the AST now. It seems to not be part of the compile output anymore or we have to specify it somehow - AND is it possible to get the AST without compiling using maybe another module?

4. Even though the documentation has some comments about the fields listed below, they don't really describe or define the fields in ways that make them clear enough and seem to assume a lot of background knowledge. What is the format and structure and purpose of the following compiler output fields - and how do i add/change/influence it/them?

   • compilationTarget
     • https://solidity.readthedocs.io/en/v0.5.1/using-the-compiler.html?highlight=outputSelection#target-options
   • optimizer (.enabled, .runs)
     • https://solidity.readthedocs.io/en/v0.5.1/using-the-compiler.html?highlight=outputSelection#using-the-commandline-compiler
   • natspec
   • devdoc
   • userdoc
   • remappings
   • sources.id
   • metadata
   • version : 1
   • functionHashes
   • ewasm.wast
   • ewasm.wasm
   • binary.bytecode.runtimeBytecode
   • deployedBytecode
   • libraries
   • binary.sourcemap.srcmapRuntime
   • sources.sourcecode.urls
   • sources.sourcecode.sourcelist
   • ast vs. legacyAST vs. no ast at all?
   • metadata.useLiteralContent
   • ir (new assembly format before desugaring)
   • evm (.assembly,.legacyAssembly.bytecode, ...)
   • evm.linkReferences
   • outputSelection seems to be very important but it's not entirely clear how to use it
     • to which solidity version(s) does it apply and what are all the possible values we can select?
   • errors[i].component
     • what does "general", "ewasm", etc... mean and what are all possible values?
   • can there be other severity levels than "error"and"warning"`?
   • methodIdentifiers
   • gasEstimates external vs internal vs creation

answers:

• Improve explanation how to get abi output. argotorg/solidity#5621
• Function hashes are gone, but I guess we removed them because they can be easily generated from the ABI data
• gas estimates are still there: <filename>.<ContractName>.evm.gasEstimates
  • runtime bytecode is now called deployedBytecode. The source maps are called
  • sourceMap and they are inside the bytecode object.
  • The source list might actually have disappeared, I'll check that here:
  • The source list (used by the source mapping) is not present in standard-json. argotorg/solidity#5623

• the input description you linked ( https://solidity.readthedocs.io/en/v0.5.1/using-the-compiler.html#input-description ) is the description of the input format as used by the latest soljson or by solc-js if the wrapper is used (by default)
• use the callback mechanism to load missing sources. Note that you cannot use any async calls during the callback, so either store what is requested and call the compiler again or load in a sync way
• ast: see above
• the fields: Yes, it does assume some background knowledge. If you don't know what the field is about, then you can probably ignore it.

• It turns out the source list is there, at least the information that is needed from the source list: sources.<sourceName>.id is the index into the old source list, i.e. the index that is used for the source references.

---

version 0.3.6 don't have

sources.compilationTarget
sources.libraries
sources.remappings
sources.sourcecode
compiler.evmVersion
compiler.language
compiler.optimizer
compiler.runs
metadata.ast.attributes
metadata.ast.id
metadata.ast.src
metadata.devdoc
metadata.userdoc

version 0.3.5 ~ 0.3.2 don't have

sources.sourcelist
binary.sourcemap.srcmap
binary.sourcemap.srcmapRuntime

---

version v0.2.2 don't have

sources.compilationTarget
sources.libraries
sources.remappings
sources.sourcecode
sources.sourcelist

compiler.evmVersion
compiler.language
compiler.optimizer
compiler.runs

metadata.ast.attributes
metadata.ast.id
metadata.ast.src
metadata.devdoc
metadata.userdoc

binary.sourcemap.srcmap
binary.sourcemap.srcmapRuntime

---

ast key is change.

0.5.x

ast.absolutePath
ast.exportedSymbols
ast.id
ast.nodeType
ast.nodes
ast.src

0.4.x

ast.attributes
ast.children
ast.id
ast.name
ast.src

---

https://solidity.readthedocs.io/en/v0.5.2/using-the-compiler.html#output-description

---

I have a question. now, output formation is an object.

{
      name: ...
      abi: ...
      sources: ...
      compiler: ...
      assembly: ...
      binary: {
        bytecodes: { bytecode, runtimeBytecode },
        sourcemap: { srcmap, srcmapRuntime },
      },
      metadata: {
        ast: ...
        devdoc: ...
        userdoc: ...
        functionHashes,
        gasEstimates,
        analysis: {
            warnings: [],
            others: []
        }
      }
    }