v0.28.0

This is the v0.28.0 release.

Upgrade notes

• New DB table for dynamic data sources.
  For new subgraph deployments, dynamic data sources will be recorded under the sgd*.data_sources$ table, rather than subgraphs.dynamic_ethereum_contract_data_source. As a consequence new deployments will not work correctly on earlier graph node versions, so downgrading to an earlier graph node version is not supported.
  See issue #3405 for other details.

What's new

• The filepath which "too expensive qeueries" are sourced from is now configurable. You can use either the GRAPH_NODE_EXPENSIVE_QUERIES_FILE environment variable or the expensive_queries_filename option in the TOML configuration. #3710
• The output you'll get from graphman query is less cluttered and overall nicer. The new options --output and --trace are available for detailed query information. #3860
• docker build will now --target the production build stage by default. When you want to get the debug build, you now need --target graph-node-debug. #3814
• Node IDs can now contain any character. The Docker start script still replaces hyphens with underscores for backwards compatibility reasons, but this behavior can be changed with the GRAPH_NODE_ID_USE_LITERAL_VALUE environment variable. With this new option, you can now seamlessly use the K8s-provided host names as node IDs, provided you reassign your deployments accordingly. #3688
• You can now use the conn_pool_size option in TOML configuration files to configure the connection pool size for Firehose providers. #3833
• Index nodes now have an endpoint to perform block number to canonical hash conversion, which will unblock further work towards multichain support. #3942
• _meta.block.timestamp is now available for subgraphs indexing EVM chains. #3738, #3902
• The deployment_eth_rpc_request_duration metric now also observes eth_getTransactionReceipt requests' duration. #3903
• New Prometheus metrics query_parsing_time and query_validation_time for monitoring query processing performance. #3760
• New command graphman config provider, which shows what providers are available for new deployments on a given network and node. #3816
  E.g. $ graphman --node-id index_node_0 --config graph-node.toml config provider mainnet
• Experimental support for GraphQL API versioning has landed. #3185
• Progress towards experimental support for off-chain data sources. #3791
• Experimental integration for substreams. #3777, #3784, #3897, #3765, and others

Bug fixes

• graphman stats now complains instead of failing silently when incorrectly setting account-like optimizations. #3918
• Fixed inconsistent logic in the provider selection when the limit TOML configuration option was set. #3816
• Fixed issues that would arise from dynamic data sources' names clashing against template names. #3851
• Dynamic data sources triggers are now processed by insertion order. #3851, #3854
• When starting, the Docker image now replaces the bash process with the graph-node process (with a PID of 1). #3803
• Refactor subgraph store tests by @evaporei in #3662
• The ethereum_chain_head_number metric doesn't get out of sync anymore on chains that use Firehose. #3771, #3732
• Fixed a crash caused by bad block data from the provider. #3944
• Fixed some minor Firehose connectivity issues via TCP keepalive, connection and request timeouts, and connection window size tweaks. #3822, #3855, #3877, #3810, #3818
• Copying private data sources' tables across shards now works as expected. #3836

Performance improvements

• Firehose GRPC stream requests are now compressed with gzip, if the server supports it. #3893
• Memory efficiency improvements within the entity cache. #3594
• Identical queries now benefit from GraphQL validation caching, and responses are served faster. #3759

Other

• Avoid leaking some sensitive information in logs. #3812

v0.27.0

• Store writes are now carried out in parallel to the rest of the subgraph process, improving indexing performance for subgraphs with significant store interaction. Metrics & monitoring was updated for this new pipelined process;
• This adds support for apiVersion 0.0.7, which makes receipts accessible in Ethereum event handlers. Documentation link;
• This introduces some improvements to the subgraph GraphQL API, which now supports filtering on the basis of, and filtering for entities which changed from a certain block;
• Support was added for Arweave indexing. Tendermint was renamed to Cosmos in Graph Node. These integrations are still in "beta";
• Callhandler block filtering for contract calls now works as intended (this was a longstanding bug);
• Gas costing for mappings is still set at a very high default, as we continue to benchmark and refine this metric;
• A new graphman fix block command was added to easily refresh a block in the block cache, or clear the cache for a given network;
• IPFS file fetching now uses files/stat, as object was deprecated;
• Subgraphs indexing via a Firehose can now take advantage of Firehose-side filtering;
• NEAR subgraphs can now match accounts for receipt filtering via prefixes or suffixes.

Upgrade notes

• In the case of you having custom SQL, there's a new SQL migration;
• On the pipelining of the store writes, there's now a new environment variable GRAPH_STORE_WRITE_QUEUE (default value is 5), that if set to 0, the old synchronous behaviour will come in instead. The value stands for the amount of write/revert parallel operations #3177;
• There's now support for TLS connections in the PostgreSQL notification_listener #3503;
• GraphQL HTTP and WebSocket ports can now be set via environment variables #2832;
• The genesis block can be set via the GRAPH_ETHEREUM_GENESIS_BLOCK_NUMBER env var #3650;
• There's a new experimental feature to limit the number of subgraphs for a specific web3 provider. Link for documentation;
• Two new GraphQL validation environment variables were included: ENABLE_GRAPHQL_VALIDATIONS and SILENT_GRAPHQL_VALIDATIONS, which are documented here;
• A bug fix for graphman index was landed, which fixed the behavior where if one deployment was used by multiple names would result in the command not working #3416;
• Another fix landed for graphman, the bug would allow the unassign/reassign commands to make two or more nodes index the same subgraph by mistake #3478;
• Error messages of eth RPC providers should be clearer during graph-node start up #3422;
• Env var GRAPH_STORE_CONNECTION_MIN_IDLE will no longer panic, instead it will log a warning if it exceeds the pool_size #3489;
• Failed GraphQL queries now have proper timing information in the service metrics #3508;
• Non-primary shards now can be disabled through setting the pool_size to 0 #3513;
• Queries with large results now have a query_id #3514;
• It's now possible to disable the LFU Cache by setting GRAPH_QUERY_LFU_CACHE_SHARDS to 0 #3522;
• GRAPH_ACCOUNT_TABLES env var is not supported anymore #3525;
• New documentation landed on the metadata tables;
• GRAPH_GRAPHQL_MAX_OPERATIONS_PER_CONNECTION for GraphQL subscriptions now has a default of 1000 #3735

v0.26.0

0.26.0

Features

• Gas metering #2414
• Adds support for Solidity Custom Errors #2577
• Debug fork tool #2995 #3292
• Automatically remove unused deployments #3023
• Fix fulltextsearch space handling #3048
• Allow placing new deployments onto one of several shards #3049
• Make NEAR subgraphs update their sync status #3108
• GraphQL validations #3164
• Add special treatment for immutable entities #3201
• Tendermint integration #3212
• Skip block updates when triggers are empty #3223 #3268
• Use new GraphiQL version #3252
• GraphQL prefetching #3256
• Allow using Bytes as well as String/ID for the id of entities #3271
• GraphQL route for dumping entity changes in subgraph and block #3275
• Firehose filters #3323
• NEAR filters #3372

Robustness

• Improve our CacheWeight estimates #2935
• Refactor GraphQL execution #3005
• Setup databases in parallel #3019
• Block ingestor now fetches receipts in parallel #3030
• Prevent subscriptions from back-pressuring the notification queue #3053
• Avoid parsing X triggers if the filter is empty #3083
• Pipeline BlockStream #3085
• More robust proofOfIndexing GraphQL route #3348

graphman

• Add run command, for running a subgraph up to a block #3079
• Add analyze command, for analyzing a PostgreSQL table, which can improve performance #3170
• Add index create command, for adding an index to certain attributes #3175
• Add index list command, for listing indexes #3198
• Add index drop command, for dropping indexes #3198

Dependency Updates

These are the main ones:

• Updated protobuf to latest version for NEAR #2947
• Update web3 crate #2916 #3120 #3338
• Update graphql-parser to v0.4.0 #3020
• Bump itertools from 0.10.1 to 0.10.3 #3037
• Bump clap from 2.33.3 to 2.34.0 #3039
• Bump serde_yaml from 0.8.21 to 0.8.23 #3065
• Bump tokio from 1.14.0 to 1.15.0 #3092
• Bump indexmap from 1.7.0 to 1.8.0 #3143
• Update ethabi to its latest version #3144
• Bump structopt from 0.3.25 to 0.3.26 #3180
• Bump anyhow from 1.0.45 to 1.0.53 #3182
• Bump quote from 1.0.9 to 1.0.16 #3112 #3183 #3384
• Bump tokio from 1.15.0 to 1.16.1 #3208
• Bump semver from 1.0.4 to 1.0.5 #3229
• Bump async-stream from 0.3.2 to 0.3.3 #3361
• Update jsonrpc-server #3313

Misc

• More context when logging RPC calls #3128
• Increase default reorg threshold to 250 for Ethereum #3308
• Improve traces error logs #3353
• Add warning and continue on parse input failures for Ethereum #3326

Upgrade Notes

When upgrading to this version, we recommend taking a brief look into these changes:

• Gas metering #2414
  • Now there's a gas limit for subgraph mappings, if the limit is reached the subgraph will fail with a non-deterministic error, you can make them recover via the environment variable GRAPH_MAX_GAS_PER_HANDLER
• Improve our CacheWeight estimates #2935
  • This is relevant because a couple of releases back we've added a limit for the memory size of a query result. That limit is based of the CacheWeight.

These are some of the features that will probably be helpful for indexers 😊

• Allow placing new deployments onto one of several shards #3049
• GraphQL route for dumping entity changes in subgraph and block #3275
• Unused deployments are automatically removed now #3023
  • The interval can be set via GRAPH_REMOVE_UNUSED_INTERVAL
• Setup databases in parallel #3019
• Block ingestor now fetches receipts in parallel #3030
  • GRAPH_ETHEREUM_FETCH_TXN_RECEIPTS_IN_BATCHES can be set to true for the old fetching behavior
• More robust proofOfIndexing GraphQL route #3348
  • A token can be set via GRAPH_POI_ACCESS_TOKEN to limit access to the POI route
• The new graphman commands 🙂

v0.25.2

This release includes two changes:

• Bug fix of blocks being skipped from processing when: a deterministic error happens and the index-node gets restarted. Issue #3236, Pull Request: #3316.
• Automatic retries for non-deterministic errors. Issue #2945, Pull Request: #2988.

This is the last patch on the 0.25 minor version, soon 0.26.0 will be released. While that we recommend updating to this version to avoid determinism issues that could be caused on graph-node restarts.

v0.25.1

This release only adds two fixes:

• The first is to address an issue with decoding the input of some calls #3194 where subgraphs that would try to index contracts related to those would fail. Now they can advance normally.
• The second one is to fix a non-determinism issue with the retry mechanism for errors. Whenever a non-deterministic error happened, we would keep retrying to process the block, however we should've clear the EntityCache on each run so that the error entity changes don't get transacted/saved in the database in the next run. This could make the POI generation non-deterministic for subgraphs that failed and retried for non-deterministic reasons, adding a new entry to the database for the POI.

We strongly recommend updating to this version as quickly as possible.

v0.25.0

Api Version 0.0.6

This release ships support for API version 0.0.6 in mappings:

• Added nonce field for Transaction objects.
• Added baseFeePerGas field for Block objects (EIP-1559).

Block Cache Invalidation and Reset

All cached block data must be refetched to account for the new Block and Trasaction
struct versions, so this release includes a graph-node startup check that will:

1. Truncate all block cache tables.
2. Bump the db_version value from 2 to 3.

(Table truncation is a fast operation and no downtime will occur because of that.)

Ethereum

• 'Out of gas' errors on contract calls are now considered deterministic errors,
  so they can be handled by try_ calls. The gas limit is 50 million.

Environment Variables

• The GRAPH_ETH_CALL_GAS environment is removed to prevent misuse, its value
  is now hardcoded to 50 million.

Multiblockchain

• Initial support for NEAR subgraphs.
• Added FirehoseBlockStream implementation of BlockStream (#2716)

Misc

• Rust docker image is now based on Debian Buster.
• Optimizations to the PostgreSQL notification queue.
• Improve PostgreSQL robustness in multi-sharded setups. (#2815)
• Added 'networks' to the 'subgraphFeatures' endpoint. (#2826)
• Check and limit the size of GraphQL query results. (#2845)
• Allow _in and _not_in GraphQL filters. (#2841)
• Add PoI for failed subgraphs. (#2748)
• Make graphman rewind safer to use. (#2879)
• Add subgraphErrors for all GraphQL schemas. (#2894)
• Add Graph-Attestable response header. (#2946)
• Add support for minimum block constraint in GraphQL queries (number_gte) (#2868).
• Handle revert cases from Hardhat and Ganache (#2984)
• Fix bug on experimental prefetching optimization feature (#2899)

v0.24.2

This release only adds a fix for an issue where certain GraphQL queries
could lead to graph-node running out of memory even on very large
systems. This release adds code that checks the size of GraphQL responses
as they are assembled, and can warn about large responses in the logs
resp. abort query execution based on the values of the two new environment
variables GRAPH_GRAPHQL_WARN_RESULT_SIZE and
GRAPH_GRAPHQL_ERROR_RESULT_SIZE. It also adds Prometheus metrics
query_result_size and query_result_max to track the memory consumption
of successful GraphQL queries. The unit for the two environment variables
is bytes, based on an estimate of the memory used by the result; it is best
to set them after observing the Prometheus metrics for a while to establish
what constitutes a reasonable limit for them.

We strongly recommend updating to this version as quickly as possible.

v0.24.1

Feature Management

This release supports the upcoming Spec Version 0.0.4 that enables subgraph features to be declared in the manifest and
validated during subgraph deployment
#2682
#2746.

Subgraphs using previous versions are still supported and won't be affected by this change.

New Indexer GraphQL query: subgraphFeatures

It is now possible to query for the features a subgraph uses given its Qm-hash ID.

For instance, the following query...

{
  subgraphFeatures(subgraphId: "QmW9ajg2oTyPfdWKyUkxc7cTJejwdyCbRrSivfryTfFe5D") {
    features
    errors
  }
}

... would produce this result:

{
  "data": {
    "subgraphFeatures": {
      "errors": [],
      "features": [
        "nonFatalErrors",
        "ipfsOnEthereumContracts"
      ]
    }
  }
}

Subraphs with any Spec Version can be queried that way.

Api Version 0.0.5

• Added better error message for null pointers in the runtime #2780.

Environment Variables

• When GETH_ETH_CALL_ERRORS_ENV is unset, it doesn't make eth_call errors to be considered determinsistic anymore #2784

Robustness

• Tolerate a non-primary shard being down during startup #2727.
• Check that at least one replica for each shard has a non-zero weight #2749.
• Reduce locking for the chain head listener #2763.

Logs

• Improve block ingestor error reporting for missing receipts #2743.

v0.24.0

0.24

Api Version 0.0.5

This release ships support for API version 0.0.5 in mappings. It contains a fix for call handlers
and the long awaited AssemblyScript version upgrade!

• AssemblyScript upgrade: The mapping runtime is updated to support up-to-date versions of the
  AssemblyScript compiler. The graph-cli/-ts releases to support this are in alpha, soon they will
  be released along with a migration guide for subgraphs.
• Call handlers fix: Call handlers will never be triggered on transactions with a failed status,
  resolving issue #2409. Done in #2511.

Logs

• The log "Skipping handler because the event parameters do not match the event signature." was downgraded from info to trace level.
• Some block ingestor error logs were upgrded from debug to info level #2666.

Metrics

• query_semaphore_wait_ms is now by shard, and has the pool and shard labels.
• deployment_failed metric added, it is 1 if the subgraph has failed and 0 otherwise.

Other

• Upgrade to tokio 1.0 and futures 0.3 #2679, the first major contribution by StreamingFast!
• Support Celo block reward events #2670.
• Reduce the maximum WASM stack size and make it configurable #2719.
• For robustness, ensure periodic updates to the chain head listener #2725.

v0.23.1

• Fix ipfs timeout detection #2584.
• Fix discrepancy between a database table and its Diesel model #2586.