v2.11.1

Substreams (v1.14.1)

• Fix another cannot resolve 'old cursor' from files in passthrough mode -- not implemented bug when receiving a request in production-mode with a cursor that is below the "linear handoff" block

v2.11.0

Substreams (v1.14.0)

Reconnection time

• Implement "QuickSave" feature to save the state of "live running" substreams stores when shutting down, and then resume processing from that point if the cursor matches.
  • Added flag substreams-tier1-quicksave-store to enable quicksave when non-empty
    (requires --common-system-shutdown-signal-delay to be set to a long enough value to save the in-flight stores)

Performance

• Rust modules will now be executed with wasmtime by default instead of wazero.
  • Prevents the whole server from stalling in certain memory-intensive operations in wazero.
  • Speed improvement: cuts the execution time in half in some circumstances.
  • Wazero is still used for modules with wbindgen and modules compiled with tinygo.
  • Set env var SUBSTREAMS_WASM_RUNTIME=wazero to revert to previous behavior.

v2.10.0

Block Model

The Ethereum block model has been updated to account for upcoming Prague fork. Namely, we added support for the new SetCode transaction's
type, added extracted SetCodeAuthorization elements from the transaction and added new gas changes that were introduced in the hard fork.

Also, totalDifficulty field is now deprecated, it has been removed entirely from geth codebase which means future reprocessing of
data wouldn't be able to populated that field anymore. If you used that field somehow, you should stop using it. At some point
we will remove the field entirely.

Also, from Prague hard-fork and onward, the Block model will now switch to version 4 of the block model (a.k.a Firehose Ethereum Block 3.0).
This means that for a given network, block.number < Prague, block will be using version 3 (a.k.a Firehose Ethereum Block 2.3) and when
block.number >= Prague, it will be version 4. This is deterministic per network as the Prague block is deterministic.

This does not change at the structure of the various element, everything stays the same in that aspect so the version 4 model is
backward compatible. What the new version changes:

• Does not populate accountCreations field anymore, this was bogus from day 1 and should never be used.
• Fix executedCode field to be more accurate now, as soon as one opcode is executed, this will be set now and not otherwise.
• The root's call BeginOrdinal is now fixed and not always 0.
• Ordinals in presence of system calls are now correctly ordered.
• The returnData is now properly populated.
• The keccakPreimage data being "." is now fixed.
• The call's input field is now properly populated on contract creation, it was omitted before.
• There is new gas changed behind recorded now mainly for full view of how gas is allocated, consumed and returned.

Reader Node

For upcoming Prague hard forks (BNB, Holesky, Sepolia, Mainnet), you will start using geth Firehose 3.0 version, so our Firehose enabled
releases suffixed with -fh3.0.

This new Firehose 3.0 geth tracer is built on the new geth Core Tracing API introduced in Geth 1.14. This new version changes how
one must start the geth binary.

So for Holesky hard-fork, you will need to use https://github.com/streamingfast/go-ethereum/releases/tag/geth-v1.15.2-fh3.0, here what
you need when you will update your reader-node's reader-node-arguments field:

• Remove --firehose-enabled and any flag starting with --firehose-....
• Add --vmtrace=firehose flag which activates Firehose output (Important do not miss this change, otherwise you will not process new blocks, will make it the default soon).
• Add --syncmode=full flag which is not set automatically anymore.

Substreams v1.13.0

Capacity Management

• Integrated the GlobalRequestPool service in the Tier1App to manage global requests pooling.

• Integrated the GlobalWorkerPool service in the Tier1App to manage global worker pooling.

• Added flag substreams-tier1-global-worker-pool-address, the address of the global worker pool to use for the substreams tier1. (disabled if empty).

• Added flag substreams-tier1-global-worker-pool-keep-alive-delay delay between two keep alive call to the global worker pool (default is 25s").

• Added flag substreams-tier1-global-request-pool-keep-alive-delay delay between two keep alive call to the global worker pool for request (default is 25s).

• Added flag substreams-tier1-default-max-request-per-user default max request per user, this will be use of the global worker pool is not reachable (default is 5).

• Added flag substreams-tier1-default-minimal-request-life-time-second default minimal request life time, this will be use of the global worker pool is not reachable (default is 180).

• Limit parallel execution of a stage's layer: Previously, the engine was executing modules in a stage's layer all in parallel.
  We now change that behavior, development mode will from now on execute every sequentially and when in production mode will
  limit parallelism to 2 (hard-coded) for now.
  The auth plugin can control that value dynamically by providing a trusted header X-Sf-Substreams-Stage-Layer-Parallel-Executor-Max-Count.

Performance

• Fixed a regression since "v1.7.3" where the SkipEmptyOutput instruction was ignored in substreams mappers
• Add shared cache for tier1 execution near HEAD, to prevent multiple tier1 instances from reprocessing the same module on the same block when it comes in (ex: foundational modules)
• Improved fetching of state caches on tier1 requests to speed up "time to first data"

Tools

• make 'compare-blocks' command support one-blocks stores as well as merged-blocks

v2.9.3

• Fixed fireeth tools geth enforce-peers --once shorthand flag registration now collapsing with fireeth tools -o (for --output).

  This means the fireeth tools geth enforce-peers command does not accept -o anymore for once and if you were using it, replace with --once.

v2.9.2

• Fixed substreams-tier2 not setting itself ready correctly on startup since v2.9.0.

• Added support for --output=bytes mode which prints the chain's specific Protobuf block as bytes, the encoding for the bytes string printed is determined by --bytes-encoding, uses hex by default.

• Added back -o as shorthand for --output in firecore tools ... sub-commands.

v2.9.1

• Add back grpc.health.v1.Health service to firehose and substreams-tier1 services (regression in 2.9.0)
• Give precedence to the tracing header X-Cloud-Trace-Context over Traceparent to prevent user systems' trace IDs from leaking passed a GCP load-balancer

v2.9.0

Reader

• Reader Node Manager HTTP API now accepts POST http://localhost:10011/v1/restart<?sync=true> to restart the underlying reader node binary sub-process. This is a alias for /v1/reload.

Tools

• Enhanced fireeth tools print merged-blocks with various small quality of life improvements:
  • Now accepts a block range instead of a single start block.
  • Passing a single block as the block range will print this single block alone.
  • Block range is now optional, defaulting to run until there is no more files to read.
  • It's possible to pass a merged blocks file directly, with or without an optional range.

Firehose

Important

This release will reject firehose connections from clients that don't support GZIP or ZSTD compression. Use --firehose-enforce-compression=false to keep previous behavior, then check the logs for incoming Substreams Blocks request logs with the value compressed: false to track users who are not using compressed HTTP connections.

Important

This release removes the old sf.firehose.v1 protocol (replaced by sf.firehose.v2 in 2022, this should not affect any reasonably recent client).

• Add support for ConnectWeb firehose requests.
• Always use gzip compression on firehose requests for clients that support it (instead of always answering with the same compression as the request).

Substreams

• The substreams-tier1 app now has two new configuration flags named respectively substreams-tier1-active-requests-soft-limit and substreams-tier1-active-requests-hard-limit
  helping better load balance active requests across a pool of tier1 instances.

  The substreams-tier1-active-requests-soft-limit limits the number of client active requests that a tier1 accepts before starting
  to be report itself as 'unready' within the health check endpoint. A limit of 0 or less means no limit.

  This is useful to load balance active requests more easily across a pool of tier1 instance. When the instance reaches the soft
  limit, it will start to be unready from the load balancer standpoint. The load balancer in return will remove it from the list
  of available instances, and new connections will be routed to remaining clients, spreading the load.

    The `substreams-tier1-active-requests-hard-limit` limits the number of client active requests that a tier1 accepts before
  

  rejecting incoming gRPC requests with 'Unavailable' code and setting itself as unready. A limit of 0 or less means no limit.

  This is useful to prevent the tier1 from being overwhelmed by too many requests, most client auto-reconnects on 'Unavailable' code
  so they should end up on another tier1 instance, assuming you have proper auto-scaling of the number of instances available.

• The substreams-tier1 app now exposes a new Prometheus metric substreams_tier1_rejected_request_counter that tracks rejected
  requests. The counter is labelled by the gRPC/ConnectRPC returned code (ok and canceled are not considered rejected requests).

• The substreams-tier2 app now exposes a new Prometheus metric substreams_tier2_rejected_request_counter that tracks rejected
  requests. The counter is labelled by the gRPC/ConnectRPC returned code (ok and canceled are not considered rejected requests).

• Properly accept and compress responses with gzip for browser HTTP clients using ConnectWeb with Accept-Encoding header

• Allow setting subscription channel max capacity via SOURCE_CHAN_SIZE env var (default: 100)

v2.8.4

Substreams

• Fix an issue preventing proper detection of gzip compression when multiple headers are set (ex: python grpc client)
• Fix an issue preventing some tier2 requests on last-stage from correctly generating stores. This could lead to some missing "backfilling" jobs and slower time to first block on reconnection.
• Fix a thread leak on cursor resolution resulting in bad counter for active connections
• Add support for zstd encoding on server

v2.8.3

Note

This release will reject connections from clients that don't support GZIP compression. Use --substreams-tier1-enforce-compression=false to keep previous behavior, then check the logs for incoming Substreams Blocks request logs with the value compressed: false to track users who are not using compressed HTTP connections.

• Fix broken tools poller command in v2.8.2

v2.8.2

Warning

Do NOT use this version with tools poller, a flag issue prevents the poller from starting up. Recommended that you upgrade to v2.8.3 ASAP

Note

This release will reject connections from clients that don't support GZIP compression. Use --substreams-tier1-enforce-compression=false to keep previous behavior, then check the logs for incoming Substreams Blocks request logs with the value compressed: false to track users who are not using compressed HTTP connections.

• Bump firehose-core to v1.6.8
• Substreams: add --substreams-tier1-enforce-compression to reject connections from clients that do not support GZIP compression
• Substreams performance: reduced the number of mallocs (patching some third-party libraries)
• Substreams performance: removed heavy tracing (that wasn't exposed to the client)
• Fixed --reader-node-line-buffer-size flag that was not being respected in reader-node-stdin app
• poller: add --max-block-fetch-duration