v1.3.7

Fixed

• Fixed a bug in data normalization for Polygon chain which would cause panics on certain blocks

Added

• support for gcp 'archive' types of snapshots

v1.3.6

Highlights

• This release implements the new CANCEL_BLOCK instruction from Firehose protocol 2.2 (fh2.2), to reject blocks that failed post-validation.
• This release fixes polygon "StateSync" transactions by grouping the calls inside an artificial transaction.

If you had previous blocks from a Polygon chain (bor), you will need to reprocess all your blocks from the node because some StateSync transactions may be missing on some blocks.

Operators

This release now supports the new Firehose node exchange format 2.2 which introduced a new exchanged message CANCEL_BLOCK. This has an implication on the Firehose instrumented Geth binary you can use with the release.

• If you use Firehose instrumented Geth binary tagged fh2.2 (like geth-v1.11.4-fh2.2-1), you must use firehose-ethereum version >= 1.3.6
• If you use Firehose instrumented Geth binary tagged fh2.1 (like geth-v1.11.3-fh2.1), you can use firehose-ethereum version >= 1.0.0

New releases of Firehose instrumented Geth binary for all chain will soon all be tagged fh2.2, so upgrade to >= 1.3.6 of firehose-ethereum will be required.

v1.3.5

Highlights

This release is required if you run on Goerli and is mostly about supporting the upcoming Shangai fork that has been activated on Goerli on March 14th.

Changed

• Added support for withdrawal balance change reason in block model, this is required for running on most recent Goerli Shangai hard fork.
• Added support for withdrawals_root on Header in the block model, this will be populated only if the chain has activated Shangai hard fork.
• --substreams-max-fuel-per-block-module will limit the number of wasmtime instructions for a single module in a single block.

v1.3.4

Highlights

Fixed the 'upgrade-merged-blocks' from v2 to v3

Blocks that were migrated from v2 to v3 using the 'upgrade-merged-blocks' should now be considered invalid.
The upgrade mechanism did not correctly fix the "caller" on DELEGATECALLs when these calls were nested under another DELEGATECALL.

You should run the upgrade-merged-blocks again if you previously used 'v2' blocks that were upgraded to 'v3'.

Backoff mechanism for bursts

This mechanism uses a leaky-bucket mechanism, allowing an initial burst of X connections, allowing a new connection every Y seconds or whenever an existing connection closes.

Use --firehose-rate-limit-bucket-size=50 and --firehose-rate-limit-bucket-fill-rate=1s to allow 50 connections instantly, and another connection every second.
Note that when the server is above the limit, it waits 500ms before it returns codes.Unavailable to the client, forcing a minimal back-off.

Fixed

• Substreams RpcCall object are now validated before being performed to ensure they are correct.
• Substreams RpcCall JSON-RPC code -32602 is now treated as a deterministic error (invalid request).
• tools compare-blocks now correctly handle segment health reporting and properly prints all differences with -diff.
• tools compare-blocks now ignores 'unknown fields' in the protobuf message, unless --include-unknown-fields=true
• tools compare-blocks now ignores when a block bundle contains the 'last block of previous bundle' (a now-deprecated feature)

Added

• support for "requester pays" buckets on Google Storage in url, ex: gs://my-bucket/path?project=my-project-id
• substreams were also bumped to current March 1st develop HEAD

v1.3.3

Changed

• Increased gRPC max received message size accepted by Firehose and Substreams gRPC endpoints to 25 MiB.

Removed

• Command fireeth init has been removed, this was a leftover from another time and the command was not working anyway.

Added

• flag common-auto-max-procs to optimize go thread management using github.com/uber-go/automaxprocs
• flag common-auto-mem-limit-percent to specify GOMEMLIMIT based on a percentage of available memory

v1.3.2

Updated

• Updated to Substreams version v0.2.0 please refer to release page for further info about Substreams changes.

Changed

• Breaking Config value substreams-stores-save-interval and substreams-output-cache-save-interval have been merged together as a single value to avoid potential bugs that would arise when the value is different for those two. The new configuration value is called substreams-cache-save-interval.

  • To migrate, remove usage of substreams-stores-save-interval: <number> and substreams-output-cache-save-interval: <number> if defined in your config file and replace with substreams-cache-save-interval: <number>, if you had two different value before, pick the biggest of the two as the new value to put. We are currently setting to 1000 for Ethereum Mainnet.

Fixed

• Fixed various issues with fireeth tools check merged-blocks
  • The stopWalk error is not reported as a real error anymore.
  • Incomplete range should now be printed more accurately.

v1.3.1

Release made to fix our building workflows, nothing different than v1.3.0.

Commits

v1.3.0...v1.3.1

v1.3.0

Changed

• Updated to Substreams v0.1.0, please refer to release page for further info about Substreams changes.

  Warning The state output format for map and store modules has changed internally to be more compact in Protobuf format. When deploying this new version and using Substreams feature, previous existing state files should be deleted or deployment updated to point to a new store location. The state output store is defined by the flag --substreams-state-store-url flag.

Added

• New Prometheus metric console_reader_trx_read_count can be used to obtain a transaction rate of how many transactions were read from the node over a period of time.

• New Prometheus metric console_reader_block_read_count can be used to obtain a block rate of how many blocks were read from the node over a period of time.

• Added --header-only support on fireeth tools firehose-client.

• Added HeaderOnly transform that can be used to return only the Block's header a few top-level fields Ver, Hash, Number and Size.

• Added fireeth tools firehose-prometheus-exporter to use as a client-side monitoring tool of a Firehose endpoint.

Deprecated

• Deprecated LightBlock is deprecated and will be removed in the next major version, its goal is now much better handled by CombineFilter transform or HeaderOnly transform if you required only Block's header.

Commits

v1.2.2...v1.3.0

v1.2.2

Substreams improvements

Performance

• Changed cache file format for stores and outputs (faster with vtproto) -- requires removing the existing state files
• Various improvements to scheduling

Fixes

• Fixed eth_call handler not flagging out of gas error as deterministic.
• Fixed Memory leak in wasmtime
• • Hotfix 'nil pointer' panic when saving uninitialized cache (from unreleased v1.2.1 tag)

Merger fixes

• Removed the unused 'previous' one-block in merged-blocks (99 inside bundle:100)
• Fix: also prevent rare bug of bundling "very old" one-blocks in merged-blocks

v1.2.0

Added

• Added sf.firehose.v2.Fetch/Block endpoint on firehose, allows fetching single block by num, num+ID or cursor.
• Added tools firehose-single-block-client to call that new endpoint.

Changed

• Renamed tools normalize-merged-blocks to upgrade-merged-blocks.

Fixed

• Fixed common-blocks-cache-dir flag's description.
• Fixed DELEGATECALL's caller (a.k.a from). -> requires upgrade of blocks to version: 3

Upgrade Procedure

Assuming that you are running a firehose deployment v1.1.0 writing blocks to folders /v2-oneblock, /v2-forked and /v2,
you will deploy a new setup that writes blocks to folders /v3-oneblock, v3-forked and /v3

This procedure describes an upgrade without any downtime. With proper parallelization, it should be possible to complete this upgrade within a single day.

1. Launch a new reader with this code, running instrumented geth binary: https://github.com/streamingfast/go-ethereum/releases/tag/geth-v1.10.25-fh2.1
   (you can start from a backup that is close to head)
2. Upgrade your merged-blocks from version: 2 to version: 3 using fireeth tools upgrade-merged-blocks /path/to/v2 /path/to/v3 {start} {stop}
   (you can run multiple upgrade commands in parallel to cover the whole blocks range)
3. Create combined indexes from those new blocks with fireeth start combined-index-builder
   (you can run multiple commands in parallel to fill the block range)
4. When your merged-blocks have been upgraded and the one-block-files are being produced by the new reader, launch a merger
5. When the reader, merger and combined-index-builder caught up to live, you can launch the relayer(s), firehose(s)
6. When the firehoses are ready, you can now switch traffic to them.