v1.1.0

v1.1.0

Added

• Added 'SendAllBlockHeaders' param to CombinedFilter transform when we want to prevent skipping blocks but still want to filter out trxs

Changed

• Reduced how many times reader read statistics is displayed down to each 30s (previously each 5s) (and re-wrote log to reader node statistics).

Fixed

• Fix fireeth tools download-blocks-from-firehose tool that was not working anymore.
• Simplify forkablehub startup performance cases.
• Fix relayer detection of a hole in stream blocks (restart on unrecoverable issue).
• Fix possible panic in hub when calls to one-block store are timing out.
• Fix merger slow one-block-file deletions when there are more than 10000 of them.

v1.0.0

BREAKING CHANGES

Project rename

• The binary name has changed from sfeth to fireeth (aligned with https://firehose.streamingfast.io/references/naming-conventions)
• The repo name has changed from sf-ethereum to firehose-ethereum

Ethereum V2 blocks (with fh2-instrumented nodes)

• This will require reprocessing the chain to produce new blocks
• Protobuf Block model is now tagged sf.ethereum.type.v2 and contains the following improvements:
  • Fixed Gas Price on dynamic transactions (post-London-fork on ethereum mainnet, EIP-1559)
  • Added "Total Ordering" concept, 'Ordinal' field on all events within a block (trx begin/end, call, log, balance change, etc.)
  • Added TotalDifficulty field to ethereum blocks
  • Fixed wrong transaction status for contract deployments that fail due to out of gas on pre-Homestead transactions (aligned with status reported by chain: SUCCESS -- even if no contract code is set)
  • Added more instrumentation around AccessList and DynamicFee transaction, removed some elements that were useless or could not be derived from other elements in the structure, ex: gasEvents
  • Added support for finalized block numbers (moved outside the proto-ethereum block, to firehose bstream v2 block)
• There are no more "forked blocks" in the merged-blocks bundles:
  • The merged-blocks are therefore produced only after finality passed (before The Merge, this means after 200 confirmations).
  • One-block-files close to HEAD stay in the one-blocks-store for longer
  • The blocks that do not make it in the merged-blocks (forked out because of a re-org) are uploaded to another store (common-forked-blocks-store-url) and kept there for a while (to allow resolving cursors)

Firehose V2 Protocol

• This will require changes in most firehose clients
• A compatibility layer has been added to still support sf.firehose.v1.Stream/Blocks but only for specific values for 'ForkSteps' in request: 'irreversible' or 'new+undo'
• The Firehose Blocks protocol is now under sf.firehose.v2 (bumped from sf.firehose.v1).
  • Step type IRREVERSIBLE renamed to FINAL
  • Blocks request now only allows 2 modes regarding steps: NEW,UNDO and FINAL (gated by the final_blocks_only boolean flag)
  • Blocks that are sent out can have the combined step NEW+FINAL to prevent sending the same blocks over and over if they are already final

Block Indexes

• Removed the Irreversible indices completely (because the merged-blocks only contain final blocks now)
• Deprecated the "Call" and "log" indices (xxxxxxxxxx.yyy.calladdrsig.idx and xxxxxxxxxx.yyy.logaddrsig.idx), now replaced by "combined" index
• Moved out the sfeth tools generate-... command to a new app that can be launched with sfeth start generate-combined-index[,...]

Flags and environment variables

• All config via environment variables that started with SFETH_ now starts with FIREETH_
• All logs now output on stderr instead of stdout like previously
• Changed config-file default from ./sf.yaml to "", preventing failure without this flag.
• Renamed common-blocks-store-url to common-merged-blocks-store-url
• Renamed common-oneblock-store-url to common-one-block-store-url now used by firehose and relayer apps
• Renamed common-blockstream-addr to common-live-blocks-addr
• Renamed the mindreader application to reader
• Renamed all the mindreader-node-* flags to reader-node-*
• Added common-forked-blocks-store-url flag used by merger and firehose
• Changed --log-to-file default from true to false
• Changed default verbosity level: now all loggers are INFO (instead of having most of them to WARN). -v will now activate all DEBUG logs
• Removed common-block-index-sizes, common-index-store-url
• Removed merger-state-file, merger-next-exclusive-highest-block-limit, merger-max-one-block-operations-batch-size, merger-one-block-deletion-threads, merger-writers-leeway
• Added merger-stop-block, merger-prune-forked-blocks-after, merger-time-between-store-pruning
• Removed mindreader-node-start-block-num, mindreader-node-wait-upload-complete-on-shutdown, mindreader-node-merge-and-store-directly, mindreader-node-merge-threshold-block-age
• Removed firehose-block-index-sizes,firehose-block-index-sizes, firehose-irreversible-blocks-index-bundle-sizes, firehose-irreversible-blocks-index-url, firehose-realtime-tolerance
• Removed relayer-buffer-size, relayer-merger-addr, relayer-min-start-offset

MIGRATION

Clients

• If you depend on the proto file, update import "sf/ethereum/type/v1/type.proto" to import "sf/ethereum/type/v2/type.proto"
• If you depend on the proto file, update all occurrences of sf.ethereum.type.v1.<Something> to sf.ethereum.type.v2.<Something>
• If you depend on sf-ethereum/types as a library, update all occurrences of github.com/streamingfast/firehose-ethereum/types/pb/sf/ethereum/type/v1 to github.com/streamingfast/firehose-ethereum/types/pb/sf/ethereum/type/v2.

Server-side

Deployment

• The reader requires Firehose-instrumented Geth binary with instrumentation version 2.x (tagged fh2)
• Because of the changes in the ethereum block protocol, an existing deployment cannot be migrated in-place.
• You must deploy firehose-ethereum v1.0.0 on a new environment (without any prior block or index data)
• You can put this new deployment behind a GRPC load-balancer that routes /sf.firehose.v2.Stream/* and /sf.firehose.v1.Stream/* to your different versions.
• Go through the list of changed "Flags and environment variables" and adjust your deployment accordingly.
  • Determine a (shared) location for your forked-blocks.
  • Make sure that you set the one-block-store and forked-blocks-store correctly on all the apps that now require it.
  • Add the generate-combined-index app to your new deployment instead of the tools command for call/logs indices.
• If you want to reprocess blocks in batches while you set up a "live" deployment:
  • run your reader node from prior data (ex: from a snapshot)
  • use the --common-first-streamable-block flag to a 100-block-aligned boundary right after where this snapshot starts (use this flag on all apps)
  • perform batch merged-blocks reprocessing jobs
  • when all the blocks are present, set the common-first-streamable-block flag to 0 on your deployment to serve the whole range

Producing merged-blocks in batch

• The reader requires Firehose-instrumented Geth binary with instrumentation version 2.x (tagged fh2)
• The reader does NOT merge block files directly anymore: you need to run it alongside a merger:
  • determine a start and stop block for your reprocessing job, aligned on a 100-blocks boundary right after your Geth data snapshot
  • set --common-first-streamable-block to your start-block
  • set --merger-stop-block to your stop-block
  • set --common-one-block-store-url to a local folder accessible to both merger and mindreader apps
  • set --common-merged-blocks-store-url to the final (ex: remote) folder where you will store your merged-blocks
  • run both apps like this fireeth start reader,merger --...
• You can run as many batch jobs like this as you like in parallel to produce the merged-blocks, as long as you have data snapshots for Geth that start at this point

Producing combined block indices in batch

• Run batch jobs like this: fireeth start generate-combined-index --common-blocks-store-url=/path/to/blocks --common-index-store-url=/path/to/index --combined-index-builder-index-size=10000 --combined-index-builder-start-block=0 [--combined-index-builder-stop-block=10000] --combined-index-builder-grpc-listen-addr=:9000

Other (non-breaking) changes

Added tools and apps

• Added tools firehose-client command with filter/index options
• Added tools normalize-merged-blocks command to remove forked blocks from merged-blocks files (cannot transform ethereum blocks V1 into V2 because some fields are missing in V1)
• Added substreams server support in firehose app (alpha) through --substreams-enabled flag

Various

• The firehose GRPC endpoint now supports requests that are compressed using gzip or zstd
• The merger does not expose PreMergedBlocks endpoint over GRPC anymore, only HealthCheck. (relayer does not need to talk to it)
• Automatically setting the flag --firehose-genesis-file on reader nodes if their reader-node-bootstrap-data-url config value is sets to a genesis.json file.
• Note to other Firehose implementors: we changed all command line flags to fit the required/optional format referred to here: https://en.wikipedia.org/wiki/Usage_message
• Added prometheus boolean metric to all apps called 'ready' with label 'app' (firehose, merger, mindreader-node, node, relayer, combined-index-builder)

v0.10.2

Changelog

Published the first Substreams Ethereum package.

• ee80996 Fix S3 walking empty filename which causes problem to downstream dependencies like the merger
• f0a8987 Updated for release v0.10.2

v0.10.1

Changelog

v0.10.1

• Fix panic in merger when no prior merged blocks exist

v0.10.0 (deleted)

Fixed

• Fixed an issue where the merger would get stuck when too many (more than 2000) one-block-files were lying around, with block numbers below the current bundle high boundary.
• Fixed tools check merged-blocks block hole detection behavior on missing ranges (bumped sf-tools)
• Fixed a deadlock issue related to s3 storage error handling (bumped dstore)
• Fixed Firehose gRPC listening address over plain text.
• Fixed automatic merging of files within the mindreader is much more robust then before.

Changed

• Renamed common atm 4 flags to blocks-cache:
  --common-blocks-cache-{enabled|dir|max-recent-entry-bytes|max-entry-by-age-bytes}
• The default text encoder use to encode log entries now emits the level when coloring is disabled.
• Default value for flag --mindreader-node-enforce-peers is now "", this has been changed because the default value was useful only in development when running a local node-manager as either the miner or a peering node.
• Updated all Firehose stack direct dependencies.
• Updated confusing flag behavior for --common-system-shutdown-signal-delay and its interaction with gRPC connection draining in firehose component sometimes preventing it from shutting down.
• Reporting an error is if flag merge-threshold-block-age is way too low (< 30s).

Added

• Added tools download-from-firehose command to fetch blocks and save them as merged-blocks files locally.
• Added cloud-gcp:// auth module (bumped dauth)
• Added block data file caching (called ATM), this is to reduce the memory usage of component keeping block objects in memory.
• Added transforms: LogFilter, MultiLogFilter, CallToFilter, MultiCallToFilter to only return transaction traces that match logs or called addresses.
• Added support for irreversibility indexes in firehose to prevent replaying reorgs when streaming old blocks.
• Added support for log and call indexes to skip old blocks that do not match any transform filter.

Removed

• Removed some old components that are not required by Firehose stack directly, the repository is as lean as it ca now.

v0.10.0-rc.5

Changed

• Renamed common atm 4 flags to blocks-cache:
  --common-blocks-cache-{enabled|dir|max-recent-entry-bytes|max-entry-by-age-bytes}

Fixed

• Fixed tools check merged-blocks block hole detection behavior on missing ranges (bumped sf-tools)
• Fixed a deadlock issue related to s3 storage error handling (bumped dstore)

Added

• Added tools download-from-firehose command to fetch blocks and save them as merged-blocks files locally.
• Added cloud-gcp:// auth module (bumped dauth)

v0.10.0-rc.4

Changelog

• 2b9183e Changed default flag value for --mindreader-node-enforce-peers to ""

v0.10.0-rc.1

Added

• Added block data file caching (called ATM), this is to reduce the memory usage of component keeping block objects in memory.
• Added transforms: LogFilter, MultiLogFilter, CallToFilter, MultiCallToFilter to only return transaction traces that match logs or called addresses.
• Added support for irreversibility indexes in firehose to prevent replaying reorgs when streaming old blocks.
• Added support for log and call indexes to skip old blocks that do not match any transform filter.

Changed

• Updated all Firehose stack direct dependencies.
• Updated confusing flag behavior for --common-system-shutdown-signal-delay and its interaction with gRPC connection draining in firehose component sometimes preventing it from shutting down.
• Reporting an error is if flag merge-threshold-block-age is way too low (< 30s).

Removed

• Removed some old components that are not required by Firehose stack directly, the repository is as lean as it ca now.

Fixed

• Fixed Firehose gRPC listening address over plain text.
• Fixed automatic merging of files within the mindreader is much more robust then before.

Full Changelog

• 3ac8ec4 Add NewEthBlockIndexer() for convenience
• 6273adf Add TestEthBlockIndexProvider_Matches()
• ad4e41d Add TestEthBlockIndexProvider_NextMatching()
• 44eb852 Add TestEthBlockIndexProvider_WithinRange()
• d21f4ac Add docstrings to EthBlockIndexer
• 85238a3 Add multi-log filter transform
• 57c299a Add stubs for BlockIndexProvider
• d341b4a Added changelog for v0.10.0-rc.1
• ad161f7 Added git push prior release to ensure GitHub created release creates the correct tag
• d1f1751 Bump node-manager (moving some backup related stuff in there)
• 633a782 Extract WriteIndex() and ReadIndex() to bstream
• 52cd6af Extract chain-agnostic ProcessBlock() logic to bstream
• dbb5101 Fixed release script when doing a real non dry run
• 58042c9 Implement Index filter on eth.Address and eth.Hash
• 2ccd1ae Instantiating ProtocolSpefificModules
• 6954b53 Merge pull request #24 from streamingfast/stepd/multi-filter
• c29b0e7 Merge pull request #25 from streamingfast/froch/block-indexer
• d96cf3a Merge remote-tracking branch 'origin/develop' into froch/block-indexer
• 52871e4 Remove LogAddressIndex implementations in favor of BlockIndex
• 4a299ae Removed blockmeta and trxdb from sf-ethereum public
• 6a2cc3a Removing snapshotter dependency
• 02e63d7 Replace NewLogAddressIndexProvider() with NewEthBlockIndexProvider()
• e42aeb5 Replace NewLogAddressIndexer() with NewEthBlockIndexer()
• 166858e Rework EthBlockIndexProvider to accept a filterFunc
• 6c3fe6a Sanitize go.mod a bit
• 7cfb743 Updated bunch of dependencies to get proper logger name
• 0f2b0f4 Updated dlauncher to latest version
• 31e4f56 Updated to correct released version
• af40cee Updated to latest dlauncher
• ff0fc01 Updated to latest dlauncher and logging library
• e528270 Updated to latest version of bstream, pbgo, dgrpc, relayer, merger and blockmeta
• c4b08b0 Updated to latest version of everything
• da46fb0 Updated to streamingfast/overseer
• 0583610 Wrap a bstream.transform.BlockIndexer as an EthBlockIndexer
• 34d9054 account indexing tool to also produce irreversible blocks and start from where it left off
• bfd49b2 add proper logging of log/call filter transforms
• fc70c15 adjust accoutn_indexer to start from lowest of irreversible/account index
• 21aca49 bump bstream for startblock index matching fix
• f73cd60 bump bstream so firehose always sends start-block even when filtered
• 321e255 bump bstream to fix firehose cursor on indexed filesource
• 7b4bbbf bump bstream with my refactors on froch PR
• add26a0 bump bstream, firehose to fix stop-block bug in indexes
• 738745b bump deps to fix mindreader merging directly
• c136aa6 bump node-manager to fix double-compressed oneblockfiles
• 30bde30 bump node-manager to fix file deletion
• 75649ed bump node-manager/merger to fix direct merging
• 878a640 bump node-manager: merge immediately on boundary
• 29a8fbb change 'transforms' to 'transform' in proto, bump some index libs
• 24ac3a1 fix confusing flag behavior for grpcShutdownGracePeriod sometimes preventing firehose from shutting down
• 50787c9 fix logfilter event matching with no topic
• 35d5311 go mod tidy
• b3725d9 iadd MultiCallToFilter transform and all that jazz
• a0e3e44 opiniated failure if merge-threshold-block-age is way too low streamingfast/node-manager#23
• 443639f remove firehose flag to gen irr.index, bump libs
• 9465b6b simplify logic in log_address_index

v0.9.0

Changelog

• 930ded3 Add TestBasicLogFilter_GetIndexProvider()
• cde9fb5 Add TestLogAddressIndexProvider_LoadRange()
• 51e6c2a Add TestLogAddressIndexProvider_Matches()
• da5c7c2 Add TestLogAddressIndexProvider_NextMatching()
• aae09d7 Add TestLogAddressIndexProvider_WithinRange()
• 428d1bc Add another case for TestLogAddressIndexProvider_NextMatching()
• cb30473 Add blockNum as log field in findIndexContaining()
• 538a4b5 Add docstring to test fixture
• 3baa7a4 Add guards for GetIndexProvider instantiation
• c4d7b34 Add missing case for TestLogAddressIndexProvider_NextMatching()
• e784dd1 Add more details to docstrings
• b139474 Add stubs for LogAddressIndexProvider
• 5b8c23a Add support for LogAddressIndex (#17)
• bfa3f76 Added .vscode to ignore list
• 4f6db8d Added CI config, added release script and config, updated version printing
• 5f2901e Added Receipt in TrxTrace when creating a new transaction trace
• f285ef5 Added missing Burn balance change reason
• 4efe69e Adjusted instructions
• 4065b9b Adjusted now removed -rpc flags
• a2d34c6 Bump snapshotter
• 6b42571 Bumpd node-manager to latest version
• 042e1c9 Bumped bstream & firehose to latest version
• f02ef9f Bumped bstream + firehose
• fe4994c Bumped dstore to latest version
• 321ad4b Bumped firehose
• f092458 Bumped firehose
• dd5113a Bumped firehose & bstream
• 05e7e7c Bumped firehose to latest version
• a4c342d Bumped node-manager and improved logging when running init command
• d298b78 Bumping to a more recent snapshotter
• 3338f5e Change method signatures for Matches() and NextMatching()
• 10caef1 Cherry-pick stepd/wip
• 7cb85bb Event log transform
• 70fc4aa First pass - add a LogAddressIndexProvider
• e28c17a Fix BasicLogFilter and added testing
• 3acfe9b Fix FindIndexContaining_LoadIndex()
• 8fef352 Fix blockstream flag
• 794e2df Fixed --deep-mind flags that are now prefixed with --firehose-deep-mind
• d11686b Fixed pbcodec.Log#BlockIndex that is wrong when fetched from transactionTraces.calls
• 1bc6aff Fixed generation of proto files
• 0496a3d Fixed go.sum
• 32f88d9 Fixed pb/dfuse not move correctly to pb/sf
• dbb5101 Fixed release script when doing a real non dry run
• 6205bad Fixed some small layout issue in motd
• 9ed057a Flesh-out LogAddressIndexProvider interface methods
• fb13f31 Implement GetIndexProvider for Transform / BasicLogFilter
• 2efada2 Implement GetIndexProvider for Transform / BasicLogFilter (#21)
• 53a4ea6 Increase indexOpsTimeout to 15 seconds
• 9873294 Initial commit
• 67bf8df Marshal / Unmarshal a LogAddressIndex from dstore
• e487c01 Marshal index file and write-out to dstore
• bb7e801 Merge branch 'develop' into stepd/irreversibility_idx
• b2b0215 Merge branch 'stepdrename' into develop
• 6f0aaf1 Merge pull request #14 from streamingfast/stepd/irreversibility_idx
• acd0e9b Merge pull request #19 from streamingfast/froch/wip
• 6bc5ef8 Moved check merged blocks logic to a shared library sf-tools
• 3aeb546 Moved versionning stuff into cli package
• 799676d Re-added Matches and Call Matches implementation
• 1a90c1b Re-generated codec protobuf
• 520bedb Reduce FindIndexContaining_LoadIndex() to a single test case
• ced9693 Remove NewLogAddressSignatureIndex()
• ac671a3 Remove deprecated comments relative to IndexProvider interface
• c1d7b1c Remove vestigial len() check in test fixture
• 433d127 Renamed deth to codec
• 269b537 Renamed scripts header
• f4c71f9 Replace dfuse-io/rpc by streamingfast/rpc
• 375b7e7 Replacement are now grouped to read better
• e08381a Rework LogAddressIndex
• 8f98820 Separate logAddressIndex and logAddressIndexer
• 859aaff Set require.False(t, done) on NextMatching result
• 1d6a9ad Try something else to remove snapshotter dependency error
• 4acfb1a Turn snapshot without compatible eth support into a debug log
• feab03b Updated logging to latest version
• cbaec85 Updated merger to latest
• 1702100 Updated sf-tools for better hole catching in summary printing
• 2963bbe add 'tools generate-irreversible-index' and parametric bundle-sizes to firehose
• 3aa7bf4 add NewLogADdressIndex
• b8a7b14 add draft implementation of log_address_index
• 3320d36 add logADdressIndex matchingBlocks impl
• d366261 add log_address_index provider from transform
• b3b5b55 add status to trxTrace in decoding proto
• 1cfc5cd add transform/indexes from proto defs
• 7874ab8 adjust merger params bump bstream, relayer, nodemanager
• 84e8452 bump bstream and firehose to feature/better_clone_handling
• b353a6f bump bstream for firehose irrIndex fixes
• 1b60819 bump bstream for firehose typo
• e830545 bump bstream for irreversible index creation on first streamable block
• ea92e86 bump bstream for some cleanups
• dca87fe bump bstream to feature/better_clone_handling
• 101fc38 bump bstream to feature/better_clone_handling
• d827714 bump bstream to feature/better_clone_handling
• 6b3e98c bump bstream to feature/better_clone_handling
• 3ddd9a8 bump bstream to feature/better_clone_handling
• c832acc bump bstream to feature/better_clone_handling
• 9455ccf bump bstream to feature/better_clone_handling
• 6f4aa03 bump bstream to feature/better_clone_handling
• 541cbf9 bump bstream to fix firehose CURSOR issues (cursor without lib)
• 0f57a42 bump bstream to fix last (incorrectly fixed) typo
• 64d1b02 bump bstream to fix new memoize/clone, add ATM cache
• 80da1bd bump bstream to fix/index-perf, adjust logaddr_idx
• 4b6f143 bump bstream to fix/index-perf, adjust logaddr_idx (#22)
• d8da678 bump bstream to hotfix atm filesource
• e570734 bump bstream to merged version of PR
• 9d1877d bump bstream, fix atm panic
• cb0ec8b bump bstream, fix some startblocks etc.
• ac0e774 bump bstream, no ProtoGenesisBlk, fixed cursors
• ee03862 bump bstream,firehose, add irreversible_blocks_index partial impl.
• df9c41f bump bstream/firehose after merge
• d4e2b1e bump bstream/firehose irreversibility_idx
• 14d6660 bump firehose and bstream to latest develop
• fb3c9c6 bump firehose for transforms/preproc fix
• 26c45b0 bump firehose, bstream
• 100a2a4 bump merger
• 598b552 bump merger and bstream
• 65e0b32 bump merger fix toOneBlockFile error handling
• c66cbf7 bump merger merged file upload fix
• bb6784b bump merger to drift metrics
• 53df86d bump merger to fix bootstrap
• 22f9145 bump merger with better log
• 1fce895 bump merger with bootstrap from merged files
• e3f2480 bump merger with fixed bootstrap
• dde40e9 bump merger with new bootstrap from lib
• 848d762 bump pbgo and bstream, proto rename from dfuse to sf
• 53b3c64 continue prototype log address index
• 299415e fix WithinRange inefficiencies
• d95a4eb fix log_filter to use Receipt and skip failed logs
• 64b38bc implement LogAddressIndexProvider.findIndexContaining()
• 8f8e350 implement LogAddressIndexProvider.loadIndex()
• 9c1bea3 log in log_address_indexer tool
• 81bdc9a logAddrIdxProvider fix New() assumptions
• d166d6d modify log_address_indexer to work with ethblock not bstreamblock
• 5f4c1dc rebase on develop with transforms
• 775af98 remove admin,debug rpc on default geth flags, nousb and lachesis references
• cbfb9ee remove unsued merger flag 'merger-max-fixable-fork'
• a5c33a2 rename proto index LogAddressMethod to LogAddressSignature
• 90e77d0 wip, debugging test cases with eth blocks
• 8095aac work in progress for filter fix, does not compile