Skip to content

Commit

Permalink
add 'substreams-tier1-quicksave-store' and bump substreams for quicks…
Browse files Browse the repository at this point in the history
…ave support
  • Loading branch information
sduchesneau committed Feb 25, 2025
1 parent 626a1bb commit bad7e2c
Show file tree
Hide file tree
Showing 4 changed files with 12 additions and 6 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ If you were at `firehose-core` version `1.0.0` and are bumping to `1.1.0`, you s

### Substreams

#### Reconnection time

* Added flag `substreams-tier1-quicksave-store` to enable quicksave of stores on tier1, allowing for a fast reconnection of clients using stores.

#### Capacity Management

* Integrated the `GlobalRequestPool` service in the `Tier1App` to manage global requests pooling.
Expand Down
2 changes: 2 additions & 0 deletions cmd/apps/substreams_tier1.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ func RegisterSubstreamsTier1App[B firecore.Block](chain *firecore.Chain[B], root
This is useful to prevent the tier1 from being overwhelmed by too many requests, most client auto-reconnects on 'Unavailable' code
so they should end up on another tier1 instance, assuming you have proper auto-scaling of the number of instances available.
`))
cmd.Flags().String("substreams-tier1-quicksave-store", "", "If enabled, substreams will use this store to put 'quicksave' data when shutting down while running requests with 'stores'. Use this flag with a non-zero --common-system-shutdown-signal-delay")
cmd.Flags().String("substreams-tier1-global-worker-pool-address", "", "Address of the global worker pool to use for the substreams tier1. (disabled if empty)")
cmd.Flags().String("substreams-tier1-global-request-pool-address", "", "Address of the global worker pool to use for the substreams tier1. (disabled if empty)")
cmd.Flags().Duration("substreams-tier1-global-worker-pool-keep-alive-delay", 25*time.Second, "Delay between two keep alive call to the global worker pool. Default is 25s")
Expand Down Expand Up @@ -163,6 +164,7 @@ func RegisterSubstreamsTier1App[B firecore.Block](chain *firecore.Chain[B], root
config.GRPCListenAddr = viper.GetString("substreams-tier1-grpc-listen-addr")
config.GRPCShutdownGracePeriod = time.Second
config.ServiceDiscoveryURL = serviceDiscoveryURL
config.QuickSaveStoreURL = viper.GetString("substreams-tier1-quicksave-store")

subRequestsClientConfig := client.NewSubstreamsClientConfig(
config.SubrequestsEndpoint,
Expand Down
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ require (
github.com/spf13/cobra v1.7.0
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.15.0
github.com/streamingfast/bstream v0.0.2-0.20250129191551-5539724f4b12
github.com/streamingfast/bstream v0.0.2-0.20250221181559-fb0809660f91
github.com/streamingfast/cli v0.0.4-0.20250116003948-fbf66c930cce
github.com/streamingfast/dauth v0.0.0-20250130223258-c615a033a660
github.com/streamingfast/dbin v0.9.1-0.20231117225723-59790c798e2c
Expand All @@ -31,7 +31,7 @@ require (
github.com/streamingfast/payment-gateway v0.0.0-20240426151444-581e930c76e2
github.com/streamingfast/pbgo v0.0.6-0.20250114182320-0b43084f4000
github.com/streamingfast/snapshotter v0.0.0-20230316190750-5bcadfde44d0
github.com/streamingfast/substreams v1.12.5-0.20250220140015-91e8dc3e5576
github.com/streamingfast/substreams v1.13.1-0.20250224235925-57717ad26c2a
github.com/streamingfast/worker-pool-protocol v0.0.0-20250218145136-4ad271e36e39
github.com/stretchr/testify v1.10.0
github.com/test-go/testify v1.1.4
Expand Down
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -2152,8 +2152,8 @@ github.com/spf13/viper v1.15.0 h1:js3yy885G8xwJa6iOISGFwd+qlUo5AvyXb7CiihdtiU=
github.com/spf13/viper v1.15.0/go.mod h1:fFcTBJxvhhzSJiZy8n+PeW6t8l+KeT/uTARa0jHOQLA=
github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8=
github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo=
github.com/streamingfast/bstream v0.0.2-0.20250129191551-5539724f4b12 h1:Kh4ZxZwv/bccTLFNugEbj9wfxuukbLkE6Mmkqc3OACw=
github.com/streamingfast/bstream v0.0.2-0.20250129191551-5539724f4b12/go.mod h1:n5wy+Vmwp4xbjXO7B81MAkAgjnf1vJ/lI2y6hWWyFbg=
github.com/streamingfast/bstream v0.0.2-0.20250221181559-fb0809660f91 h1:n4Ws5vS8dF4J6WYP7mE0C9DsE1Inju7BKd6XWI4T18c=
github.com/streamingfast/bstream v0.0.2-0.20250221181559-fb0809660f91/go.mod h1:n5wy+Vmwp4xbjXO7B81MAkAgjnf1vJ/lI2y6hWWyFbg=
github.com/streamingfast/cli v0.0.4-0.20250116003948-fbf66c930cce h1:RWla1PaRrlDf/MOwVoN/dJhIM/dXa9O4rmKZkv9T5bg=
github.com/streamingfast/cli v0.0.4-0.20250116003948-fbf66c930cce/go.mod h1:qOksW3DPhHVYBo8dcYxS7K3Q09wlcOChSdopeOjLWng=
github.com/streamingfast/dauth v0.0.0-20250130223258-c615a033a660 h1:QwuploKvaXyvwpxhFWKmPFrvesJRJb7gK+gsgEbAY5c=
Expand Down Expand Up @@ -2197,8 +2197,8 @@ github.com/streamingfast/shutter v1.5.0 h1:NpzDYzj0HVpSiDJVO/FFSL6QIK/YKOxY0gJAt
github.com/streamingfast/shutter v1.5.0/go.mod h1:B/T6efqdeMGbGwjzPS1ToXzYZI4kDzI5/u4I+7qbjY8=
github.com/streamingfast/snapshotter v0.0.0-20230316190750-5bcadfde44d0 h1:Y15G1Z4fpEdm2b+/70owI7TLuXadlqBtGM7rk4Hxrzk=
github.com/streamingfast/snapshotter v0.0.0-20230316190750-5bcadfde44d0/go.mod h1:/Rnz2TJvaShjUct0scZ9kKV2Jr9/+KBAoWy4UMYxgv4=
github.com/streamingfast/substreams v1.12.5-0.20250220140015-91e8dc3e5576 h1:7g3xTNv5C3w5+3l8/IvM5RbF+fwZRsaIE66f7+uZm1s=
github.com/streamingfast/substreams v1.12.5-0.20250220140015-91e8dc3e5576/go.mod h1:2/yOQAnNP4EQ66buwMF4eOVUgPN6RWrh8VxHT5LL5vc=
github.com/streamingfast/substreams v1.13.1-0.20250224235925-57717ad26c2a h1:DgOAtqrmy+3kMH7XrSYjd7HepwdNq92C5LvdE4Qwork=
github.com/streamingfast/substreams v1.13.1-0.20250224235925-57717ad26c2a/go.mod h1:BSlwo+Ihg3ygSNsbPoy9DXagM3eKoXSo2zQBjMW5plg=
github.com/streamingfast/wazero v0.0.0-20241202185309-91287c3640ed h1:LU6/c376zP1cMAo9L6rFLyjo0W7RU+hIh7BegH8Zo5M=
github.com/streamingfast/wazero v0.0.0-20241202185309-91287c3640ed/go.mod h1:yAI0XTsMBhREkM/YDAK/zNou3GoiAce1P6+rp/wQhjs=
github.com/streamingfast/worker-pool-protocol v0.0.0-20250218145136-4ad271e36e39 h1:NBBLx99rrGz/hxwHjHi+QyN07DfqcDC4zzuBEsH0/vE=
Expand Down

0 comments on commit bad7e2c

Please sign in to comment.