Feature/web3 holder provider

.github/workflows/ai-pr-reviewer.yml

@@ -0,0 +1,32 @@
+name: Code Review
+
+permissions:
+  contents: read
+  pull-requests: write
+
+on:
+  pull_request:
+  pull_request_review_comment:
+    types: [created]
+
+concurrency:
+  group:
+    ${{ github.repository }}-${{ github.event.number || github.head_ref ||
+    github.sha }}-${{ github.workflow }}-${{ github.event_name ==
+    'pull_request_review_comment' && 'pr_comment' || 'pr' }}
+  cancel-in-progress: ${{ github.event_name != 'pull_request_review_comment' }}
+
+jobs:
+  review:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: coderabbitai/ai-pr-reviewer@latest
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        with:
+          debug: false
+          review_simple_changes: false
+          review_comment_lgtm: false
+          openai_heavy_model: gpt-4
+

COMPONENTS.md

@@ -0,0 +1,57 @@
+# Census3 Service documentation
+
+## Index
+
+1. [SQLite Database](#sqlite-database)
+2. [Scanner](#scanner)
+    1. [Holders providers](#holders-providers)
+3. [API](#api)
+    1. [Toolbox](#toolbox)
+
+---
+
+### SQLite Database
+Contains all the service data. It contains six tables:
+* *Tokens*: Contains the addresses, chainIDs and other attributes of the registered tokens in the service.
+* *TokenTypes*: The relationship between the token type ID's and their names. It does not define the supported types, this is done by the scanner.
+* *Strategies*: The list of strategies to create counts using the registered tokens.
+* *Strategy Tokens*: The details of the tokens used in each strategy (such as the minimum balance required).
+* *Censuses*: The list of censuses created.
+* *Token Holders*: The list of addresses involved in the transfer of registered tokens.
+
+Check out [`./db`](./db/) folder.
+
+### Scanner
+This component iterates until the service stops, keeping the holders and balances of each registered token updated. To do this, it performs the following steps:
+1. Retrieves the token information from the database, trying to scan the new and smaller tokens first.
+2. Depending on the type of token, it selects its owner provider and obtains the latest updates.
+1. If the creation block of the token contract has not yet been calculated, the scanner will calculate it before updating its token holders.
+3. Updates the token and its holder information in the database.
+4. Loop forever.
+
+Check out [`./scanner`](./scanner) folder.
+
+#### Holders Providers
+These components are defined by an interface that supports the creation of different types of holder providers, such as Web3 based (ERC20, ERC721, ERC777) or external service based (POAP or Gitcoin Passport).
+
+This interface defines all the methods the scanner needs to work, and retains the logic of how the holder and balance updates are calculated.
+
+See some examples in the [`./scanner/providers`](./scanner/providers) folder.
+
+### API
+The API has two goals:
+* Provide information: It exposes the token information, the results of the scanner and the built strategies and censuses.
+* Create resources: It allows to the user to register new tokens, create new strategies or build new censuses.
+
+Check out [`./api`](./api) folder and the [API specification](./api/README.md).
+
+#### Toolbox
+It includes the following packages:
+* **[`internal/lexer`](./internal/lexer/)**: The lexer package helps to analyse the predicate of strategies, allowing to define a syntax to combine the token holders of a group of tokens.
+* **[`internal/queue`](./internal/queue/)**: The queue package allows to the API and other components of the Census3 service to run processes in the background, providing a way to know the status of a job or retrieve its results.
+* **[`internal/roundedcensus`](./internal/roundedcensus/)**: The roundedcensus package provides an algorithm to anonymize participant balances in a token based census while maintaining a certain level of accuracy.
+* **[`internal/strategyoperators`](./internal/strategyoperators/)**: This package defines the differents operators to be used in the strategies, and how they combine the holders of different tokens.
+
+
+## Schema
+![Components](https://i.postimg.cc/7LYXtDcF/census3-schema.png)

README.md

@@ -27,17 +27,17 @@ Then, it allows creating a merkle tree census (compatible with [Vocdoni](https:/
 The service suports the following token types:
 * ERC20
 * ERC721
-* ERC1155 (*coming soon*)
 * ERC777
-* Nation3 (veNation)
-* wANT
 * POAP
+* Gitcoin Passport Score
+* Gitcoin Passport Shields (*coming soon*)
+* ERC1155 (*coming soon*)
 
 
 #### About censuses
  * The censuses are published on [IPFS](https://ipfs.tech/) after their creation. 
  * Census3 uses [go.vocdoni.io/dvote/tree/arbo](go.vocdoni.io/dvote/tree/arbo) to build the census merkle trees.
- * The censuses can be created with the holders of just one token or a combination of tokens, using **complex strategies**  (*coming soon*).
+ * The censuses can be created with the holders of just one token or a combination of tokens, using **complex strategies**.
  * The censuses are *zk-friendly* and can also be used for anonymous voting.
 
 #### About complex strategies
@@ -71,11 +71,18 @@ go build -o census3 ./cmd/census3
 ```sh
 ./census3 --help
 Usage of ./census3:
-      --connectKey string      connect group key for IPFS connect
-      --dataDir string         data directory for persistent storage (default "<$HOME>/.census3")
-      --logLevel string        log level (debug, info, warn, error) (default "info")
-      --port int               HTTP port for the API (default 7788)
-      --web3Providers string   the list of URL's of available web3 providers (separated with commas)
+      --adminToken string          the admin UUID token for the API
+      --connectKey string          connect group key for IPFS connect
+      --dataDir string             data directory for persistent storage (default "~/.census3")
+      --logLevel string            log level (debug, info, warn, error) (default "info")
+      --port int                   HTTP port for the API (default 7788)
+      --scannerCoolDown duration   the time to wait before next scanner iteration (default 2m0s)
+      --initialTokens string       path of the initial tokens json file
+      --web3Providers string       the list of URLs of available web3 providers
+      --gitcoinEndpoint string     Gitcoin Passport API access token
+      --gitcoinCooldown duration   Gitcoin Passport API cooldown (default 6h0m0s)
+      --poapAPIEndpoint string     POAP API endpoint
+      --poapAuthToken string       POAP API access token
 ```
 
 Example:
@@ -91,21 +98,29 @@ Example:
 
 1. Create your config file using the [`.env` file](example.env) as a template and save it the root.
 ```sh
+# Domain name for TLS
+DOMAIN=your.own.domain.xyz
 # A web3 endpoint provider
-WEB3_PROVIDERS=https://rpc-endoint.example1.com,https://rpc-endoint.example2.com
-
+CENSUS3_WEB3PROVIDERS="https://rpc-endoint.example1.com,https://rpc-endoint.example2.com"
 # Internal port for the service (80 and 443 are used by traefik)
-PORT=7788
-
-# Domain name for TLS
-# DOMAIN=your.own.domain.xyz
-DOMAIN=localhost
-
+CENSUS3_PORT=7788
 # Log level (info, debug, warning, error)
-LOGLEVEL=debug
-
+CENSUS3_LOGLEVEL="debug"
 # IPFS connect key for discovering nodes
-CONNECT_KEY=yourIPFSConnectKey
+# CONNECT_KEY=yourIPFSConnectKey
+CENSUS3_CONNECTKEY="census3key"
+# Internal data folder
+CENSUS3_DATADIR="./.census3"
+# Scanner cooldown duration
+CENSUS3_SCANNERCOOLDOWN="20s"
+# UUID admin token for protected endpoints
+CENSUS3_ADMINTOKENS="UUID"
+# POAP API configuration
+CENSUS3_POAPAPIENDPOINT="poapAPIEndpoint"
+CENSUS3_POAPAUTHTOKEN="yourPOAPAuthToken"
+# Gitcoin API configuration
+CENSUS3_GITCOINAPIENDPOINT="gitcoinAPIEndpoint"
+CENSUS3_INITIALTOKENS="/app/initial_tokens.json"
 ```
 
 2. Run the services with `docker compose`:

api/api.go

@@ -15,10 +15,9 @@ import (
 	"github.com/vocdoni/census3/db"
 	"github.com/vocdoni/census3/db/annotations"
 	queries "github.com/vocdoni/census3/db/sqlc"
-	"github.com/vocdoni/census3/queue"
-	"github.com/vocdoni/census3/service"
-	"github.com/vocdoni/census3/service/web3"
-	"github.com/vocdoni/census3/state"
+	"github.com/vocdoni/census3/internal/queue"
+	"github.com/vocdoni/census3/scanner/providers"
+	"github.com/vocdoni/census3/scanner/providers/web3"
 	"go.vocdoni.io/dvote/api/censusdb"
 	storagelayer "go.vocdoni.io/dvote/data"
 	"go.vocdoni.io/dvote/data/downloader"
@@ -33,26 +32,26 @@ import (
 )
 
 type Census3APIConf struct {
-	Hostname      string
-	Port          int
-	DataDir       string
-	GroupKey      string
-	Web3Providers web3.NetworkEndpoints
-	ExtProviders  map[state.TokenType]service.HolderProvider
-	AdminToken    string
+	Hostname        string
+	Port            int
+	DataDir         string
+	GroupKey        string
+	Web3Providers   web3.NetworkEndpoints
+	HolderProviders map[uint64]providers.HolderProvider
+	AdminToken      string
 }
 
 type census3API struct {
-	conf         Census3APIConf
-	db           *db.DB
-	endpoint     *api.API
-	censusDB     *censusdb.CensusDB
-	queue        *queue.BackgroundQueue
-	w3p          web3.NetworkEndpoints
-	storage      storagelayer.Storage
-	downloader   *downloader.Downloader
-	extProviders map[state.TokenType]service.HolderProvider
-	cache        *lru.Cache[CacheKey, any]
+	conf            Census3APIConf
+	db              *db.DB
+	endpoint        *api.API
+	censusDB        *censusdb.CensusDB
+	queue           *queue.BackgroundQueue
+	w3p             web3.NetworkEndpoints
+	storage         storagelayer.Storage
+	downloader      *downloader.Downloader
+	holderProviders map[uint64]providers.HolderProvider
+	cache           *lru.Cache[CacheKey, any]
 }
 
 func Init(db *db.DB, conf Census3APIConf) (*census3API, error) {
@@ -61,12 +60,12 @@ func Init(db *db.DB, conf Census3APIConf) (*census3API, error) {
 		return nil, err
 	}
 	newAPI := &census3API{
-		conf:         conf,
-		db:           db,
-		w3p:          conf.Web3Providers,
-		queue:        queue.NewBackgroundQueue(),
-		extProviders: conf.ExtProviders,
-		cache:        cache,
+		conf:            conf,
+		db:              db,
+		w3p:             conf.Web3Providers,
+		queue:           queue.NewBackgroundQueue(),
+		holderProviders: conf.HolderProviders,
+		cache:           cache,
 	}
 	// get the current chainID
 	log.Infow("starting API", "web3Providers", conf.Web3Providers.String())
@@ -212,7 +211,7 @@ func (capi *census3API) CreateInitialTokens(tokensPath string) error {
 			Decimals:      token.Decimals,
 			TotalSupply:   annotations.BigInt(token.TotalSupply),
 			CreationBlock: 0,
-			TypeID:        uint64(state.TokenTypeFromString(token.Type)),
+			TypeID:        providers.TokenTypeID(token.Type),
 			Synced:        false,
 			Tags:          token.Tags,
 			ChainID:       token.ChainID,

api/censuses.go

@@ -12,7 +12,7 @@ import (
 	"github.com/ethereum/go-ethereum/common"
 	queries "github.com/vocdoni/census3/db/sqlc"
 	"github.com/vocdoni/census3/internal"
-	"github.com/vocdoni/census3/roundedcensus"
+	"github.com/vocdoni/census3/internal/roundedcensus"
 	"go.vocdoni.io/dvote/httprouter"
 	api "go.vocdoni.io/dvote/httprouter/apirest"
 	"go.vocdoni.io/dvote/log"
@@ -140,7 +140,7 @@ func (capi *census3API) createAndPublishCensus(req *CreateCensusRequest, qID str
 	}
 	// init some variables to get computed in the following steps
 	strategyHolders, censusWeight, totalTokensBlockNumber, err := CalculateStrategyHolders(
-		internalCtx, capi.db.QueriesRO, capi.w3p, req.StrategyID, strategy.Predicate)
+		internalCtx, capi.db.QueriesRO, capi.holderProviders, req.StrategyID, strategy.Predicate)
 	if err != nil {
 		return 0, ErrEvalStrategyPredicate.WithErr(err)
 	}

api/helpers.go

@@ -13,10 +13,10 @@ import (
 
 	"github.com/ethereum/go-ethereum/common"
 	queries "github.com/vocdoni/census3/db/sqlc"
-	"github.com/vocdoni/census3/lexer"
-	"github.com/vocdoni/census3/service/web3"
-	"github.com/vocdoni/census3/state"
-	"github.com/vocdoni/census3/strategyoperators"
+	"github.com/vocdoni/census3/internal/lexer"
+	"github.com/vocdoni/census3/internal/strategyoperators"
+	"github.com/vocdoni/census3/scanner/providers"
+	"github.com/vocdoni/census3/scanner/providers/web3"
 	"go.vocdoni.io/dvote/api/censusdb"
 	"go.vocdoni.io/dvote/censustree"
 	storagelayer "go.vocdoni.io/dvote/data"
@@ -208,8 +208,8 @@ func InnerCensusID(blockNumber, strategyID uint64, anonymous bool) uint64 {
 // The evaluator uses the strategy operators to evaluate the predicate which
 // uses the database queries to get the token holders and their balances, and
 // combines them.
-func CalculateStrategyHolders(ctx context.Context, qdb *queries.Queries, w3p web3.NetworkEndpoints,
-	id uint64, predicate string,
+func CalculateStrategyHolders(ctx context.Context, qdb *queries.Queries,
+	providers map[uint64]providers.HolderProvider, id uint64, predicate string,
 ) (map[common.Address]*big.Int, *big.Int, uint64, error) {
 	// TODO: write a benchmark and try to optimize this function
 
@@ -237,15 +237,20 @@ func CalculateStrategyHolders(ctx context.Context, qdb *queries.Queries, w3p web
 	// number, used to create the census id.
 	totalTokensBlockNumber := uint64(0)
 	for _, token := range strategyTokens {
-		w3endpoint, exists := w3p.EndpointByChainID(token.ChainID)
+		provider, exists := providers[token.TypeID]
 		if !exists {
-			return nil, nil, 0, fmt.Errorf("web3 endpoint not found for chain id %d", token.ChainID)
+			return nil, nil, 0, fmt.Errorf("provider not found for token type id %d", token.TypeID)
 		}
-		w3 := state.Web3{}
-		if err := w3.Init(ctx, w3endpoint, common.BytesToAddress(token.ID), state.TokenType(token.TypeID)); err != nil {
-			return nil, nil, 0, err
+		if !provider.IsExternal() {
+			if err := provider.SetRef(web3.Web3ProviderRef{
+				HexAddress: common.Bytes2Hex(token.ID),
+				ChainID:    token.ChainID,
+			}); err != nil {
+				return nil, nil, 0, err
+			}
 		}
-		currentBlockNumber, err := w3.LatestBlockNumber(ctx)
+
+		currentBlockNumber, err := provider.LatestBlockNumber(ctx, []byte(token.ExternalID))
 		if err != nil {
 			return nil, nil, 0, err
 		}