Feat/enable backfill

README.md

@@ -32,6 +32,7 @@ For developers, detailed documentation for `libfloresta` is available [here](htt
   - [Building with Nix](#building-with-nix)
 - [Running](#running)
   - [Assume Utreexo](#assume-utreexo)
+  - [Backfill](#backfill)
   - [Compact Filters](#compact-filters)
   - [Getting Help](#getting-help)
   - [Wallet](#wallet)
@@ -70,7 +71,7 @@ git clone https://github.com/vinteumorg/Floresta.git
 go to the Floresta directory
 
 ```bash
-cd Floresta/
+$ cd Floresta/
 ```
 
 and build with cargo build
@@ -151,9 +152,9 @@ pkgs.floresta-node
 
 After building, florestad and floresta-cli will be available in the target directory. You can run the full node with
 ```bash
-./target/release/florestad
+$ ./target/release/florestad
 # or, if you installed it with cargo install
-florestad
+$ florestad
 ```
 
 You may run it as a background process with the `--daemon` flag.
@@ -165,41 +166,52 @@ florestad --daemon
 This will start the full node, and you can connect to it with an Electrum wallet or with the `floresta-cli` tool.
 
 ```bash
-floresta-cli getblockchaininfo
+$ floresta-cli getblockchaininfo
 ```
 
 For more information on how to use the `floresta-cli` tool, you can check the [api documentation](https://github.com/vinteumorg/Floresta/blob/master/crates/floresta-cli/README.md).
 
 Before running you can create the SSL certificates. If you don't do it, it will display a logging `Failed to load SSL certificates, ignoring SSL`. However, it is not mandatory to have the certificates to run the full node.
 
-### Assume Utreexo
+#### Assume Utreexo
+
 If you want to skip the IBD process, you can use the `--assume-utreexo` flag. This flag will start the node at a given height, with the state
 provided by this implementation. Therefore, you're trusting that we are giving you the correct state. Everything after that height will be
 verified by the node just like any other node.
 
 ```bash
-florestad --assume-utreexo
+$ florestad --assume-utreexo
+```
+
+#### Backfill
+
+If you want to get a node up and running in a few minutes, but still want to validate everything, you can start `florestad` with `assumeutreexo` and pass the `--backfill` flag. This will download the blocks from genesis to the assumed height, validate them and compare with the provided value, all in the background. This way, you can start using the node right away, and it will be fully validated in a few hours/days depending on your hardware.
+
+This is the default behavior of `florestad` if no flags are provided.
+
+```bash
+$ florestad --assume-utreexo --backfill
 ```
 
 ### Compact Filters
 
 Floresta supports compact block filters, which can be used to scan for transactions in a block without downloading the entire block. You can start the node with the `--cfilters` flag to download the filters for the blocks that you're interested in. You can also use the `--filters-start-height` flag to specify the block height that you want to start downloading the filters from. This is useful if you want to download only the filters for a specific range of blocks.
 
 ```bash
-florestad --cfilters --filters-start-height 800000
+$ florestad --cfilters --filters-start-height 800000
 ```
 
 ### Getting Help
 
 You can get a list of all the available commands by running
 
 ```bash
-floresta-cli help
+$ floresta-cli help
 ```
 
 and you can get the cli parameters by running
 ```bash
-floresta-cli help <command>
+$ floresta-cli help <command>
 ```
 
 ### Wallet
@@ -211,21 +223,21 @@ call the `rescan` rpc after adding the wallet.
 You can add new descriptors to the wallet with the `importdescriptor` rpc.
 
 ```bash
-floresta-cli importdescriptor "wpkh(xpub6CFy3kRXorC3NMTt8qrsY9ucUfxVLXyFQ49JSLm3iEG5gfAmWewYFzjNYFgRiCjoB9WWEuJQiyYGCdZvUTwPEUPL9pPabT8bkbiD9Po47XG/<0;1>/*)"
+$ floresta-cli importdescriptor "wpkh(xpub6CFy3kRXorC3NMTt8qrsY9ucUfxVLXyFQ49JSLm3iEG5gfAmWewYFzjNYFgRiCjoB9WWEuJQiyYGCdZvUTwPEUPL9pPabT8bkbiD9Po47XG/<0;1>/*)"
 ```
 
 The rescan assumes that you have compact block filters for the blocks that you're scanning. You can either download all the filters
 (about 11GB on mainnet) or, if you know the block range that you're interested in, you can download only the filters for that range
 using the `--filters-start-height` option. Let's you know that none of your wallets are older than block 800,000. Just start the node with.
 
 ```bash
-./target/release/florestad --cfilters --filters-start-height 800000
+$ ./target/release/florestad --cfilters --filters-start-height 800000
 ```
 
 if you add a wallet and want to rescan the blocks from 800,000 to the current height, you can use the `rescan` rpc.
 
 ```bash
-floresta-cli rescan 800000
+$ floresta-cli rescan 800000
 ```
 
 Once you have a transaction cached in your watch-only, you can use either the rpc or integrated electrum server to retrieve information about your wallet. You can use wallets like Electrum or Sparrow to connect to your node and retrieve information about your wallet. Just connect with the server running at `127.0.0.1:50001:t`. On electrum you may want to use the `--oneserver` flag to connect to a single server, for better privacy.
@@ -237,15 +249,15 @@ Once you have a transaction cached in your watch-only, you can use either the rp
 The tests in `floresta-cli` depend on the compiled `florestad` binary. Make sure to build the entire project first by running:
 
 ```bash
-cargo build
+$ cargo build
 ```
 
 ### Testing Options
 
 There's a set of tests that you can run with:
 
 ```bash
-cargo test
+$ cargo test
 ```
 
 For the full test suite, including long-running tests, use:
@@ -303,8 +315,8 @@ poetry run poe pre-commit
 * Manual way without poetry: install dependencies and run the test script. This is discouraged since that can lead to inconsistences between different python versions:
 
 ```bash
-pip3 install -r tests/requirements.txt
-python tests/run_tests.py
+$ pip3 install -r tests/requirements.txt
+$ python tests/run_tests.py
 ```
 
 ## Running Benchmarks

crates/floresta-chain/src/pruned_utreexo/chain_state.rs

@@ -1004,6 +1004,10 @@ impl<PersistedState: ChainStore> UpdatableChainstate for ChainState<PersistedSta
         self.reorg(new_tip)
     }
 
+    fn get_acc(&self) -> Stump {
+        self.acc()
+    }
+
     fn mark_block_as_valid(&self, block: BlockHash) -> Result<(), BlockchainError> {
         let header = self.get_disk_block_header(&block)?;
         let height = header.height().unwrap();
@@ -1183,16 +1187,14 @@ impl<PersistedState: ChainStore> UpdatableChainstate for ChainState<PersistedSta
         final_height: u32,
         acc: Stump,
     ) -> Result<super::partial_chain::PartialChainState, BlockchainError> {
-        let blocks = (initial_height..=final_height)
-            .flat_map(|height| {
+        let blocks = (0..=final_height)
+            .map(|height| {
                 let hash = self
                     .get_block_hash(height)
                     .expect("Block should be present");
-                self.get_disk_block_header(&hash)
-            })
-            .filter_map(|header| match header {
-                DiskBlockHeader::FullyValid(header, _) => Some(header),
-                _ => None,
+                *self
+                    .get_disk_block_header(&hash)
+                    .expect("Block should be present")
             })
             .collect();
 
@@ -1205,7 +1207,6 @@ impl<PersistedState: ChainStore> UpdatableChainstate for ChainState<PersistedSta
             current_acc: acc,
             final_height,
             assume_valid: false,
-            initial_height,
             current_height: initial_height,
         };
 

crates/floresta-chain/src/pruned_utreexo/mod.rs

@@ -142,7 +142,6 @@ pub trait UpdatableChainstate {
     /// [PartialChainState] to completion by downloading blocks inside that chainstate's range.
     /// If all goes right, it'll end without error, and you should mark blocks in this range as
     /// valid.
-    ///
     /// Since this chainstate may start from a height with an existing UTXO set, you need to
     /// provide a [Stump] for that block.
     fn get_partial_chain(
@@ -156,6 +155,8 @@ pub trait UpdatableChainstate {
     /// This mimics the behaviour of checking every block before this block, and continues
     /// from this point
     fn mark_chain_as_assumed(&self, acc: Stump, tip: BlockHash) -> Result<bool, BlockchainError>;
+    /// Returns the current accumulator
+    fn get_acc(&self) -> Stump;
 }
 
 /// [ChainStore] is a trait defining how we interact with our chain database. This definitions
@@ -206,6 +207,10 @@ impl<T: UpdatableChainstate> UpdatableChainstate for Arc<T> {
         T::flush(self)
     }
 
+    fn get_acc(&self) -> Stump {
+        T::get_acc(self)
+    }
+
     fn toggle_ibd(&self, is_ibd: bool) {
         T::toggle_ibd(self, is_ibd)
     }

crates/floresta-chain/src/pruned_utreexo/partial_chain.rs

@@ -51,11 +51,6 @@ pub(crate) struct PartialChainStateInner {
     /// and to build the accumulator. We assume this is sorted by height, and
     /// should contains all blocks in this interval.
     pub(crate) blocks: Vec<BlockHeader>,
-    /// The height this interval starts at. This [initial_height, final_height), so
-    /// if we break the interval at height 100, the first interval will be [0, 100)
-    /// and the second interval will be [100, 200). And the initial height of the
-    /// second interval will be 99.
-    pub(crate) initial_height: u32,
     /// The height we are on right now, this is used to keep track of the progress
     /// of the sync.
     pub(crate) current_height: u32,
@@ -97,19 +92,17 @@ unsafe impl Send for PartialChainState {}
 unsafe impl Sync for PartialChainState {}
 
 impl PartialChainStateInner {
-    /// Returns the height we have synced up to so far
-    pub fn current_height(&self) -> u32 {
-        self.current_height
-    }
-
     /// Whether or not we have synced up to the final height
     pub fn is_sync(&self) -> bool {
         self.current_height == self.final_height
     }
 
     pub fn get_block(&self, height: u32) -> Option<&BlockHeader> {
-        let index = height - self.initial_height;
-        self.blocks.get(index as usize)
+        if height >= self.blocks.len() as u32 {
+            return None;
+        }
+
+        self.blocks.get(height as usize)
     }
 
     #[cfg(feature = "bitcoinconsensus")]
@@ -199,6 +192,7 @@ impl PartialChainStateInner {
                 block.block_hash()
             );
         }
+
         self.update_state(height, acc);
 
         Ok(height)
@@ -216,6 +210,7 @@ impl PartialChainStateInner {
                 BlockValidationErrors::BadMerkleRoot,
             ));
         }
+
         if height >= self.chain_params().params.bip34_height
             && block.bip34_block_height() != Ok(height as u64)
         {
@@ -323,6 +318,10 @@ impl UpdatableChainstate for PartialChainState {
         self.inner().current_acc.roots.clone()
     }
 
+    fn get_acc(&self) -> Stump {
+        self.inner().current_acc.clone()
+    }
+
     //these are no-ops, you can call them, but they won't do anything
 
     fn flush(&self) -> Result<(), BlockchainError> {
@@ -382,7 +381,6 @@ impl BlockchainInterface for PartialChainState {
     }
 
     fn get_block_hash(&self, height: u32) -> Result<bitcoin::BlockHash, BlockchainError> {
-        let height = height - self.inner().initial_height;
         self.inner()
             .blocks
             .get(height as usize)
@@ -392,8 +390,8 @@ impl BlockchainInterface for PartialChainState {
 
     fn get_best_block(&self) -> Result<(u32, bitcoin::BlockHash), Self::Error> {
         Ok((
-            self.inner().current_height(),
-            self.get_block_hash(self.inner().current_height())?,
+            self.inner().final_height,
+            self.get_block_hash(self.inner().final_height)?,
         ))
     }
 
@@ -552,7 +550,6 @@ mod tests {
             final_height: 1,
             blocks,
             error: None,
-            initial_height: 0,
         }
         .into()
     }
@@ -578,7 +575,6 @@ mod tests {
             final_height: 100,
             blocks: parsed_blocks.iter().map(|block| block.header).collect(),
             error: None,
-            initial_height: 0,
         }
         .into();
         parsed_blocks.remove(0);
@@ -604,7 +600,8 @@ mod tests {
             parsed_blocks.push(block);
         }
         // The file contains 150 blocks, we split them into two chains.
-        let (blocks1, blocks2) = parsed_blocks.split_at(101);
+        let split = parsed_blocks.clone();
+        let (blocks1, blocks2) = split.split_at(101);
         let mut chainstate1 = PartialChainStateInner {
             assume_valid: true,
             consensus: Consensus {
@@ -613,28 +610,26 @@ mod tests {
             current_height: 0,
             current_acc: Stump::default(),
             final_height: 100,
-            blocks: blocks1.iter().map(|block| block.header).collect(),
+            blocks: parsed_blocks.iter().map(|block| block.header).collect(),
             error: None,
-            initial_height: 0,
         };
+
         // We need to add the last block of the first chain to the second chain, so that
         // the second chain can validate all its blocks.
-        let mut blocks2_headers = vec![blocks1.last().unwrap()];
-        blocks2_headers.extend(blocks2);
-
-        let blocks2_headers = blocks2_headers.iter().map(|block| block.header).collect();
-
-        let mut blocks1 = blocks1.iter();
-        blocks1.next();
+        for (height, block) in blocks1.iter().enumerate() {
+            // skip the genesis block
+            if height == 0 {
+                continue;
+            }
 
-        for block in blocks1 {
             let proof = Proof::default();
             let inputs = HashMap::new();
             let del_hashes = vec![];
             chainstate1
                 .process_block(block, proof, inputs, del_hashes)
                 .unwrap();
         }
+
         // The state after 100 blocks, computed ahead of time.
         let roots = [
             "a2f1e6db842e13c7480c8d80f29ca2db5f9b96e1b428ebfdbd389676d7619081",
@@ -661,9 +656,8 @@ mod tests {
             current_height: 100,
             current_acc: acc2,
             final_height: 150,
-            blocks: blocks2_headers,
+            blocks: parsed_blocks.iter().map(|block| block.header).collect(),
             error: None,
-            initial_height: 100,
         }
         .into();
 

crates/floresta-electrum/src/electrum_protocol.rs

@@ -928,6 +928,7 @@ mod test {
     use tokio::net::TcpListener;
     use tokio::net::TcpStream;
     use tokio::sync::Mutex;
+    use tokio::sync::RwLock;
     use tokio::task;
     use tokio::time::timeout;
     use tokio_rustls::rustls::Certificate;
@@ -1065,6 +1066,7 @@ mod test {
                 chain.clone(),
                 Arc::new(Mutex::new(Mempool::new(Pollard::default(), 0))),
                 None,
+                Arc::new(RwLock::new(false)),
             )
             .unwrap();