From 4b1617de8a2d8d6a6765a1dd51055d3debfc3c9d Mon Sep 17 00:00:00 2001 From: Hubert Bugaj Date: Fri, 21 Feb 2025 11:59:52 +0100 Subject: [PATCH 1/2] chore: migrate snapshot and backup docs, remove unused --- docs/dictionary.txt | 1 + .../users/guides/advanced/_category_.json | 2 +- docs/docs/users/guides/advanced/backups.md | 62 ++++++++ .../guides/advanced/generating_snapshots.md | 37 +++++ docs/docs/users/guides/methods_filtering.md | 1 + .../users/guides/monitoring/_category_.json | 2 +- documentation/mindmaps/2023-07-14.md | 44 ------ documentation/src/SUMMARY.md | 8 - documentation/src/backups.md | 95 ----------- .../mainnet_compatibility.md | 84 ---------- .../notes_and_sketches.md | 3 - .../state_migration.md | 117 -------------- .../state_migration_spike.md | 149 ------------------ documentation/src/glossary.md | 122 -------------- documentation/src/snapshot_export.md | 50 ------ 15 files changed, 103 insertions(+), 674 deletions(-) create mode 100644 docs/docs/users/guides/advanced/backups.md delete mode 100644 documentation/mindmaps/2023-07-14.md delete mode 100644 documentation/src/backups.md delete mode 100644 documentation/src/developer_documentation/mainnet_compatibility.md delete mode 100644 documentation/src/developer_documentation/notes_and_sketches.md delete mode 100644 documentation/src/developer_documentation/state_migration.md delete mode 100644 documentation/src/developer_documentation/state_migration_spike.md delete mode 100644 documentation/src/glossary.md delete mode 100644 documentation/src/snapshot_export.md diff --git a/docs/dictionary.txt b/docs/dictionary.txt index cb075d46a84e..392128bf8de3 100644 --- a/docs/dictionary.txt +++ b/docs/dictionary.txt @@ -86,6 +86,7 @@ rustup S3-compatible semver serverless +stateroots subcommands swappiness TabItem diff --git a/docs/docs/users/guides/advanced/_category_.json b/docs/docs/users/guides/advanced/_category_.json index fa338fd55290..f5a6aec83d91 100644 --- a/docs/docs/users/guides/advanced/_category_.json +++ b/docs/docs/users/guides/advanced/_category_.json @@ -1,4 +1,4 @@ { "label": "Advanced", - "position": 2 + "position": 7 } diff --git a/docs/docs/users/guides/advanced/backups.md b/docs/docs/users/guides/advanced/backups.md new file mode 100644 index 000000000000..b373dbf3a932 --- /dev/null +++ b/docs/docs/users/guides/advanced/backups.md @@ -0,0 +1,62 @@ +# Forest Backups + +> "_The condition of any backup is unknown until a restore is attempted._" +> Everyone who deals with backups. + +## Manual backups + +The manual way requires knowledge of Forest internals and how it structures its +data directory (which is not guaranteed to stay the same). Thus, it is +recommended to use alternatives. + +## Backups with the `forest-tool` + +Forest comes with a `forest-tool` binary, which handles creating and recovering +backups. + +### Basic usage + +:warning: **The Forest node should be offline during the backup process, +especially when backing up the database** + +`forest-tool backup create` will create a backup file in the current working +directory. It will contain the p2p key-pair used to derive the `PeerId` and the +keystore. If storing anywhere, make sure to encrypt it. + +```shell +forest-tool backup create +``` + +Sample output: + +```console +Adding /home/rumcajs/.local/share/forest/libp2p/keypair to backup +Adding /home/rumcajs/.local/share/forest/keystore.json to backup +Backup complete: forest-backup-2024-02-22_17-18-43.tar +``` + +Afterwards, you can use `forest-tool backup restore ` to restore +those files. Note that this assumes that Forest is using the default +configuration - if it's not the case, provide the configuration `TOML` file via +the `--daemon-config` parameter. + +```shell +forest-tool backup restore forest-backup-2024-02-22_17-18-43.tar +``` + +Sample output: + +```console +Restoring /home/rumcajs/.local/share/forest/libp2p/keypair +Restoring /home/rumcajs/.local/share/forest/keystore.json +Restore complete +``` + +There are other flags to the backup tool, most notably `--all`, that will back +up the entire Forest data directory. Note that this includes the whole +blockstore, which, for mainnet, can reach hundreds of gigabytes. It is not +recommended outside development. + +### CLI reference + +Details on the `forest-tool backup` command and its subcommands can be found at the [CLI reference](../../reference/cli#forest-tool-backup). diff --git a/docs/docs/users/guides/advanced/generating_snapshots.md b/docs/docs/users/guides/advanced/generating_snapshots.md index 426546ee8402..8bac21e74413 100644 --- a/docs/docs/users/guides/advanced/generating_snapshots.md +++ b/docs/docs/users/guides/advanced/generating_snapshots.md @@ -2,3 +2,40 @@ title: Generating Snapshots sidebar_position: 1 --- + +# Snapshot exporting πŸ“Έ + +## Hardware requirements + +To export a mainnet snapshot, you need a setup with at least 16 GB of RAM. On a +machine with rapid NVMe, the default export should take around 30 +minutes. + +The requirements for calibnet snapshots are lower, but it is still recommended +to have at least 8 GB of RAM. The export should take less than a minute. + +## Running the node + +Wait until the node is fully synced. You can use the command: + +```shell +forest-cli sync wait +``` + +## Exporting the snapshot + +To export the snapshot with the defaults, run: + +```shell +forest-cli snapshot export +``` + +The snapshot will be exported with 2000 recent stateroots to the current directory. The snapshot will be +compressed. + +For mainnet, you should expect a file of over 70 GB. For calibnet, you should +expect a file of over 5 GB. Note that the snapshot size grows over time. + +### CLI reference + +Details on the `forest-cli snapshot export` command and its subcommands can be found at the [CLI reference](../../reference/cli#forest-cli-snapshot). diff --git a/docs/docs/users/guides/methods_filtering.md b/docs/docs/users/guides/methods_filtering.md index 7b330eab2a05..ef6627f9e56a 100644 --- a/docs/docs/users/guides/methods_filtering.md +++ b/docs/docs/users/guides/methods_filtering.md @@ -1,5 +1,6 @@ --- title: RPC methods filtering +sidebar_position: 5 --- # RPC methods filtering diff --git a/docs/docs/users/guides/monitoring/_category_.json b/docs/docs/users/guides/monitoring/_category_.json index de72be0badd6..0576abfe80ae 100644 --- a/docs/docs/users/guides/monitoring/_category_.json +++ b/docs/docs/users/guides/monitoring/_category_.json @@ -1,4 +1,4 @@ { "label": "Monitoring", - "position": 2 + "position": 6 } diff --git a/documentation/mindmaps/2023-07-14.md b/documentation/mindmaps/2023-07-14.md deleted file mode 100644 index d80c7b67ccce..000000000000 --- a/documentation/mindmaps/2023-07-14.md +++ /dev/null @@ -1,44 +0,0 @@ -# Mindmap of Forest work: 2023-07-03 and 2023-07-14 - -```mermaid -mindmap - root((🌲 Forest 🌲)) - id(πŸ‘· Maintenance πŸ‘·
25 PRs) - (Dead code) - (Dependencies) - (Streamify) - (Tracing) - id(πŸ“¦ Features πŸ“¦
11 PRs) - (CLI commands) - (CAR backed blockstore) - (Network Version 17) - (Progress log) - id))🐞 Bug fixes 🐞
2 PRs(( - Abort on panic - Get null tipset - id{{β™» CI/Project β™»
6 PRs}} - (Merge queue) - (Disable devnet) - id(πŸ’‘ Infrastructure πŸ’‘) - (Custom monitoring) -``` - -















- -# Sprint goals: - -```mermaid -mindmap - root((🌲 Forest 🌲)) - id(Features) - Indexed zstd.car - Diff files - Archive - id(Correctness) - Validation of random tipsets - Test mainnet migrations - id(Internal) - Forest tool - More benchmarks - Maintenance -``` diff --git a/documentation/src/SUMMARY.md b/documentation/src/SUMMARY.md index fbff7aa61d52..c012fa979e5b 100644 --- a/documentation/src/SUMMARY.md +++ b/documentation/src/SUMMARY.md @@ -3,28 +3,20 @@ # User Guide - [Introduction](./introduction.md) - - [Snapshot exporting](./snapshot_export.md) - - [Backups](./backups.md) - [Troubleshooting](./trouble_shooting.md) - - [Glossary](./glossary.md) - [Bootstrap node](./bootstrap_node.md) - [Offline Forest](./offline-forest.md) # Developer documentation - [Developer documentation](./developer_documentation/introduction.md) - - [Application architecture](./developer_documentation/application_architecture.md) - [Database migrations](./developer_documentation/database_migrations.md) - [Local GH Actions](./developer_documentation/local_actions.md) - - [Mainnet compatibility](./developer_documentation/mainnet_compatibility.md) - [Memory analysis](./developer_documentation/memory-analysis.md) - [Release checklist](./developer_documentation/release_checklist.md) - - [State migration](./developer_documentation/state_migration.md) - [State migration guide](./developer_documentation/state_migration_guide.md) - - [State migration spike NV17-NV18](./developer_documentation/state_migration_spike.md) - [Test plan](./developer_documentation/test_plan.md) - [Devnet Notes](./developer_documentation/devnet_notes.md) - [Archie and Fuzzy](./developer_documentation/archie_and_fuzzy.md) - [RPC API Compatibility](./developer_documentation/rpc_api_compatibility.md) - - [Notes and sketches](./developer_documentation/notes_and_sketches.md) - [ChainMuxer/TipsetProcessor state machine](./developer_documentation/chain_muxer_state_machine.md) diff --git a/documentation/src/backups.md b/documentation/src/backups.md deleted file mode 100644 index 8870dc593ad5..000000000000 --- a/documentation/src/backups.md +++ /dev/null @@ -1,95 +0,0 @@ -# Forest Backups - -> "_The condition of any backup is unknown until a restore is attempted._" -> Everyone who deals with backups. - -## Manual backups - -The manual way requires knowledge of Forest internals and how it structures its -data directory (which is not guaranteed to stay the same). Thus, it is -recommended to use alternatives. - -## Backups with the `forest-tool` - -Forest comes with a `forest-tool` binary, which handles creating and recovering -backups. - -### Basic usage - -:warning: **The Forest node should be offline during the backup process, -especially when backing up the blockstore.** - -`forest-tool backup create` will create a backup file in the current working -directory. It will contain the p2p keypair used to derive the `PeerId` and the -keystore. If storing anywhere, make sure to encrypt it. - -``` -❯ forest-tool backup create -Adding /home/rumcajs/.local/share/forest/libp2p/keypair to backup -Adding /home/rumcajs/.local/share/forest/keystore.json to backup -Backup complete: forest-backup-2024-02-22_17-18-43.tar -``` - -Afterwards, you can use `forest-tool backup restore ` to restore -those files. Note that this assumes that Forest is using the default -configuration - if it's not the case, provide the configuration TOML file via -the `--daemon-config` parameter. - -``` -❯ forest-tool backup restore forest-backup-2024-02-22_17-18-43.tar -Restoring /home/rumcajs/.local/share/forest/libp2p/keypair -Restoring /home/rumcajs/.local/share/forest/keystore.json -Restore complete -``` - -There are other flags to the backup tool, most notably `--all`, that will back -up the entire Forest data directory. Note that this includes the whole -blockstore, which, for mainnet, can reach hundreds of gigabytes. It is not -recommended outside development. - -### `backup` - -``` -Create and restore backups - -Usage: forest-tool backup - -Commands: - create Create a backup of the node. By default, only the p2p keypair and keystore are backed up. The node must be offline - restore Restore a backup of the node from a file. The node must be offline - help Print this message or the help of the given subcommand(s) -``` - -### `backup create` - -``` -Create a backup of the node. By default, only the p2p keypair and keystore are backed up. The node must be offline - -Usage: forest-tool backup create [OPTIONS] - -Options: - --backup-file Path to the output backup file if not using the default - --all Backup everything from the Forest data directory. This will override other options - --no-keypair Disables backing up the keypair - --no-keystore Disables backing up the keystore - --backup-chain Backs up the blockstore for the specified chain. If not provided, it will not be backed up - --include-proof-params Include proof parameters in the backup - -d, --daemon-config Optional TOML file containing forest daemon configuration. If not provided, the default configuration will be used - -h, --help Print help -``` - -### `backup restore` - -``` -Restore a backup of the node from a file. The node must be offline - -Usage: forest-tool backup restore [OPTIONS] - -Arguments: - Path to the backup file - -Options: - -d, --daemon-config Optional TOML file containing forest daemon configuration. If not provided, the default configuration will be used - --force Force restore even if files already exist WARNING: This will overwrite existing files - -h, --help Print help -``` diff --git a/documentation/src/developer_documentation/mainnet_compatibility.md b/documentation/src/developer_documentation/mainnet_compatibility.md deleted file mode 100644 index 8261a2a0fbda..000000000000 --- a/documentation/src/developer_documentation/mainnet_compatibility.md +++ /dev/null @@ -1,84 +0,0 @@ -# Testing for Mainnet Compatibility - -Forest development can be like hitting a moving target and sometimes Forest -falls behind the network. This document should serve as a way to easily identify -if Forest can sync all the way up to the network head using a simple -step-by-step process. - -## Prerequisites - -Some command-line tools and software is required to follow this guide. - -- A fresh copy of the Forest repository that has been built -- Lotus installed -- curl (to download snapshots) -- sha256sum (optional, used to verify snapshot integrity) - -## Grab a snapshot and run Forest - -Refer to the mdbook documentation on how to download a snapshot and run forest - -Warning: FileCoin snapshots as of this writing are over 75GB. Verify you have -enough space on your system to accommodate these large files. - -- Use `make mdbook` in Forest's root directory -- Open `http://localhost:3000` -- Navigate to `2. Basic Usage` in the menu on the right -- Scroll down to `Forest Import Snapshot Mode` - -## Let Forest sync - -This step may take a while. We want Forest to get as far along in the syncing -process as it can get. If it syncs up all the way to the network head, CONGRATS! -Forest is up to date and on mainnet. Otherwise, Forest is not on mainnet. - -If Forest starts to error and can't get past a block while syncing. Make note of -which block it is. We can use that block to help debug any potential state -mismatches. - -## Is Forest on the latest network version? - -Something easy to check is if Forest is on the latest Filecoin network version. -A repository exists where we can see all of the released network versions -[here](https://github.com/filecoin-project/tpm/tree/master/Network%20Upgrades). -Navigate the codebase to see mention of the latest network upgrade. If a -snapshot fails to sync at a certain epoch, it's entirely possible that the -snapshot was behind an epoch when a version upgrade started. Grab a new snapshot -by referring to the mdbook documentation. - -## Debugging State Mismatches - -Statediffs can only be printed if we import a snapshot containing the stateroot -data from Lotus. This means there will not be a pretty statediff if Forest is -already synced to the network when the stateroot mismatch happens. By default, -snapshots only contain stateroot data for the previous 2000 epochs. So, if you -have a statediff at epoch X, download a snapshot for epoch X+100 and tell Forest -to re-validate the snapshot from epoch X. - -Steps to print a state-diff: - -1. Note the epoch of the state-root mismatch. State-roots can only be checked - for the parents of a tipset so the failing epoch may be 1 higher than you - think. -2. Download a recent snapshot dated _before_ the failing epoch. -3. Import the snapshot into Lotus and sync to HEAD. -4. Export a new snapshot 100 epochs _after_ the failing epoch. -5. Convert the `.car.zst` file to `.forest.car.zst` with: - `forest-tool snapshot compress {snapshot.car.zst}` -6. Use the `forest-tool` binary to print the state-diff: - `forest-tool archive diff {snapshot.forest.car.zst} --epoch {failing_epoch}` - -## FVM Traces - -Within FVM, we can enable tracing to produce execution traces. Given an -offending epoch, we can produce them both for Forest and for Lotus to find -mismatches. - -To confirm: the execution traces format is not uniform across implementations, -so it takes a certain amount of elbow grease to find the differences. Lotus is -capable of spitting this out in JSON for nice UX - -## Dated resources - -For more (but dated) information, see -[this document.](https://www.notion.so/chainsafe/Interop-debugging-6adabf9222d7449bbfeaacb1ec997cf8) diff --git a/documentation/src/developer_documentation/notes_and_sketches.md b/documentation/src/developer_documentation/notes_and_sketches.md deleted file mode 100644 index 30e49be4e7ee..000000000000 --- a/documentation/src/developer_documentation/notes_and_sketches.md +++ /dev/null @@ -1,3 +0,0 @@ -# Notes and sketches - -The notes found here may be out of date. diff --git a/documentation/src/developer_documentation/state_migration.md b/documentation/src/developer_documentation/state_migration.md deleted file mode 100644 index 3ed580e83d0d..000000000000 --- a/documentation/src/developer_documentation/state_migration.md +++ /dev/null @@ -1,117 +0,0 @@ -It's unclear how we can support migrations without adding a lot of code -complexity. This document is meant to shed light on the matter and illuminate a -sustainable path forward. As a start we will consider a migration going from -nv15 to nv16. - -# Migration path investigation from nv15 to nv16 - -## Findings - -1. Actor IDs definitely changed - -For following actors only their CID have changed: - -- init -- cron -- account -- power -- miner -- paymentchannel -- multisig -- reward -- verifiedregistry - -Those are just simple code migration. - -For system and market actors there's both code and state changes. That's why -there is dedicated logic for their migration. - -The system actor need to update the state tree with its new state that holds now -the `ManifestData` CID. - -For the market actor more work is involved to upgrade actor state due to support -for UTF-8 string label encoding in deal proposals and pending proposals (see -[FIP-0027](https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0027.md)). - -2. Some gas calculations changed? - -I don't think we are concerned by this. Gas metering can change at a given -protocol upgrade for one or many actors but the impact is irrelevant as it -doesn't modify blockchain data structures. Gas calculations should only impact -code and in our case the nv16 ref-fvm is already supporting the new gas changes. - -3. drand calculation changed? - -Ditto. - -4. What else changed? - -Nothing else as far I can see. - -## Open questions - -- pre-migration framework + caching: how much do we need a similar approach in - Forest? Are there other alternatives? We can definitely skip this part at - first. For information the old nv12 state migration in forest took around - 13-15 secs. - -- Seen in Lotus: `UpgradeRefuelHeight`. What's Refuel for? - -- Migration logic is in spec-actors (go actors), what the future of this given - clients moved to builtin-actors (rust actors) and ref-fvm? In an ideal world - we might want a shared migration logic. - -- Implement Lite migration? - - > should allow for easy upgrades if actors code needs to change but state does - > not. Example provided above the function to perform all the migration - > duties. Check actors_version_checklist.md for the rest of the steps. - -- What are non-deferred actors in the context of a migration? - -- The `migrationJobResult` struct is using a `states7` actor instead of a - `states8` one (in go spec-actors). Typo or are there some good reasons? - -## Changes rough proposal - -To support nv15 to nv16 migration we need to: - -- [ ] Make forest sync again on nv15 and be able to support multiple network - versions. -- [ ] Understand existing forest migration framework (used in the past for nv12 - migration). Can we reuse most of the code as is? -- [ ] Implementation of the nv16 migration logic (replicating same logic as in - spec-actors). -- [ ] Implementation of unit tests covering this migration. -- [ ] Implementation of a migration schedule that will select the right - migration path. -- [ ] Test migration using the exported calibnet and mainnet snapshots and - respectively measure the elapsed time and memory usage. - -## Test snapshots - -For testing a calibnet migration two snapshots have been exported with Lotus: - -- lotus_snapshot_2022-Aug-5_height_1044460.car -- lotus_snapshot_2022-Aug-5_height_1044659.car - -They are respectively exported 200 and 1 epochs before the Skyr upgrade (the 200 -version could be useful if we decide to implement a pre-migration like in -Lotus). - -For testing a mainnet migration, one snapshot has been retrieved from Protocol -Labs s3 bucket using the [lily-shed](https://github.com/kasteph/lily-shed/) -util: - -- minimal_finality_stateroots_1955760_2022-07-05_00-00-00.car - -This one is 4560 epochs before. If needed we can extract closer snapshots later. - -Those snapshots have been uploaded to our Digital Ocean Spaces. - -## Additional resources - -`what changed` between versions is maintained in the -[tpm repo](https://github.com/filecoin-project/tpm/tree/master/Network%20Upgrades), -e.g. all the changes in -[NV15 -> NV16](https://github.com/filecoin-project/tpm/blob/master/Network%20Upgrades/v16.md) diff --git a/documentation/src/developer_documentation/state_migration_spike.md b/documentation/src/developer_documentation/state_migration_spike.md deleted file mode 100644 index d19b10222927..000000000000 --- a/documentation/src/developer_documentation/state_migration_spike.md +++ /dev/null @@ -1,149 +0,0 @@ -## State migration spike πŸ›‚ - -### What is state migration? - -State migration is a process where the `StateTree` contents are transformed from -an older form to a newer form. Certain Actors may need to be created or migrated -as well. - -### Why do we need to migrate? - -Migration is required when the structure of the state changes. This happens when -new fields are added or existing ones are modified. Migration is **not** -required in case of new behaviour. - -In case of NV18, the `StateTree` changed from version 4 to version 5. See -https://github.com/filecoin-project/ref-fvm/pull/1062 - -### What to upgrade? - -We need to upgrade the `StateTree` which is represented as -`HAMT` to the latest version. - -On top of that, we need to migrate certain actors. In the case of NV18 upgrade, -it's the -[init](https://github.com/filecoin-project/go-state-types/blob/d8fdbda2ad86de55bcde7f567c6da9c5f430c7a1/builtin/v10/migration/init.go#L32) -and -[system](https://github.com/filecoin-project/go-state-types/blob/master/builtin/v10/migration/system.go#L24) -actor. -[EAM](https://github.com/filecoin-project/go-state-types/blob/master/builtin/v10/migration/eam.go) -actor needs to be created. - -### When to upgrade? - -There is a separate upgrade schedule for each network. In Lotus, it is defined -in -[upgrades.go](https://github.com/filecoin-project/lotus/blob/dbbcf4b2ee9626796e23a096c66e67ff350810e4/chain/consensus/filcns/upgrades.go#L83). -In Venus, in -[fork.go](https://github.com/filecoin-project/venus/blob/master/pkg/fork/fork.go) -which has the same structure. - -For the case of NV18, it is defined as - -```go -Height: build.UpgradeHyggeHeight, -Network: network.Version18, -Migration: UpgradeActorsV10, -PreMigrations: []stmgr.PreMigration{{ - PreMigration: PreUpgradeActorsV10, - StartWithin: 60, - DontStartWithin: 10, - StopWithin: 5, -}}, -Expensive: true, -``` - -### How to upgrade? - -Iterate over the state of each actor at the given epoch and write the new state -along with any specific changes to the respective state. This involves iterating -over each of the HAMT nodes storing the state and writing them to the database. - -[Lotus upgrade method](https://github.com/filecoin-project/lotus/blob/58900a70333a11a903cf9fe3f29e6a5c309cb000/chain/consensus/filcns/upgrades.go#L1591-L1612) -and the -[module](https://github.com/filecoin-project/go-state-types/tree/master/builtin/v10/migration) -dedicated to Actors `v10` migration. The core logic is -[here](https://github.com/filecoin-project/go-state-types/blob/master/builtin/v10/migration/top.go#L28). -The same module is used by Venus. - -Forks migrations: handled by -[fork.go](https://github.com/filecoin-project/lotus/blob/58900a70333a11a903cf9fe3f29e6a5c309cb000/chain/stmgr/forks.go#L42-L53) -entities. - -### Where to upgrade? - -It should be done most likely in the apply blocks method. - -[Lotus](https://github.com/filecoin-project/lotus/blob/74d94af03418c799350fc0f40d3758c23cd82ab8/chain/consensus/compute_state.go#L178): - -```go -// handle state forks -// XXX: The state tree -pstate, err = sm.HandleStateForks(ctx, pstate, i, em, ts) -if err != nil { - return cid.Undef, cid.Undef, xerrors.Errorf("error handling state forks: %w", err) -} -``` - -In -[Forest](https://github.com/ChainSafe/forest/blob/main/blockchain/state_manager/src/lib.rs#L421-L424) -we already have a hint from the past: - -```rust -if epoch_i == turbo_height { - todo!("cannot migrate state when using FVM - see https://github.com/ChainSafe/forest/issues/1454 for updates"); -} -``` - -We can try with something simplistic to get it running, it's not an issue. -Afterwards we can implement a proper schedule with functors. - -### Challenges - -- Doing the state migration efficiently; we need to traverse every entry in the - state trie. Lotus does pre-migration which are filling relevant caches to - speed up the eventual full migration at the upgrade epoch. We might need to do - something like this as well; it might not be necessary for the first - iteration - depends on how performant the migration process would be in the - Forest itself. -- No test network. While we can use existing snapshots from before the upgrade - to test state migration, it is not sustainable if we want to continuously - support calibration network. We either require a local devnet for testing - migration **before** they actually happen on real networks or we can try - supporting more bleeding-edge networks. The former approach is more solid, but - the latter might be easier to implement at first (and would give Forest more - testnets support which is always welcome). -- There may be forks, so we probably need to keep the pre-migration and - post-migration state in two caches for some back and forths. This in Lotus is - handled with - [HandleStateForks](https://github.com/filecoin-project/lotus/blob/f641139bf237e6e955a9a2f33cfc05ba52430b1b/chain/stmgr/forks.go#L175). -- For EAM Actor we may need some Ethereum methods we have not yet implemented. - Perhaps what `builtin-actors` and `ref-fvm` expose will be enough. - -### Current Forest implementation - -For the moment Forest does not support migrations. The -[code](https://github.com/ChainSafe/forest/blob/state-migration-spike/vm/state_migration/src/lib.rs) -that was meant for this is not used at the moment. Most probably we will be able -to utilise it. - -### Plan - -We should start by adding an `nv18` to the state migration -[crate](https://github.com/ChainSafe/forest/tree/state-migration-spike/vm/state_migration/src), -along the lines of the -[Go equivalent](https://github.com/filecoin-project/go-state-types/blob/master/builtin/v10/migration/init.go). -Most likely this would mean adding some missing structures, related to the `v10` -actors (Ethereum ones). - -Then try to plug it in -[apply_blocks](https://github.com/ChainSafe/forest/blob/main/blockchain/state_manager/src/lib.rs#L421-L424). -This may work for calibration network. Afterwards, we will most likely need to -iterate to achieve acceptable performance for mainnet. Some ideas on how to -achieve this can be taken from Lotus/Venus, e.g., pre-migration caching. - -### Sources - -- Rahul's article: https://hackmd.io/@tbdrqGmwSXiPjxgteK3hMg/r1D6cVM_u -- Lotus codebase - https://github.com/filecoin-project/lotus -- Venus codebase - https://github.com/filecoin-project/venus diff --git a/documentation/src/glossary.md b/documentation/src/glossary.md deleted file mode 100644 index 785387a7d3c7..000000000000 --- a/documentation/src/glossary.md +++ /dev/null @@ -1,122 +0,0 @@ -## Algorand - -[Algorand](https://www.algorand.com/) is a proof-of-stake blockchain -cryptocurrency protocol. - -## BLAKE2b - -BLAKE2 is a cryptographic hash function based on -[BLAKE](). The design goal -was to replace the widely used, but broken, MD5 and SHA-1 algorithms in -applications requiring high performance in software. - -## BLS - -[BLS](https://en.wikipedia.org/wiki/BLS_digital_signature) stands for -Boneh–Lynn–Shacham cryptographic signature scheme, which is a cryptographic -signature scheme which allows a user to verify that a signer is authentic. - -## CBOR - -[CBOR](https://cbor.io/) stands for the Concise Binary Object Representation, -which is a data format whose design goals include the possibility of extremely -small code size, fairly small message size, and extensibility without the need -for version negotiation. - -## CID - -[CID](https://spec.filecoin.io/#section-libraries.multiformats.cids) is short -for Content Identifier, a self describing content address used throughout the -IPFS ecosystem. CIDs are used in Filecoin to identify files submitted to the -decentralized storage network. For more detailed information, see -[the github documentation for it](https://github.com/ipld/cid). - -## IPLD - -[IPLD](https://github.com/ipld) stands for InterPlanetary Linked Data, which is -a series of standards and formats for describing data in a -content-addressing-emphatic way. - -## JWT - -[JWT](https://en.wikipedia.org/wiki/JSON_Web_Token) stands for JSON Web Token, -which is a proposed Internet standard for creating data with optional signature -and/or optional encryption whose payload holds JSON that asserts some number of -claims. The tokens are signed either using a private secret or a public/private -key. - -## SECP - -types of elliptic curves used for ECDSA see -[here](https://www.johndcook.com/blog/2018/08/21/a-tale-of-two-elliptic-curves/) - -## multisig - -A [multi-signature](https://lotus.filecoin.io/lotus/manage/multisig/) (multisig) -wallet refers to a wallet that requires multiple keys to authorize a `FIL` -transactions. - -## Tipset - -Tipset is a structure that contains a non-empty collection of blocks that have -distinct miners and all specify identical epoch, parents, weight, height, state -root, receipt root - -## Tipsetkey - -A set of CIDs forming a unique key for a tipset. - -## mempool - -mempool stands for the Message Pool, which is the component of forest that -handles pending messages for inclusion in the chain. Messages are added either -directly for locally published messages or through pubsub propagation. - -## Merkle - -[Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree) is a tree in which -every node is labelled with the cryptographic hash of a data block, and every -node that is not a leaf (called a branch, inner node, or inode) is labelled with -the cryptographic hash of the labels of its child nodes. A hash tree allows -efficient and secure verification of the contents of a large data structure. - -## IPFS - -[IPFS](https://github.com/ipfs/ipfs) stands for InterPlanetary File System which -is a peer-to-peer hypermedia protocol to make the web faster, safer, and more -open. - -## Proof of Spacetime (PoSt) - -PoSt stands for Proof-of-Spacetime is a procedure by which a storage-miner can -prove to the Filecoin network they have stored and continue to store a unique -copy of some data on behalf of the network for a period of time. - -## HAMT - -[HAMT](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) stands for Hash -array mapped trie, which is an implementation of an associative array that -combines the characteristics of a hash table and an array mapped trie. - -## VRF - -[VRF](https://en.wikipedia.org/wiki/Verifiable_random_function) stands for a -Verifiable Random Function that receives a Secret Key (SK) and a seed and -outputs proof of correctness and output value. VRFs must yield a proof of -correctness and a unique & efficiently verifiable output. - -## MDNS - -[MDNS](https://en.wikipedia.org/wiki/Multicast_DNS) stands for Multicast DNS, -which is a protocol, that resolves hostnames to IP addresses within small -networks that do not include a local name server. - -## Kademlia - -[Kademlia](https://en.wikipedia.org/wiki/Kademlia) is a distributed hash table -for decentralized peer-to-peer computer networks. - -## LibP2P - -[LibP2P](https://libp2p.io/) is a modular system of protocols, specifications -and libraries that enable the development of peer-to-peer network applications. diff --git a/documentation/src/snapshot_export.md b/documentation/src/snapshot_export.md deleted file mode 100644 index befa181b568b..000000000000 --- a/documentation/src/snapshot_export.md +++ /dev/null @@ -1,50 +0,0 @@ -# Snapshot exporting πŸ“Έ - -## Hardware requirements - -To export a mainnet snapshot, you need a setup with at least 10 GB of RAM. On a -machine with rapid NVMe SSD (around 7000MB/s), the export should take around 30 -minutes. - -The requirements for calibnet snapshots are lower, but it is still recommended -to have at least 4 GB of RAM. The export should take less than a minute. - -## Running the node - -You need to have a running node to be able to export a snapshot. If you don't -have one, you can follow the [usage guide](./basic_usage.md). - -Wait until the node is fully synced. You can use the command: - -```shell -forest-cli sync wait -``` - -## Exporting the snapshot - -Usage of the ` snapshot export` command: - -```shell -Usage: forest-cli snapshot export [OPTIONS] - -Options: - -o Snapshot output filename or directory. Defaults to - `./forest_snapshot_{chain}_{year}-{month}-{day}_height_{epoch}.car.zst`. [default: .] - --skip-checksum Skip creating the checksum file - --dry-run Don't write the archive - -h, --help Print help -``` - -The snapshot will be exported with 2000 recent stateroots. - -To export the snapshot with the defaults, run: - -```shell -forest-cli snapshot export -``` - -it will write the snapshot to the current directory. The snapshot will be -compressed. - -For mainnet, you should expect a file of over 50 GB. For calibnet, you should -expect a file of around 1-2 GB. From a5ea6bb0f059c98f0276ac32894a4d8127071f28 Mon Sep 17 00:00:00 2001 From: Hubert Bugaj Date: Fri, 21 Feb 2025 12:21:30 +0100 Subject: [PATCH 2/2] fix: link check --- docs/docs/users/guides/advanced/backups.md | 2 +- docs/docs/users/guides/advanced/generating_snapshots.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/users/guides/advanced/backups.md b/docs/docs/users/guides/advanced/backups.md index b373dbf3a932..f7b52bc566f8 100644 --- a/docs/docs/users/guides/advanced/backups.md +++ b/docs/docs/users/guides/advanced/backups.md @@ -59,4 +59,4 @@ recommended outside development. ### CLI reference -Details on the `forest-tool backup` command and its subcommands can be found at the [CLI reference](../../reference/cli#forest-tool-backup). +Details on the `forest-tool backup` command and its subcommands can be found at the [CLI reference](../../reference/cli.md#forest-tool-backup). diff --git a/docs/docs/users/guides/advanced/generating_snapshots.md b/docs/docs/users/guides/advanced/generating_snapshots.md index 8bac21e74413..931f63fc335f 100644 --- a/docs/docs/users/guides/advanced/generating_snapshots.md +++ b/docs/docs/users/guides/advanced/generating_snapshots.md @@ -38,4 +38,4 @@ expect a file of over 5 GB. Note that the snapshot size grows over time. ### CLI reference -Details on the `forest-cli snapshot export` command and its subcommands can be found at the [CLI reference](../../reference/cli#forest-cli-snapshot). +Details on the `forest-cli snapshot export` command and its subcommands can be found at the [CLI reference](../../reference/cli.md#forest-cli-snapshot).