From 526c9b2390e45b056c98f64984d41fde255ee9fb Mon Sep 17 00:00:00 2001 From: Fynn Date: Fri, 15 Mar 2024 17:49:45 +0800 Subject: [PATCH] BEP-365: Support multi-database based on data pattern --- BEPs/{BEP363.md => BEP365.md} | 52 +++++++++++++++----- BEPs/assets/{bep-363 => bep-365}/folder.png | Bin 2 files changed, 39 insertions(+), 13 deletions(-) rename BEPs/{BEP363.md => BEP365.md} (91%) rename BEPs/assets/{bep-363 => bep-365}/folder.png (100%) diff --git a/BEPs/BEP363.md b/BEPs/BEP365.md similarity index 91% rename from BEPs/BEP363.md rename to BEPs/BEP365.md index 8e6c25bb..3e657fed 100644 --- a/BEPs/BEP363.md +++ b/BEPs/BEP365.md @@ -1,8 +1,34 @@ -# 1. Summary +
+  BEP: 365
+  Title: Support multi-database based on data pattern
+  Status: Draft
+  Type: Standards
+  Created: 2024-03-15
+
+ + +# BEP-364: Support multi-database based on data pattern + +- [BEP-364: Support multi-database based on data pattern](#bep-364-support-multi-database-based-on-data-pattern) + - [1. Summary](#1-summary) + - [2. Motivation](#2-motivation) + - [3. Specification](#3-specification) + - [3.1 Multi-Databases](#31-multi-databases) + - [3.2 Folder Structure](#32-folder-structure) + - [4. Rationale](#4-rationale) + - [4.1 Block DataBase](#41-block-database) + - [4.2 Trie DataBase](#42-trie-database) + - [4.3 Original DataBase](#43-original-database) + - [5. Compatibility](#5-compatibility) + - [6. Test](#6-test) + - [7. License](#7-license) + + +## 1. Summary This BEP proposes an efficient storage solution on the BSC(BNB Smart Chain) which can improve the performance and maintainability significantly by splitting blockchain data into multi-databases based on data patterns. -# 2. Motivation +## 2. Motivation Currently, all of the BSC node's data, except for historical block and state data, is stored in a single key-value database instance, with different types of data segregated by different prefixes, as shown in the table below. With the rapid increase in the amount of data, several problems are being faced: @@ -43,9 +69,9 @@ Currently, all of the BSC node's data, except for historical block and state dat This BEP aims to address these issues and provide a more efficient and high-performance way to store and maintain block and state data. -# 3. Specification +## 3. Specification -## 3.1 Multi-Databases +### 3.1 Multi-Databases Chaindata will be divided into three stores, BlockStore, TrieStore, and Others, according to data schema and read/write behavior. After database separation, the data layout obtained by `db inspect` is as the below table shows: @@ -79,17 +105,17 @@ Chaindata will be divided into three stores, BlockStore, TrieStore, and Others, | | | Bloombit index | | | | | Contract codes | | -## 3.2 Folder Structure +### 3.2 Folder Structure The folder structure for multiple databases is shown below. The original database is located within the `chaindata/` folder, and new `block/` and `state/` folders have been introduced to store block and trie data. Additionally, there is an `ancient` folder for storing historical data under each of these directories. -[folder structure](./assets/bep-363/folder.png) +[folder structure](./assets/bep-365/folder.png) -# 4. Rationale +## 4. Rationale Separate databases according to data schema and read/write behavior can improve the performance, scalability, and maintainability of chain nodes. -## 4.1 Block DataBase +### 4.1 Block DataBase Diving deeper into the data flow of block-related data, recent blocks are first stored in a key-value (KV) database and then appended sequentially to the ancient database and deleted from the KV database when reaching the ancient threshold, which wastes some disk bandwidth. @@ -97,26 +123,26 @@ Originally the latest 90K blocks will be retained in the key-value database by d In summation, the block data retained in the kv database will be reduced to 20-30 blocks after using the finalized block as the chain freeze indicator, this part of the data can be stored in memory, but considering the robustness and stability of the node, using a separated kv database will be more reasonable and probability most of the block data are deleted when flush from memtable to sst file and then append them to the ancient database. Therefore, unnecessary bandwidth consumption can be reduced. -## 4.2 Trie DataBase +### 4.2 Trie DataBase The trie node data constitutes roughly half of the overall key-value database volume and grows rapidly. Latency to read and write trie node data significantly impacts the block execution and verification performance. Improving its read and write speed can contribute to an overall improvement in blockchain performance. Through in-depth analysis of the data model and read/write behavior of Trie node data, it is evident that Trie nodes exhibit significant overwrite operations in path mode. The keys are relatively ordered along the paths. If this data is stored with other hash-keyed data in the same key-value database, it would significantly amplify the cost of database compaction, leading to senseless bandwidth consumption. If splitting the database for trie data, the independent db can process compaction with a more simplified LSM hierarchy, which would reduce the read/write latency of the entire database and improve the performance. -## 4.3 Original DataBase +### 4.3 Original DataBase After separating the block and trie data, the remaining data is stored in the original database containing the account/storage snapshot, txIndex, and contract code. Due to the reduction in the amount of data, the depth of the LSM tree is reduced, thus the read/write performance of the original database will be improved. It is worth mentioning that during the execution phase of the blockchain, there is frequent access to snapshot data. This reduction of reading latency of snapshot data will contribute significantly to overall execution performance. -# 5. Compatibility +## 5. Compatibility There are no compatibility issues and no hard fork is required, but the node needs to be resync. -# 6. Test +## 6. Test TBD -# 7. License +## 7. License The content is licensed under [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/BEPs/assets/bep-363/folder.png b/BEPs/assets/bep-365/folder.png similarity index 100% rename from BEPs/assets/bep-363/folder.png rename to BEPs/assets/bep-365/folder.png