Stonemq aims to outperform Kafka in scenarios with massive-scale queue clusters, delivering enhanced performance and efficiency to achieve cost reduction and operational optimization.
In use cases involving large clusters with countless queues—particularly in enterprise business services and public cloud services—there can be tens of thousands of partitions and partition leaders. Regardless of whether the queues contain messages, the volume of messages, or the flow rate, the presence or absence of active consumers in each partition poses a significant burden for cluster operators. Partition growth negatively impacts cluster throughput, while node failures or restarts often result in partition leader and controller switching, creating critical operational challenges. Stonemq addresses these inefficiencies.
Cluster performance should remain consistent regardless of partition growth. Queues with varying traffic volumes need consolidation to enable more efficient message flow—akin to containerized shipping for multiple clients. This is precisely the vision of Stonemq. While solutions like Pulsar utilize journaling for centralized message handling, Stonemq seeks to retain Kafka’s replication , which simplifies and standardizes cluster operation and maintenance.We believe this protocol is both straightforward and highly efficient, forming the backbone of our solution.Additionally, by reusing Kafka's client-server communication protocol, StoneMQ ensures seamless migration without requiring any changes to the user's client applications. This approach significantly reduces adoption costs for users, enabling a smooth transition to StoneMQ while retaining the familiar and reliable interface they are accustomed to.
For a more detailed analysis and introduction to the project's inception, please refer to the blog: Blog Link.
- version 0.1.0 released
- Supports single-node message sending and receiving.
- Implements group consumption functionality.
-
Make sure you have Rust installed (recommended via Rustup):
rustc --version
-
Clone the repository:
git clone https://github.com/jonefeewang/stonemq.git cd stonemq -
Build the release version using Cargo:
cargo build --release
-
Run the application:
./target/release/stonemq
Run the following command to launch the StoneMQ server:
./stonemqUse a Kafka client library (or REST API) with a minimum supported version of Kafka v0.11 to connect to the StoneMQ server for publishing and subscribing to messages. StoneMQ uses the same default server port as Kafka: 9092. This port can be customized in the configuration file.
StoneMQ comes with a built-in topic, topic_a, which is pre-configured with 2 partitions:
topic_a-0topic_a-1
By default, the system sets up with 2 journals.
The configuration file, conf.toml, is located in the project’s root directory. Adjust settings as needed to customize the server's behavior.
StoneMQ provides a graceful shutdown mechanism that allows the server to complete all pending tasks before exiting. This is particularly useful in scenarios where the server needs to ensure all messages are processed or stored before shutting down.
To trigger a graceful shutdown, you can send a termination signal to the server. In a Unix-like environment, you can use Ctrl+C to send the SIGINT signal or use the following command to send the SIGINT signal:
kill -2 <pid>This will allow the server to perform a graceful shutdown, ensuring all pending messages are processed and stored.
-
Install Rust Toolchain: Ensure Rust is installed on your system. Use Rustup to get started:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh rustup default stable
-
Clone the Repository:
git clone https://github.com/jonefeewang/stonemq.git cd stonemq -
Run Tests: Run all unit tests to ensure everything is working correctly:
cargo test -
Check Code Formatting: Ensure the code adheres to the formatting rules:
cargo fmt -- --check
-
Lint (Static Analysis):
Run clippy to check for potential issues:
cargo clippy
-
Log Parsing
An executable log_parser, source file is
log_parser.rs, is available in the project'sbindirectory. This tool can be used to parselog,index,andcheckpointfiles. -
Benchmarking and Monitoring
The entry file
stonemq.rsin thebindirectory allows you to specify the current running mode: prod, dev, or perf. The default mode is prod.- Dev Mode: Suitable for development.
- Perf Mode: Enables the use of tokio-console to monitor the state of Tokio tasks within the process.
Additionally, you can specify whether to enable trace. When enabled, added spans can be observed in Open Jaeger for in-depth monitoring.
You can also specify the log output level through environment variables. The project uses dotenv crate to configure environment variables for ease of management and flexibility.
Contributions are welcome and greatly appreciated! Whether it's reporting a bug, submitting code, improving documentation, or suggesting ideas, your help is valuable.
- Fork the repository and create a new branch for your feature/bugfix.
- Ensure all tests pass, and your code adheres to the project's standards.
- Submit a pull request with a clear description of your changes.
If you encounter issues or have questions, feel free to reach out to the community:
- 🐛 Submit Issues: GitHub Issues
- 📢 Join Discussions: GitHub Discussions
Stonemq is distributed under the Apache License 2.0. See the LICENSE file for more details.