vm broker ‐ lxmin implementation for backups
This walkthru explains how to setup multiple lxmin servers backing up to a multinode multidrive minio cluster.
Each vm broker server contains its own lxmin service as a means of reducing load on the backup system i.e.
- a vm broker request for a specific instance goes to the node where it is hosted.
- a vm broker request for a specific backup goes to the node which is able to list it. In turn the lxmin services connect to a single minio endpoint which, in this example, allows access to a multinode multidrive minio cluster.
Sample implementation in vm broker:
Obtain wildcard certificates issued by a signing authority. Copy to for following files on the server.
In this example, I used *.lab.min.dev certificates
mkdir -p $HOME/.minio/certs/CAs
mkdir -p $HOME/.minio/certs_intel/CAs
vi $HOME/.minio/certs_intel/public.crt
vi $HOME/.minio/certs_intel/private.key
mkdir $HOME/lxmin
cd $HOME/lxmin
wget https://github.com/minio/lxmin/releases/latest/download/lxmin-linux-amd64
chmod +x lxmin-linux-amd64
sudo mv lxmin-linux-amd64 /usr/local/bin/lxmin-allan
NOTE - May need to run as User=root and Group=root due to error: "Unable get instance config"
sudo vi /etc/systemd/system/lxmin-allan.service
###
[Unit]
Description=Lxmin
Documentation=https://github.com/minio/lxmin/blob/master/README.md
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/lxmin-allan
[Service]
User=allan
Group=allan
EnvironmentFile=/etc/default/lxmin-allan
ExecStart=/usr/local/bin/lxmin-allan
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
###
Change the values are required
sudo vi /etc/default/lxmin-allan
###
## MinIO endpoint configuration
LXMIN_ENDPOINT=https://node5.lab.min.dev:19000
LXMIN_BUCKET="lxc-backup"
LXMIN_ACCESS_KEY="REDACTED"
LXMIN_SECRET_KEY="REDACTED"
LXMIN_NOTIFY_ENDPOINT="https://webhook.site/REDACTED"
## LXMIN address
LXMIN_ADDRESS=":8000"
## LXMIN server certificate and client trust certs.
LXMIN_TLS_CERT="$HOME./lxmin/certs_intel/public.crt"
LXMIN_TLS_KEY="$HOME/.lxmin/certs_intel/private.key"
###
sudo systemctl enable --now lxmin-allan.service
sudo systemctl start lxmin-allan.service
sudo systemctl status lxmin-allan.service
sudo journalctl -f -u lxmin-allan.service
The following commands are also useful
sudo systemctl disable lxmin-allan.service
sudo systemctl stop lxmin-allan.service
sudo systemctl restart lxmin-allan.service
sudo systemctl start lxmin-allan.service
cd $HOME
mkdir minio
cd $HOME/minio
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/minio-allan
mkdir -p $HOME/.minio/certs/CA
wget https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64
mv certgen-linux-amd64 certgen
chmod +x certgen
./certgen -host "127.0.0.1,localhost,1.2.3.4,1.2.3.4,1.2.3.4,1.2.3.4"
mv public.crt $HOME/.minio/certs/public.crt
mv private.key $HOME/.minio/certs/private.key
cat $HOME/.minio/certs/public.crt | openssl x509 -text -noout
sudo vi /etc/systemd/system/minio-allan.service
###
[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio-allan
[Service]
WorkingDirectory=/usr/local
User=allan
Group=allan
ProtectProc=invisible
EnvironmentFile=-/etc/default/minio-allan
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio-allan\"; exit 1; fi"
ExecStart=/usr/local/bin/minio-allan server $MINIO_OPTS $MINIO_VOLUMES
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Specifies the maximum number of threads this process can create
TasksMax=infinity
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
# Built for ${project.name}-${project.version} (${project.name})
###
Change the values are required
sudo vi /etc/default/minio-allan
###
# do paths in your home directory
MINIO_CI_CD=1
# Set the hosts and volumes MinIO uses at startup
# The command uses MinIO expansion notation {x...y} to denote a
# sequential series.
#
# The following example covers four MinIO hosts
# with 4 drives each at the specified hostname and drive locations.
# The command includes the port that each MinIO server listens on
# (default 9000)
MINIO_VOLUMES="https://node{4...7}.lab.min.dev:19000/home/allan/disk{0...1}/minio"
#MINIO_VOLUMES="https://1.2.3.{4...4}:19000/home/allan/disk{0...1}/minio"
# Set all MinIO server options
#
# The following explicitly sets the MinIO Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.
MINIO_OPTS="--address :19000 --console-address :19001 --certs-dir /home/allan/.lxmin/certs_intel"
# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.
MINIO_ROOT_USER=REDACTED
# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.
MINIO_ROOT_PASSWORD=REDACTED
# Set to the URL of the load balancer for the MinIO deployment
# This value *must* match across all MinIO servers. If you do
# not have a load balancer, set this value to to any *one* of the
# MinIO hosts in the deployment as a temporary measure.
MINIO_SERVER_URL="https://node5.lab.min.dev:19000"
###
sudo systemctl enable --now minio-allan.service
sudo systemctl start minio-allan.service
sudo systemctl status minio-allan.service
sudo journalctl -f -u minio-allan.service
https://node5.lab.min.dev:19001/login
Use mc. See https://github.com/minio/mc
mc alias set myminio https://node7.lab.min.dev MINIO_ROOT_USER MINIO_ROOT_PASSWORD
Access the following lxmin endpoint. Assumption: the wildcard certificate and key are available at $HOME/.vm-broker/ssl
curl -X GET "https://node4.lab.min.dev:8000/1.0/instances/*/backups" -H "Content-Type: application/json" --cert $HOME/.vm-broker/ssl/tls.crt --key $HOME/.vm-broker/ssl/tls.key
$ curl -X GET "https://node4.lab.min.dev:8000/1.0/instances/*/backups" -H "Content-Type: application/json" --cert $HOME/.vm-broker/ssl/tls.crt --key $HOME/.vm-broker/ssl/tls.key | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1100 100 1100 0 0 7766 0 --:--:-- --:--:-- --:--:-- 8148
{
"metadata": [
{
"instance": "delete",
"name": "backup_2023-08-08-09-3627",
"created": "2023-08-08T16:37:40.627Z",
"size": 583116000,
"optimized": false,
"compressed": false
},
{
"instance": "delete",
"name": "backup_2023-08-08-10-3807",
"created": "2023-08-08T17:39:21.241Z",
"size": 583446188,
"optimized": false,
"compressed": false
},
{
"instance": "delete",
"name": "backup_2023-08-08-15-2135",
"created": "2023-08-08T22:22:49.79Z",
"size": 583774771,
"optimized": false,
"compressed": false
},
{
"instance": "dilvm1",
"name": "backup_2023-08-08-15-2618",
"created": "2023-08-08T22:27:53.034Z",
"size": 888696535,
"optimized": false,
"compressed": false
},
{
"instance": "new0",
"name": "backup_2023-08-08-17-5231",
"created": "2023-08-09T00:53:52.235Z",
"size": 617889321,
"optimized": false,
"compressed": false
},
{
"instance": "new2",
"name": "backup_2023-08-08-15-2805",
"created": "2023-08-08T22:29:26.87Z",
"size": 617917969,
"optimized": false,
"compressed": false
},
{
"instance": "test-error",
"name": "backup_2023-08-08-16-1501",
"created": "2023-08-08T23:16:09.232Z",
"size": 516457141,
"optimized": false,
"compressed": false
}
],
"status": "Success",
"status_code": 200,
"type": "sync"
}