Skip to content

minio healing tests

Jump to bottom
Allan Roger Reid edited this page Dec 4, 2024 · 2 revisions

Summary

Test 1 - Object healed Test 2 - Object deleted. minio dirs cleaned up of all data related to object Test 3 - Object new version is the only version available, in both mc and on minio dirs Test 4 - Object old version is only one visible. Minio data directory is not cleaned up

On ansible

ssh -p 20056 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Run:

cd ~/ansible_scripts && git checkout pool-expansion
cd ~ && ANSIBLE_TIMEOUT=90 HOST_KEY_CHECKING=false ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook -i /home/ubuntu/pool-expansion.yml -l webservers /home/ubuntu/ansible-scripts/ansible/create.yml

Connect to client

ssh -p 20003 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
loginctl enable-linger ubuntu

mkdir -p ~/mc && cd ~/mc && rm -rf mc* && wget http://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc && cd ~
~/mc/mc alias set pool-expansion https://pool-expansion-0-0.lab.min.dev:9000 minioadmin minioadmin
~/mc/mc admin info pool-expansion

Scenario 1

xl meta is fine but some parts (2?) are missing Run heal and make sure heal works

Setup objects

rm -rf ~/objects
mkdir ~/objects
~/mc/mc rb --force pool-expansion/versioned
~/mc/mc mb pool-expansion/versioned
~/mc/mc version enable pool-expansion/versioned
cd ~/objects
# dd if=/dev/urandom bs=100M count=1 iflag=fullblock of=object_a

wget https://sample-videos.com/img/Sample-jpg-image-30mb.jpg -O image.jpg
~/mc/mc cp --recursive ~/objects pool-expansion/versioned
~/mc/mc ls pool-expansion/versioned/objects

~/mc/mc admin config set pool-expansion scanner speed=slowest
~/mc/mc admin config get pool-expansion scanner
On 2 nodes (pool-expansion-0-0.lab.min.dev, pool-expansion-0-1.lab.min.dev)
ssh -p 20607 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20709 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Run

mkdir -p /tmp/minio-backup/data/versioned/objects/image.jpg/519d5441-8deb-4f0e-a95c-b4344e3c9fd8/ && \
mv /tmp/data/versioned/objects/image.jpg/519d5441-8deb-4f0e-a95c-b4344e3c9fd8/part.1 /tmp/minio-backup/data/versioned/objects/image.jpg/519d5441-8deb-4f0e-a95c-b4344e3c9fd8/
On client
~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects

ubuntu@pool-expansion-client:~/objects$ ~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects
 ◓  versioned/objects/object_a
    1/2 objects; 129 MiB in 1s
    ┌────────┬───┬─────────────────────┐
    │ Green  │ 5 │  71.4% █████████    │
    │ Yellow │ 1 │  14.3% ██           │
    │ Red    │ 0 │   0.0%              │
    │ Grey   │ 0 │   0.0%              │
    └────────┴───┴─────────────────────┘

Test 1 - Object healed

ubuntu@pool-expansion-0-2:~$ ls -la /tmp/data/versioned/objects/image.jpg/519d5441-8deb-4f0e-a95c-b4344e3c9fd8/
total 6851
drwxrwxr-x 2 ubuntu ubuntu       3 Feb 14 22:29 .
drwxrwxr-x 3 ubuntu ubuntu       4 Feb 14 22:29 ..
-rw-rw-r-- 1 ubuntu ubuntu 7006634 Feb 14 22:29 part.2
ubuntu@pool-expansion-0-2:~$ ls -la /tmp/data/versioned/objects/image.jpg/519d5441-8deb-4f0e-a95c-b4344e3c9fd8/
total 15052
drwxrwxr-x 2 ubuntu ubuntu       4 Feb 14 22:35 .
drwxrwxr-x 3 ubuntu ubuntu       4 Feb 14 22:35 ..
-rw-rw-r-- 1 ubuntu ubuntu 8389120 Feb 14 22:35 part.1
-rw-rw-r-- 1 ubuntu ubuntu 7006634 Feb 14 22:35 part.2

Scenario 2

xl meta is fine but all parts are missing ( harsha says it will delete ) run heal and make sure xl meta is not deleted ( if it's deleted then we do not want to run heal )

Setup objects

rm -rf ~/objects
mkdir ~/objects
~/mc/mc rb --force pool-expansion/versioned
~/mc/mc mb pool-expansion/versioned
~/mc/mc version enable pool-expansion/versioned
cd ~/objects
# dd if=/dev/urandom bs=100M count=1 iflag=fullblock of=object_a

wget https://sample-videos.com/img/Sample-jpg-image-30mb.jpg -O image.jpg
~/mc/mc cp --recursive ~/objects pool-expansion/versioned
~/mc/mc ls pool-expansion/versioned/objects

~/mc/mc admin config set pool-expansion scanner speed=slowest
~/mc/mc admin config get pool-expansion scanner
On ALL nodes
ssh -p 20607 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20158 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20709 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20692 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Run

mkdir -p /tmp/minio-backup/data/versioned/objects/image.jpg/8fbf138a-3e55-4a8c-8a84-ee2a9f246f29/ && \
mv /tmp/data/versioned/objects/image.jpg/8fbf138a-3e55-4a8c-8a84-ee2a9f246f29/part.* /tmp/minio-backup/data/versioned/objects/image.jpg/8fbf138a-3e55-4a8c-8a84-ee2a9f246f29/
On client
~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects

ubuntu@pool-expansion-client:~/objects$ ~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects
 ◓  /
    0/1 objects; 0 B in 1s
    ┌────────┬───┬─────────────────────┐
    │ Green  │ 4 │  66.7% ████████     │
    │ Yellow │ 0 │   0.0%              │
    │ Red    │ 0 │   0.0%              │
    │ Grey   │ 0 │   0.0%              │
    └────────┴───┴─────────────────────┘

Test 2 - Object deleted. minio dirs cleaned up of all data related to object

ubuntu@pool-expansion-0-2:~$ ls -laR /tmp/data/versioned/objects/image.jpg/
ls: cannot access '/tmp/data/versioned/objects/image.jpg/': No such file or directory

Scenario 3

xl meta is fine but all parts are missing upload new version run mc ls --versions (make sure old version is still listed) run mc admin heal make sure all versions are still there

Setup objects

rm -rf ~/objects
mkdir ~/objects
~/mc/mc rb --force pool-expansion/versioned
~/mc/mc mb pool-expansion/versioned
~/mc/mc version enable pool-expansion/versioned
cd ~/objects
# dd if=/dev/urandom bs=100M count=1 iflag=fullblock of=object_a

wget https://sample-videos.com/img/Sample-jpg-image-30mb.jpg -O image.jpg
~/mc/mc cp --recursive ~/objects pool-expansion/versioned
~/mc/mc ls pool-expansion/versioned/objects

~/mc/mc admin config set pool-expansion scanner speed=slowest
~/mc/mc admin config get pool-expansion scanner
On ALL nodes
ssh -p 20607 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20158 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20709 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20692 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Run

mkdir -p /tmp/minio-backup/data/versioned/objects/image.jpg/9031d76b-4e34-4fb1-908c-259061b04da8/ && \
mv /tmp/data/versioned/objects/image.jpg/9031d76b-4e34-4fb1-908c-259061b04da8/part.* /tmp/minio-backup/data/versioned/objects/image.jpg/9031d76b-4e34-4fb1-908c-259061b04da8/
On client
~/mc/mc cp --recursive ~/objects pool-expansion/versioned
~/mc/mc ls --versions pool-expansion/versioned/objects

[2024-02-15 00:32:49 UTC]  29MiB STANDARD 52064c53-6eb8-43f0-aa58-967f3e64dcb6 v2 PUT image.jpg
[2024-02-15 00:31:59 UTC]  29MiB STANDARD bdd01d40-548a-4b3e-9bcc-af55fea73f69 v1 PUT image.jpg

~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects

ubuntu@pool-expansion-client:~/objects$ ~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects
 ◓  /
    0/2 objects; 29 MiB in 1s
    ┌────────┬───┬─────────────────────┐
    │ Green  │ 5 │  71.4% █████████    │
    │ Yellow │ 0 │   0.0%              │
    │ Red    │ 0 │   0.0%              │
    │ Grey   │ 0 │   0.0%              │
    └────────┴───┴─────────────────────┘

Test 3 - Object new version is the only version available, in both mc and on minio dirs

ubuntu@pool-expansion-0-3:~$ ls -laR /tmp/data/versioned/objects/image.jpg/
/tmp/data/versioned/objects/image.jpg/:
total 3
drwxrwxr-x 3 ubuntu ubuntu   4 Feb 15 00:33 .
drwxrwxr-x 3 ubuntu ubuntu   3 Feb 15 00:31 ..
drwxrwxr-x 2 ubuntu ubuntu   4 Feb 15 00:32 a1b9d0ce-15e1-4875-b375-2c096bfc5008
-rw-rw-r-- 1 ubuntu ubuntu 404 Feb 15 00:33 xl.meta

/tmp/data/versioned/objects/image.jpg/a1b9d0ce-15e1-4875-b375-2c096bfc5008:
total 15052
drwxrwxr-x 2 ubuntu ubuntu       4 Feb 15 00:32 .
drwxrwxr-x 3 ubuntu ubuntu       4 Feb 15 00:33 ..
-rw-rw-r-- 1 ubuntu ubuntu 8389120 Feb 15 00:32 part.1
-rw-rw-r-- 1 ubuntu ubuntu 7006634 Feb 15 00:32 part.2

~/mc/mc ls --versions pool-expansion/versioned/objects

[2024-02-15 00:32:49 UTC]  29MiB STANDARD 52064c53-6eb8-43f0-aa58-967f3e64dcb6 v1 PUT image.jpg

Scenario 4

Over writing new meta with old: upload object (with versioning ) backup xl meta upload new version restore old xl meta ( overwriting the new) run heal and see what happens

Setup objects

rm -rf ~/objects
mkdir ~/objects
~/mc/mc rb --force pool-expansion/versioned
~/mc/mc mb pool-expansion/versioned
~/mc/mc version enable pool-expansion/versioned
cd ~/objects
# dd if=/dev/urandom bs=100M count=1 iflag=fullblock of=object_a

wget https://sample-videos.com/img/Sample-jpg-image-30mb.jpg -O image1.jpg
~/mc/mc cp --recursive ~/objects pool-expansion/versioned
~/mc/mc ls --versions pool-expansion/versioned/objects

[2024-02-15 00:19:37 UTC]  29MiB STANDARD c51f6a0e-21b0-45a8-9aa0-a4c78efa98a6 v1 PUT image1.jpg

~/mc/mc admin config set pool-expansion scanner speed=slowest
~/mc/mc admin config get pool-expansion scanner
On ALL nodes
ssh -p 20607 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20158 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20709 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
ssh -p 20692 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Run

mkdir -p /tmp/minio-backup/data/versioned/objects/image1.jpg && \
mv /tmp/data/versioned/objects/image1.jpg/xl.meta /tmp/minio-backup/data/versioned/objects/image1.jpg/xl.meta

ls -la /tmp/data/versioned/objects/image1.jpg/

total 2
drwxrwxr-x 3 ubuntu ubuntu 3 Feb 15 00:20 .
drwxrwxr-x 3 ubuntu ubuntu 3 Feb 15 00:19 ..
drwxrwxr-x 2 ubuntu ubuntu 4 Feb 15 00:19 42ec5f3e-aa1d-4f3c-8fc9-5ba6b6fa5046

Run on client

~/mc/mc cp --recursive ~/objects pool-expansion/versioned

Test on each node

ls -la /tmp/data/versioned/objects/image1.jpg/

total 4
drwxrwxr-x 4 ubuntu ubuntu   5 Feb 15 00:21 .
drwxrwxr-x 3 ubuntu ubuntu   3 Feb 15 00:19 ..
drwxrwxr-x 2 ubuntu ubuntu   4 Feb 15 00:21 183f8254-d71b-4435-aa23-965a0112464c
drwxrwxr-x 2 ubuntu ubuntu   4 Feb 15 00:19 42ec5f3e-aa1d-4f3c-8fc9-5ba6b6fa5046
-rw-rw-r-- 1 ubuntu ubuntu 404 Feb 15 00:21 xl.meta

cp /tmp/minio-backup/data/versioned/objects/image1.jpg/xl.meta /tmp/data/versioned/objects/image1.jpg/xl.meta

ls -la /tmp/data/versioned/objects/image1.jpg/

total 4
drwxrwxr-x 4 ubuntu ubuntu   5 Feb 15 00:21 .
drwxrwxr-x 3 ubuntu ubuntu   3 Feb 15 00:19 ..
drwxrwxr-x 2 ubuntu ubuntu   4 Feb 15 00:21 183f8254-d71b-4435-aa23-965a0112464c
drwxrwxr-x 2 ubuntu ubuntu   4 Feb 15 00:19 42ec5f3e-aa1d-4f3c-8fc9-5ba6b6fa5046
-rw-rw-r-- 1 ubuntu ubuntu 404 Feb 15 00:22 xl.meta
On client
~/mc/mc ls --versions pool-expansion/versioned/objects

[2024-02-15 00:19:37 UTC]  29MiB STANDARD c51f6a0e-21b0-45a8-9aa0-a4c78efa98a6 v1 PUT image1.jpg

~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects

~/mc/mc ls --versions pool-expansion/versioned/objects

[2024-02-15 00:19:37 UTC]  29MiB STANDARD c51f6a0e-21b0-45a8-9aa0-a4c78efa98a6 v1 PUT image1.jpg

ubuntu@pool-expansion-client:~$ ~/mc/mc admin heal -r --scan=deep pool-expansion/versioned/objects
 ◓  versioned/objects/object_a
    0/2 objects; 129 MiB in 1s
    ┌────────┬───┬─────────────────────┐
    │ Green  │ 6 │  85.7% ███████████  │
    │ Yellow │ 0 │   0.0%              │
    │ Red    │ 0 │   0.0%              │
    │ Grey   │ 0 │   0.0%              │
    └────────┴───┴─────────────────────┘

Test 4 - Object old version is only one visible. Minio data directory is not cleaned up

Clone this wiki locally