Veeam: How-to Deploy, and Configure MinIO with Erasure Coding Enabled, Immutability, and Let’s Encrypt

Greetings friends, just a few days ago Veeam officially announced the support for MinIO Immutability on its HCL. This is wonderful news for us to test this functionality in our labs, or in case we are using Linux storage with MinIO for production.

In this blog entry, we’re going to jump into the pool and see: how to install and configure MinIO with Erasure Coding Enabled, Immutability, Let’s Encrypt, and finally, we’ll see how to configure it in Veeam. As it has gone out of my hands, I leave you with a menu to move faster:

  1. MinIO System Requirements
    1. What is MinIO Erasure Coding?
    2. Why use MinIO Erasure Coding?
    3. Disk Configuration to use MinIO Erasure Coding
  2. How-to Deploy, and Configure MinIO, including object Immutability, and Let’s Encrypt
    1. How-to configure  Let’s Encrypt for MinIO
    2. Run MinIO with Erasure Coding
    3. Creating a new Bucket in MinIO with the Immutability attribute
  3. Veeam Backup & Replication Configuration with Immutable Storage
    1. Backup Repository Creation – Object Storage
    2. Scale-out Backup Repository Creation – with MinIO Immutable as Capacity Tier
    3. Backup Copy Job creation, pointing to this new SOBR with Immutability
    4. Trying to delete Backups from an Object Storage with Immutability Enabled

MinIO System Requirements

In this guide, we are not going to try to deploy MinIO in production, as it usually requires about 96GB of RAM, certain cores, etc. In my case I am deploying everything in 4vCPU and 16GB of RAM since I will have only one VBR sending backups, so nothing serious.

One of the most important things to be able to run Immutability in MinIO, and that it is supported by Veeam, is that we need the MinIO RELEASE.2020-07-12T19-14-17Z version or higher, and also we need the MinIO server to be running with Erasure Coding.

What is MinIO Erasure Coding?

MinIO Erasure Coding is a mathematical algorithm to reconstruct lost or corrupted data. MinIO uses Reed-Solomon code to fragment objects into variable data and parity blocks. For example, in a 12-unit configuration, an object can be split into a variable number of data and parity blocks in all units – from six data and six parity blocks to ten data and two parity blocks.

By default, MinIO divides objects into N/2 data and parity units. However, you can use storage classes to use a custom configuration. We recommend the N/2 data and parity blocks, as this ensures the best protection against drive failure.

In the 12-drive example above, with the MinIO server running at the default settings, you can lose any of the six drives and still reliably rebuild data from the remaining drives.

Why use MinIO Erasure Coding?

MinIO Erasure Coding protects data from multiple drive failures, unlike RAID or replication. For example, RAID6 can protect against the failure of two drives, while MinIO Erasure Coding can lose up to half of the drives and still keep the data safe. In addition, MinIO Erasure Coding is at the object level and can recover one object at a time. In the case of RAID, recovery can only be done at the volume level, resulting in high downtime. Since MinIO encodes each object individually, it can cure objects incrementally. Storage servers, once deployed, should not require disk replacement or healing for the life of the server. MinIO’s Erasure Coding backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.

More information can be found here – https://docs.min.io/docs/minio-erasure-code-quickstart-guide.html

Disk Configuration to use MinIO Erasure Coding

In this lab I am going to use 4 disks for my MinIO server, 100GB each, using of course THIN in VMware, so as not to consume all the space of my VSAN, the VM configuration looks like this:

Once we have all the records, we will have to do the next steps for each record, I leave you the steps with the sdb, but you will have to do it with sdc, sdd, and sde:

We’ll start by partitioning the disk:

fdisk /dev/sdb

We will have to press n to create a new one, using all the space, and of type 8e, which is LVM to be able to handle it better later in case we want to grow:

Welcome to fdisk (util-linux 2.31.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xab657906.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-209715199, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-209715199, default 209715199): 

Created a new partition 1 of type 'Linux' and of size 100 GiB.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

We will now proceed to create the LVM, with all its settings, as follows:

pvcreate /dev/sdb1
vgcreate vg_xfs_minio_1 /dev/sdb1
lvcreate -L +99G -n xfs_minio_1 vg_xfs_minio_1
mkfs.xfs /dev/vg_xfs_minio_1/xfs_minio_1

We’ll end up creating a new directory and mounting that new partition:

mkdir /root/xfs_minio_1
mount /dev/vg_xfs_minio_1/xfs_minio_1 /root/xfs_minio_1/

If we want to check that everything went well, with this command we can see everything:

df -hT /root/xfs_minio/
Filesystem                         Type  Size  Used Avail Use% Mounted on
/dev/mapper/vg_xfs_minio-xfs_minio_1 xfs    99G  134M   99G   1% /root/xfs_minio_1

Repeat this for each record, you should have something like this:

df -h
Filesystem                              Size  Used Avail Use% Mounted on
udev                                    5.9G     0  5.9G   0% /dev
tmpfs                                   1.2G  1.2M  1.2G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv        19G  4.9G   13G  28% /
tmpfs                                   5.9G     0  5.9G   0% /dev/shm
tmpfs                                   5.0M     0  5.0M   0% /run/lock
tmpfs                                   5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/sda2                               976M  145M  765M  16% /boot
tmpfs                                   1.2G     0  1.2G   0% /run/user/1000
/dev/mapper/vg_xfs_minio_1-xfs_minio_1   99G   12G   88G  12% /root/xfs_minio_1
/dev/mapper/vg_xfs_minio_2-xfs_minio_2   99G   12G   88G  12% /root/xfs_minio_2
/dev/mapper/vg_xfs_minio_3-xfs_minio_3   99G   12G   88G  12% /root/xfs_minio_3
/dev/mapper/vg_xfs_minio_4-xfs_minio_4   99G   12G   88G  12% /root/xfs_minio_4

Don’t forget to add them to the fstab, to do this remove the UID with blkid /dev/mapper/vg_xfs_minio_3-xfs_minio_3, in my case for example:

echo 'UUID=155b62fb-e7cb-424f-91d7-3e09217d7e7f /root/xfs_minio_1 xfs defaults 1 1' >> /etc/fstab
echo '85f1550b-5459-4427-8a45-8fe139cfaebe /root/xfs_minio_2 xfs defaults 1 1' >> /etc/fstab
echo '0c47b7cd-edc3-4ef9-9b6a-0d3f2dc476db /root/xfs_minio_3 xfs defaults 1 1' >> /etc/fstab
echo '2833348b-7053-49f1-a96d-d00ee8f071ed /root/xfs_minio_4 xfs defaults 1 1' >> /etc/fstab

We are ready, so let’s move on to the next step.

How-to Deploy, and Configure MinIO, including object Immutability, and Let’s Encrypt

Well, well, we’ve come to this part, so let’s not delay any longer, let’s go there, we’ll install the latest version of MinIO using the following command:

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio

Once we have everything downloaded and ready, we could launch a simple instance of MinIO, to see what works, without any Erasure Coding, or anything, just for testing, like this:

./minio server /home/oper/test
Endpoint:  http://192.168.1.185:9000  http://127.0.0.1:9000    
AccessKey: minioadmin 
SecretKey: minioadmin 

Browser Access:
   http://192.168.1.185:9000  http://127.0.0.1:9000    

Command-line Access: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.1.185:9000 minioadmin minioadmin

Object API (Amazon S3 compatible):
   Go:         https://docs.min.io/docs/golang-client-quickstart-guide
   Java:       https://docs.min.io/docs/java-client-quickstart-guide
   Python:     https://docs.min.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
Detected default credentials 'minioadmin:minioadmin', please change the credentials immediately using 'MINIO_ACCESS_KEY' and 'MINIO_SECRET_KEY'

If I access the IP and the port, I see that we can enter a MinIO interface and everything works fine:

Once I enter the credentials, I will make sure I have a modern version of MinIO, especially superior to the one recommended by Veeam in their HCL: Okay, since everything works fine for me, let’s take the step and set everything up more securely and elegantly using Let’s Encrypt.

How-to Configure Let’s Encrypt for MinIO

As always, we will make use of certbot, which will help us generate the whole process, and validate our Let’s Encrypt certificates, update the system and install certbot:

sudo apt-get install software-properties-common
sudo add-apt-repository universe
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot

There are several ways to validate our domain, I find the manual DNS mode, in case we don’t have Cloudflare, etc, the easiest one, so I will do it this way:

certbot --manual --preferred-challenges dns certonly -d minio.jorgedelacruz.es --staple-ocsp -m [email protected] --agree-tos

This will launch the next output, which we will have to configure in our public DNS:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for minio.jorgedelacruz.es

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: yes

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name
_acme-challenge.minio.jorgedelacruz.es with the following value:

OIOLPq0SBAH41d5hKAEL08OueThzMyboRGduxdi5L3c

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Once we put this .TXT entry in our public DNS, we click on Enter and if everything has gone well we’ll see the following:

Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/minio.jorgedelacruz.es/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/minio.jorgedelacruz.es/privkey.pem
   Your cert will expire on 2020-10-19. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

If we want to check the files, etc, we will do so:

ls -l /etc/letsencrypt/live/minio.jorgedelacruz.es/
total 4
lrwxrwxrwx 1 root root  46 Jul 21 15:43 cert.pem -> ../../archive/minio.jorgedelacruz.es/cert1.pem
lrwxrwxrwx 1 root root  47 Jul 21 15:43 chain.pem -> ../../archive/minio.jorgedelacruz.es/chain1.pem
lrwxrwxrwx 1 root root  51 Jul 21 15:43 fullchain.pem -> ../../archive/minio.jorgedelacruz.es/fullchain1.pem
lrwxrwxrwx 1 root root  49 Jul 21 15:43 privkey.pem -> ../../archive/minio.jorgedelacruz.es/privkey1.pem
-rw-r--r-- 1 root root 692 Jul 21 15:43 README

We are going to copy these files to the MinIO folder of the user from whom we want to run MinIO, in my case, it is root, so as not to complicate it, but in your case to secure it a little more:

cp /etc/letsencrypt/live/minio.jorgedelacruz.es/fullchain.pem /root/.minio/certs/public.crt
cp /etc/letsencrypt/live/minio.jorgedelacruz.es/privkey.pem /root/.minio/certs/private.key

Run MinIO with Erasure Coding

There is not much mystery, since we have the SSL certificate, and we also have our partitions and mount points ready, we will have everything ready to launch MinIO with Erasure Coding, which comes with high availability, etc. Of course, it’s all on a server, but it’s for testing:

./minio server --address ":443" /root/xfs_minio_1/ /root/xfs_minio_2/ /root/xfs_minio_3/ /root/xfs_minio_4/
Formatting 1st zone, 1 set(s), 4 drives per set.
WARNING: Host local has more than 2 drives of set. A host failure will result in data becoming unavailable.
Status:         4 Online, 0 Offline. 
Endpoint:  https://192.168.1.185  https://127.0.0.1    
AccessKey: minioadmin 
SecretKey: minioadmin 

Browser Access:
   https://192.168.1.185  https://127.0.0.1    

Command-line Access: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio https://192.168.1.185 minioadmin minioadmin

Object API (Amazon S3 compatible):
   Go:         https://docs.min.io/docs/golang-client-quickstart-guide
   Java:       https://docs.min.io/docs/java-client-quickstart-guide
   Python:     https://docs.min.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide

Certificate:
    Signature Algorithm: SHA256-RSA
    Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
    Validity
        Not Before: Tue, 21 Jul 2020 14:43:33 GMT
        Not After : Mon, 19 Oct 2020 14:43:33 GMT

Certificate:
    Signature Algorithm: SHA256-RSA
    Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
    Validity
        Not Before: Thu, 17 Mar 2016 16:40:46 GMT
        Not After : Wed, 17 Mar 2021 16:40:46 GMT

Detected default credentials 'minioadmin:minioadmin', please change the credentials immediately using 'MINIO_ACCESS_KEY' and 'MINIO_SECRET_KEY'

It would be nice if you change the username and password to have more security as well. Besides, as I said, MinIO tells us that we have everything on the same server, so it’s not really resistant to physical failures, but it’s ok. If we go via HTTPS, we’ll see good news:

Creating a new Bucket in MinIO with the Immutability attribute

This command will fail you if you are not using a modern version of MinIO, and if you are not using Erasure Coding, we need minio-client, so we will download it quickly:

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc

We’ll set it up quickly by adding a new host, like this:

./mc config host add minio-veeam https://minio.jorgedelacruz.es YOURS3ACCESSKEY YOURSECERTKEY --api S3v4

Now we can launch the following command, which has the -l (–with-lock) attribute

./mc mb --debug -l minio-veeam/veeam-immutable

And if everything went well, we’ll see what happens next:

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Tue, 21 Jul 2020 17:53:15 GMT
Location: /veeamimmutable
Server: MinIO/RELEASE.2020-07-20T02-25-16Z
Vary: Origin
X-Amz-Request-Id: 1623D601B41CBFAE
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  99.203176ms

Bucket created successfully `minio-veeam/veeamimmutable`.

And of course, by HTTPS, we’ll see the following in the client:

Congratulations! We have everything ready, our particular MinIO, ready, with HTTPS, and using immutable Storage. Let’s go to the last step, the configuration of Veeam Backup & Replication.

Veeam Backup & Replication Configuration with Immutable Storage

Backup Repository Creation – Object Storage

We have everything ready, we will have to be using Veeam Backup & Replication v10 to be able to use MinIO with immutability, we will go to the Backup Infrastructure – Repositories part, and we will create a new one:

We will select Object Storage type: We will select the type S3-Compatible, and once inside we will introduce a name and description for this new Object Storage: We will select the server, which we have with HTTPS and a valid FQDN and the credentials we have:

We’ll select the bucket, which in my case is the one we’ve previously created as immutable, as well as entering a folder where we want to save the copies, and the number of days we want to make our immutable backups:

Finally, if we are happy with everything, we will click on Finish: We are now ready to move to the next step.

Scale-out Backup Repository Creation – with MinIO Immutable as Capacity Tier

We’ll go to Scale-out Backup Repositories, and create a new one: We will enter a name and description that we want:

As Performance Tier, we will select a local disk where we already have copies, or where we want to launch copies, be it daily, or the GFS, etc: In the part of Capacity Tier, we will select the new Object Storage to MinIO that already had Immutable configured, and we will select what we want to send, in my case I have selected the copies, besides moving the backups that are complete also, as they are weekly, monthly, etc:

Backup Copy Job creation, pointing to this new SOBR with Immutability

We will create a new Backup Copy job, I prefer pruning as it gives us more reports and visibility: We will select the VM, or the job or jobs that we want to make a copy to this Object Storage repository with Immutability: We will select the Backup Repository, as well as the restore points to be saved: The Backup Copy job will start and continue to be launched every day to copy the new restore points from one location to another:

Trying to delete Backups from an Object Storage with Immutability Enabled

If we go to our Object Storage in Veeam, and we want to remove restore points that are in this Object Storage, this is the error that we will get: That’s all friends, I hope you liked this blog post so detailed and long, more than 2500 words, I hope it serves you.

Author: jorgeuk

Father, writing in https://www.jorgedelacruz.es and https://jorgedelacruz.uk Blogger, Systems Engineer @veeam - vExpert 2014/2020 & NTC 2018/19

9 Thoughts

  1. I have exactly the same problem (not implemented with object locking enabled) on version 2020-08-10T22:04:32Z

  2. Hello,
    I’ve helped Bill by email, and what it happened was he didn’t followed all the steps, meaning, start MinIO with multiple disks, etc. You NEED to start minio that way in order to enable immutability, as that enables erasure coding.

  3. Super, thanks. Would it be possible to show the same without a domain to create a certificate for? I have tried for hours to create a self signed certificate but to no avail.
    Thank you.

  4. Hi Jorge,

    Very nice article, can Minio run automatically without using the manual command line method ?

    thanks

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.