Veeam: How-to Deploy, and Configure MinIO with Erasure Coding Enabled, Immutability, and Let’s Encrypt

Greetings friends, just a few days ago Veeam officially announced the support for MinIO Immutability on its HCL. This is wonderful news for us to test this functionality in our labs, or in case we are using Linux storage with MinIO for production.

In this blog entry, we’re going to jump into the pool and see: how to install and configure MinIO with Erasure Coding Enabled, Immutability, Let’s Encrypt, and finally, we’ll see how to configure it in Veeam. As it has gone out of my hands, I leave you with a menu to move faster:

  1. MinIO System Requirements
    1. What is MinIO Erasure Coding?
    2. Why use MinIO Erasure Coding?
    3. Disk Configuration to use MinIO Erasure Coding
  2. How-to Deploy, and Configure MinIO, including object Immutability, and Let’s Encrypt
    1. How-to configure  Let’s Encrypt for MinIO
    2. Run MinIO with Erasure Coding
    3. Creating a new Bucket in MinIO with the Immutability attribute
  3. Veeam Backup & Replication Configuration with Immutable Storage
    1. Backup Repository Creation – Object Storage
    2. Scale-out Backup Repository Creation – with MinIO Immutable as Capacity Tier
    3. Backup Copy Job creation, pointing to this new SOBR with Immutability
    4. Trying to delete Backups from an Object Storage with Immutability Enabled

MinIO System Requirements

In this guide, we are not going to try to deploy MinIO in production, as it usually requires about 96GB of RAM, certain cores, etc. In my case I am deploying everything in 4vCPU and 16GB of RAM since I will have only one VBR sending backups, so nothing serious.

One of the most important things to be able to run Immutability in MinIO, and that it is supported by Veeam, is that we need the MinIO RELEASE.2020-07-12T19-14-17Z version or higher, and also we need the MinIO server to be running with Erasure Coding.

What is MinIO Erasure Coding?

MinIO Erasure Coding is a mathematical algorithm to reconstruct lost or corrupted data. MinIO uses Reed-Solomon code to fragment objects into variable data and parity blocks. For example, in a 12-unit configuration, an object can be split into a variable number of data and parity blocks in all units – from six data and six parity blocks to ten data and two parity blocks.

By default, MinIO divides objects into N/2 data and parity units. However, you can use storage classes to use a custom configuration. We recommend the N/2 data and parity blocks, as this ensures the best protection against drive failure.

In the 12-drive example above, with the MinIO server running at the default settings, you can lose any of the six drives and still reliably rebuild data from the remaining drives.

Why use MinIO Erasure Coding?

MinIO Erasure Coding protects data from multiple drive failures, unlike RAID or replication. For example, RAID6 can protect against the failure of two drives, while MinIO Erasure Coding can lose up to half of the drives and still keep the data safe. In addition, MinIO Erasure Coding is at the object level and can recover one object at a time. In the case of RAID, recovery can only be done at the volume level, resulting in high downtime. Since MinIO encodes each object individually, it can cure objects incrementally. Storage servers, once deployed, should not require disk replacement or healing for the life of the server. MinIO’s Erasure Coding backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.

More information can be found here –

Disk Configuration to use MinIO Erasure Coding

In this lab I am going to use 4 disks for my MinIO server, 100GB each, using of course THIN in VMware, so as not to consume all the space of my VSAN, the VM configuration looks like this:

Once we have all the records, we will have to do the next steps for each record, I leave you the steps with the sdb, but you will have to do it with sdc, sdd, and sde:

We’ll start by partitioning the disk:

We will have to press n to create a new one, using all the space, and of type 8e, which is LVM to be able to handle it better later in case we want to grow:

We will now proceed to create the LVM, with all its settings, as follows:

We’ll end up creating a new directory and mounting that new partition:

If we want to check that everything went well, with this command we can see everything:

Repeat this for each record, you should have something like this:

Don’t forget to add them to the fstab, to do this remove the UID with blkid /dev/mapper/vg_xfs_minio_3-xfs_minio_3, in my case for example:

We are ready, so let’s move on to the next step.

How-to Deploy, and Configure MinIO, including object Immutability, and Let’s Encrypt

Well, well, we’ve come to this part, so let’s not delay any longer, let’s go there, we’ll install the latest version of MinIO using the following command:

Once we have everything downloaded and ready, we could launch a simple instance of MinIO, to see what works, without any Erasure Coding, or anything, just for testing, like this:

If I access the IP and the port, I see that we can enter a MinIO interface and everything works fine:

Once I enter the credentials, I will make sure I have a modern version of MinIO, especially superior to the one recommended by Veeam in their HCL: Okay, since everything works fine for me, let’s take the step and set everything up more securely and elegantly using Let’s Encrypt.

How-to Configure Let’s Encrypt for MinIO

As always, we will make use of certbot, which will help us generate the whole process, and validate our Let’s Encrypt certificates, update the system and install certbot:

There are several ways to validate our domain, I find the manual DNS mode, in case we don’t have Cloudflare, etc, the easiest one, so I will do it this way:

This will launch the next output, which we will have to configure in our public DNS:

Once we put this .TXT entry in our public DNS, we click on Enter and if everything has gone well we’ll see the following:

If we want to check the files, etc, we will do so:

We are going to copy these files to the MinIO folder of the user from whom we want to run MinIO, in my case, it is root, so as not to complicate it, but in your case to secure it a little more:

Run MinIO with Erasure Coding

There is not much mystery, since we have the SSL certificate, and we also have our partitions and mount points ready, we will have everything ready to launch MinIO with Erasure Coding, which comes with high availability, etc. Of course, it’s all on a server, but it’s for testing:

It would be nice if you change the username and password to have more security as well. Besides, as I said, MinIO tells us that we have everything on the same server, so it’s not really resistant to physical failures, but it’s ok. If we go via HTTPS, we’ll see good news:

Creating a new Bucket in MinIO with the Immutability attribute

This command will fail you if you are not using a modern version of MinIO, and if you are not using Erasure Coding, we need minio-client, so we will download it quickly:

We’ll set it up quickly by adding a new host, like this:

Now we can launch the following command, which has the -l (–with-lock) attribute

And if everything went well, we’ll see what happens next:

And of course, by HTTPS, we’ll see the following in the client:

Congratulations! We have everything ready, our particular MinIO, ready, with HTTPS, and using immutable Storage. Let’s go to the last step, the configuration of Veeam Backup & Replication.

Veeam Backup & Replication Configuration with Immutable Storage

Backup Repository Creation – Object Storage

We have everything ready, we will have to be using Veeam Backup & Replication v10 to be able to use MinIO with immutability, we will go to the Backup Infrastructure – Repositories part, and we will create a new one:

We will select Object Storage type: We will select the type S3-Compatible, and once inside we will introduce a name and description for this new Object Storage: We will select the server, which we have with HTTPS and a valid FQDN and the credentials we have:

We’ll select the bucket, which in my case is the one we’ve previously created as immutable, as well as entering a folder where we want to save the copies, and the number of days we want to make our immutable backups:

Finally, if we are happy with everything, we will click on Finish: We are now ready to move to the next step.

Scale-out Backup Repository Creation – with MinIO Immutable as Capacity Tier

We’ll go to Scale-out Backup Repositories, and create a new one: We will enter a name and description that we want:

As Performance Tier, we will select a local disk where we already have copies, or where we want to launch copies, be it daily, or the GFS, etc: In the part of Capacity Tier, we will select the new Object Storage to MinIO that already had Immutable configured, and we will select what we want to send, in my case I have selected the copies, besides moving the backups that are complete also, as they are weekly, monthly, etc:

Backup Copy Job creation, pointing to this new SOBR with Immutability

We will create a new Backup Copy job, I prefer pruning as it gives us more reports and visibility: We will select the VM, or the job or jobs that we want to make a copy to this Object Storage repository with Immutability: We will select the Backup Repository, as well as the restore points to be saved: The Backup Copy job will start and continue to be launched every day to copy the new restore points from one location to another:

Trying to delete Backups from an Object Storage with Immutability Enabled

If we go to our Object Storage in Veeam, and we want to remove restore points that are in this Object Storage, this is the error that we will get: That’s all friends, I hope you liked this blog post so detailed and long, more than 2500 words, I hope it serves you.

Author: jorgeuk

Father, writing in and Blogger, Systems Engineer @veeam - vExpert 2014/2020 & NTC 2018/19

9 Thoughts

  1. I have exactly the same problem (not implemented with object locking enabled) on version 2020-08-10T22:04:32Z

  2. Hello,
    I’ve helped Bill by email, and what it happened was he didn’t followed all the steps, meaning, start MinIO with multiple disks, etc. You NEED to start minio that way in order to enable immutability, as that enables erasure coding.

  3. Super, thanks. Would it be possible to show the same without a domain to create a certificate for? I have tried for hours to create a self signed certificate but to no avail.
    Thank you.

  4. Hi Jorge,

    Very nice article, can Minio run automatically without using the manual command line method ?


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.