MinIO is a great solution when you need to organize object storage easily and simply. Elementary setup, many platforms and good performance have done their job in the field of people's love. So we had no other way than to declare compatibility a month ago . Including such an important feature as Immutability. In fact, MinIO has a whole in the documentation on our integration.
Therefore, today we will talk about how:
- Setting up MinIO is very fast.
- Setting up MinIO is a little less fast, but much better.
- Use it as an Archive Tier for the Veeam SOBR scalable repository.
What are you?
A brief introduction for those who have not encountered MinIO. It is an open source object storage compatible with the Amazon S3 API. It is released under the Apache v2 license and adheres to the philosophy of spartan minimalism.
That is, it does not have a sprawling GUI with dashboards, charts and numerous menus. MinIO simply launches its server with one command, on which you can simply store data using the full power of the S3 API. But it should be noted that this simplicity can be deceptive when it comes to the resources used. RAM and CPU are absorbed perfectly, but the reasons will be discussed below. And, by the way, such combines as FreeNAS and TrueNAS use MinIO under the hood.
This introduction can be completed.
Setting up MinIO is very fast
Setting it up is so fast that we'll cover it for Windows and Linux. There are options for docker, and for kubernetis, and even for makosi, but the meaning will be the same everywhere.
So, in the case of Windows, we go to the official website and download the latest version. We also see the instructions for launching:
minio.exe server F:Data
And there is a link to a slightly more detailed . There is no point in not believing instructions, so we run it and get something like this answer.
That's all! The repository is working and you can start working with it. I wasn't joking when I said that MinIO is minimalism and just works. If you follow the link offered at startup, then the maximum of the functions available there is to create a bucket. And you can start writing data.
For Linux lovers, everything remains no less simple. The simplest instruction:
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /dataThe result will be indistinguishable from what has been seen before.
The MinIO setup is a little more meaningful
As we understand, the previous paragraph is a prank for test purposes. And, to be honest, it is for testing MinIO that we use it very widely, which we are not at all ashamed to admit. Of course, it works, but it's embarrassing to take it beyond the test benches. Therefore, we pick up a file and begin to bring to mind.
HTTPS
The first mandatory step on the way to production is encryption. There are already a million and a thousand manuals on the network for adding certificates to MiniIO, but their general plan is as follows:
- Create a certificate
- In the case of Windows, put it in C: Users% User%. miniocerts
- In case of Linux in ${HOME}/.minio/certs
- Restarting the server
Banal Let's Encrypt is boring and described everywhere, so our path is the path of a samurai, so in the case of Windows, download , and in the case of Linux, we just check that we have openssl installed. And do some console magic:
- Create keys: openssl ecparam -genkey -name prime256v1 | openssl ec -out private.key
- Create a certificate by key: openssl req -new -x509 -days 3650 -key private.key -out public.crt
- Copy private.key and public.crt to the folder above
- Restart MinIO
If everything went as it should, then approximately the following lines will appear in the status.
Enable MinIO Erasure Coding
First, a couple of words about the subject. In a nutshell: this is software data protection from damage and loss. Like a raid, only much more reliable. If classic RAID6 can afford to lose two disks, then MinIO calmly survives the loss of half. The technology is described in more detail in . But if you take the essence, then this is the implementation of Reed-Solomon codes: all information is stored in the form of data blocks, to which there are parity blocks. And it seems that all this has already been done many times, only there is an important βbutβ: we can explicitly indicate the ratio of parity blocks to data blocks for stored objects.
Do you want 1:1? Please!
Do you want 5:2? No problem!
A very important feature if you are running multiple nodes at once and want to find your own balance between maximum data security and wasted resources. Out of the box, MinIO uses the N/2 formula (where N is the total number of disks), i.e. divides your data between N/2 data disks and N/2 parity disks. Translating into human: you can lose half of the disks and recover the data. This ratio is given by , allowing you to choose what is more important: reliability or capacity.
The guide gives an example: let's say you have a 16-disk installation and you need to save a 100 MB file. If the default settings are used (8 disks for data, 8 for parity blocks), then the file will eventually occupy almost double the volume, i.e. 200 MB. If the ratio of disks is 10/6, then 160 MB will be needed. 14/2 - 114 MB.
Another important difference from raids is that in the event of disk failure, MinIO will work at the object level, restoring one by one without stopping the entire system. Whereas a regular raid will be forced to regenerate the entire volume, which will take an unpredictable amount of time. The author remembers a disk shelf, which, after the loss of two disks, went for recalculation for a week and a half. It was very unpleasant.
And, an important note: MinIO divides all disks for Erasure Coding into sets from 4 to 16 disks, using the maximum possible set size. And in the future, one element of information will be stored only within one set.
This all sounds great, but how difficult will it be to set up? Let's get a look. We take the command to run and simply list the disks on which we need to create the storage. If everything is done correctly, then in the report we will see the number of disks involved. And the advice is that it is not worth adding half of the disks to one host at once, because this leads to data loss.
c:minio>minio.exe server F: G: H: I: J: K:
Next, to manage and configure the MinIO server, we need an agent, which you can download everything from the official site.
In order not to erase your fingers, each time typing the address and access keys (and this is not safe), it is convenient to immediately create an alias at the first start using the formula mc alias set [YOUR-ACCESS-KEY] [YOUR-SECRET-KEY]
mc alias set veeamS3 https://172.17.32.52:9000 YOURS3ACCESSKEY YOURSECERTKEOr you can add your host directly:
mc config host add minio-veeam https://minio.jorgedelacruz.es YOURS3ACCESSKEY YOURSECERTKEYAnd then we will create an immutable bucket with a beautiful command
mc mb --debug -l veeamS3/immutable
mc: <DEBUG> PUT /immutable/ HTTP/1.1
Host: 172.17.32.52:9000
User-Agent: MinIO (windows; amd64) minio-go/v7.0.5 mc/2020-08-08T02:33:58Z
Content-Length: 0
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20200819/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-bucket-object-lock-enabled;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Bucket-Object-Lock-Enabled: true
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200819T092241Z
Accept-Encoding: gzip
mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Wed, 19 Aug 2020 09:22:42 GMT
Location: /immutable
Server: MinIO/RELEASE.2020-08-16T18-39-38Z
Vary: Origin
X-Amz-Request-Id: 162CA0F9A3A3AEA0
X-Xss-Protection: 1; mode=block
mc: <DEBUG> Response Time: 253.0017ms--debug allows you to see not just the final message, but more detailed information.
-l means --with-lock, which means immutable
If we now return to the web interface, then our new bucket will appear there.
For now, that's all. We have created a secure storage and are ready to move on to integrating with Veeam.
You can also make sure that everything works perfectly:
c:minio>mc admin info veeamS3
β 172.17.32.52:9000
Uptime: 32 minutes
Version: 2020-08-16T18:39:38Z
Network: 1/1 OK
Drives: 6/6 OK
0 B Used, 1 Bucket, 0 Objects
6 drives online, 0 drives offlineMinIO and Veeam
Attention! If for some incredible reason you want to work via HTTP, then create a DWORD key at HKEY_LOCAL_MACHINESOFTWAREVeeamVeeam Backup and Replication SOBRArchiveS3DisableTLS. Set its value to 1 and remember that we strongly disapprove of such behavior and do not advise anyone.
Attention again! If, due to some misunderstanding, you continue to use Windows 2008 R2, then when you try to connect MinIO to Veeam, you will most likely receive something like this: Failed to establish connection to Amazon S3 endpoint. It is treated with the official patch from .
Well, the preparations are over, let's open the VBR interface and go to the Backup Infrastructure tab, where we will call the wizard for adding a new repository.
Of course, we are interested in Object storage, namely S3 Compatible. In the wizard that opens, set the name, go through the steps with the address and account. If required, do not forget to specify the gate through which requests to the storage will be proxied.
Then select the bucket, folder and check the box Make recent backups immutable. Or we don't. But since we have made a repository with support for this function, it would be a sin not to use it.
Next > Finish and enjoy the result.
Now we need to add it to the SOBR repository as a Capacity Tier. To do this, either create a new one or edit an existing one. We are interested in the Capacity Tier step.
Here we need to choose which scenario we will work with. All options are pretty well described in another so I won't repeat myself
And upon completion of the wizard, tasks for copying or transferring backups will be automatically launched. But if your plans do not include immediately giving out the load on all systems, then be sure to set the allowable intervals for working on the Window button.
And, of course, you can make separate Backup Copy jobs. Some people think that this is even more convenient, since they are somewhat more transparent and predictable for a user who does not want to delve into the details of how a capacity shooting gallery works. And there are enough details, so once again I recommend the corresponding article at the link above.
And finally - the answer to the insidious question: what will happen if you still take and try to remove the backup from the Immutable storage?
Here's the answer:
That's all for today. According to the true tradition, catch a list of useful topics on the topic:
- Manual
- Usage example .
- General setup guide .
- branch on our about S3 storage.
Source: habr.com