Death, divorce, moving are the three most stressful situations in the life of any person.
"American Horror Story".
- Andryukha, I'm leaving home, help with the move, everything won't fit me :(
- Okay, how many are there?
– Tons* 7-8…
*Ton (slang) - Terabyte.
Recently, in the process of Internet surfing, I noticed that despite the availability on Habré and similar resources of a lot of materials about the methods and models of migration of various types of data, questions on this topic still appear on the network. Which, for some reason, do not always receive detailed answers. This fact prompted me one day to collect notes on the implementation of a similar solution and issue them as a separate post.
In general, I have to transfer data from one device, system and service to another with some annoying frequency. Which, through trial and error, allowed me not only to get acquainted with a lot of interesting products, but also to find a balance between the functionality and the cost of the solution I want to talk about
Design
As it turned out as a result of design and survey work, the quality and efficiency of the migration process depends not only on the technical characteristics of the "sites" where the data is or will be located, but also on their physical location.
The migration manager is a computing node on which the "logic" of the process operates - the migration management software.
That is, in total there are two models for placing a “migration manager”
- Model A. If at least one of the sites is accessible only from within the local network, then the “migration manager” should also be placed on the same network. For the performance and time of migration is still limited by the speed and uptime of the channel connecting the sites.
- Model B. If both the source and the data receiver have access outside the local network, then the “migration manager” should be settled where the speed and uptime of the channel between them will be obviously better.
In order to somehow decompose the above, I propose to return to the tasks from the main issue of the article, and formalize them into a technical task.
First, I need to find out if the software I use supports the "clouds": Mail.ru, Yandex, Google Drive, Mega, Nextloud?
Short answer: YES!
I use .
Rclone - rsync for cloud storage. Open Source software designed to synchronize files and folders with more than 45 types and types of storages.
Here are just some of them:
Alibaba Cloud (Aliyun) Object Storage System (OSS)
- Amazon S3
– ceph
— Digital Ocean Spaces
—Dropbox
- Google Cloud Storage
— Google drive
- Google Photos
- http
-IBM COS S3
— Mail.ru Cloud
—Mega
—Microsoft Azure Blob Storage
— Microsoft One Drive
— Minio
— Nextcloud
- Openstack Swift
— Oracle Cloud Storage
— ownCloud
— Rackspace Cloud Files
- rsync.net
- SFTP
- WebDAV
— Yandex Disk
Main functionality:
- Checking the integrity of files using MD5/SHA1 hashes.
- Saving timestamps for creating / modifying files.
- Support for partial synchronization.
— Copy only new files.
- Synchronization (one-way).
- Checking files (by hashes).
- Ability to sync from one cloud account to another.
- Encryption support.
- Support for local file caching.
- Ability to mount cloud services through FUSE.
On my own, I’ll add that Rclone also helps me solve the lion’s share of tasks related to automating data backup in .
The next task is to choose a placement model for the "migration manager".
To all data sources - which are various public cloud services, there is access via the Internet. Including through the API. Two of the three receivers - too. It is not clear where Nextcloud itself is deployed and what access to it is there?
I counted five options:
- On your own server in your home/corporate network.
- On your own server in a rented rack of the service provider's data center.
- On a server rented from a service provider.
- On a virtual server (VDS/VPS) with a service/hosting provider
- From a service provider
Given that Nextcloud is still software for creating and using cloud storage, we can safely say that access to it via the Internet is available in all five options. And in this case, the optimal placement model for the “migration manager” will be - model B.
According to the model chosen as a platform for the "migration manager", I will choose one of the best, from my point of view, options - a virtual server in Russia's largest Internet traffic exchange point MSK-IX.
The third decision that needs to be made is to decide on the configuration of the virtual server.
When choosing VDS configuration options, you need to be guided by the required performance, which depends on the width of the channels between the sites, the number and size of files being moved, the number of migration streams, and settings. As for the OS, Rclone is a cross-platform software that runs under various operating systems, including Windows and Linux.
If you plan to run several migration processes, and even at a certain frequency, then you should consider renting a VDS with payment for resources.
Creation
According to the above, when creating the prototype for this article, I chose VDS in the following configuration.
costing 560 rubles / month. with 15% coupon discount NO STRESS.
This choice is due to the fact that the node under Windows OS, in order to comply with the conditions of our TOR, is configured more easily than under other operating systems available for order.
Offtopic: By the way, for greater security, this virtual server is assigned to one of the nodes . and access to it via RDP is allowed only from there ...
After creating a VDS and getting access to the desktop via RDP, the first thing to do is to prepare the environment for Rclone and Web-GUI. Those. install a new default browser, for example Chrome, since the originally installed IE 11, unfortunately, does not always work correctly with the software used.
After preparing the environment, download the archive with the software package and unpack it.
Next - in the Windows command line mode, execute the command to go to the folder with the extracted files. I have it in my admin home folder:
C:UsersAdministrator>cd rclone
After the transition, we execute the command to launch Rclone from the Web-GUI:
C:UsersAdministratorrclone>rclone rcd --rc-web-gui --rc-user=”login” --rc-pass=”password” -L
where “login” and “password” are the username and password you specified, of course, without quotes.
After executing the command, the terminal displays
2020/05/17 22:34:10 NOTICE: Web GUI exists. Update skipped.
2020/05/17 22:34:10 NOTICE: Serving Web GUI
2020/05/17 22:34:10 NOTICE: Serving remote control on http://127.0.0.1:5572/
and the Rclone web graphical interface opens automatically in the browser.
Despite the fact that the Web-GUI is still in the test version and does not yet have all the Rclone management capabilities that the command line interface has, its capabilities are quite sufficient for data migration. And even a little more.
Setting
The next step is to set up connections to the sites where the data is or will be located. And the first in line will be the main data receiver - Nextcloud.
1. To do this, go to the section Settings Web GUI.
2. We initiate the creation of a new configuration - button New Config.
3. Set the name of the site - field Name of this drive (For your reference): Nextcloud.
4. Choose the type or type of storage Select: For Nextcloud and Owncloud, the main communication interface is WebDAV.
5. Next, click on Step 2: Setup drive open the list of connection parameters and fill in.
- 5.1. URL of http host to connect to URL — hypertext link of the WebDAV interface. In Nextcloud, they are located in the settings - the lower left corner of the interface.
- 5.2. Name of the Webdav site/service/software you are using is the name of the WebDAV interface. The field is optional, for yourself, so as not to get confused if there are many such connections.
- 5.3. User name — Username during authorization
- 5.4. Password — Password for authorization
- 5.5. Bearer token instead of user/pass (eg a Macaroon) and Command to run to get a bearer token in advanced options, additional parameters and authorization commands. They are not used in my Nextcloud.
6. Next click Create config and in order to make sure that the configuration has been created, go to the section config web interface ... Through the same page, the newly created configuration can be deleted or edited.
In order to check the operability of the connection to the site, go to the section Discover... In field remotes enter the name of the configured site and click Open. If you see a list of files and directories, the connection to the site is working.
For greater persuasiveness, you can create/delete a folder or download/delete a file via the web interface.
The second in order to be connected platform will be Yandex disk.
- The first four steps are similar to the Nextcloud connection process.
- Then we leave everything as it is, that is, the fields in Step 2: Setup drive leave blank, do not change anything in the advanced options.
- We press Create Config.
- The Yandex authorization page opens in the browser, after passing which, a message about successful connection and an offer to return to Rclone.
- What we do by checking the section Config.
Migration
When we have two sites connected, we can already migrate data between them. The actual process is similar to checking the connection to Nextcloud, which we performed earlier.
- We pass to Discover.
- Choose a template 2-side by side.
- In each of remotes Specify the name of your site.
- We press Open.
- We see the directory of files and folders of each of them.
To start the migration process, all that remains is to select the desired folder with files in the data source directory and drag it with the mouse to the destination directory.
The mechanism for adding the remaining sites and actions for migrating data between them is similar to the operations performed above. If you encounter errors while working, then you can study the details about them in the terminal where Rclone is running with Web-GUI.
In general, the documentation is extensive and available on the site and on the Internet, and should not cause any difficulties in use. On this, I consider the first post on how to transfer files from one cloud to another, bypassing your PC, complete.
PS If you do not agree with the last statement, write in the comments: what “topic is not disclosed” and in what vein it is worth continuing.
Source: habr.com