For a long time I wanted to write about how I store my files, how I make backups, but my hands never got around to it. Recently, an article appeared here, somewhat similar to mine, but with a different approach.
I've been trying for years to find the perfect method for storing files. I think I found it, but there is always something to improve, if there are any thoughts on how to make it better, I will read it with pleasure.
I'll start by telling you a few words about myself, I'm a web developer and take pictures in my free time. Hence the conclusion that I need to store work and personal projects, photos, videos, and other files.
I have about 680 GB of files, of which 90 percent are photos and videos.
Cycling files in my vaults:
Here is an approximate diagram of how and where all my files are stored.
Now the details.
As you can see, the heart of everything is my NAS, namely the Synology DS214, one of the simplest NAS-s from Synology, however, it copes with everything I need.
dropbox
My working machine is macbook pro 13, 2015. I have 512GB there, but not all files fit by themselves, I only store what I need at the moment. I synchronize all my personal files and folders with Dropbox, I know that this is not very reliable, but it only performs the synchronization function. And he does it best, at least from what I have tried. And I tried all the famous and not very clouds.
Synology also has its own cloud, you can deploy it on your NAS, I tried several times to transfer from Dropbox to Synology Cloud Station, but there were always synchronization problems, always some errors, or not everything was synchronized.
All important files are stored in the Dropbox folder, sometimes I save something to the desktop, so as not to lose something, I made a symlink to the Dropbox folder using the MacDropAny program.
The Download folder is not synchronized in any way, but there is nothing important there, only temporary files. If I download something important, then I copy it to the appropriate folder in Dropbox.
My adventures with DropboxOnce upon a time, somewhere in 2013-2014, I kept all the files in Dropbox and only in it, there were no backups. Then I didn’t have 1Tb, that is, I didn’t pay for it, I had about 25Gb, which I earned by inviting friends or other tasks.
One fine morning, I turned on the computer and all my files disappeared, I also received an email from Dropbox where they apologize and that my files were gone due to their fault. They gave me a link where I could restore my files, but nothing was restored by itself. For that, they gave me 1Tb for a year, after which I became their client, no matter how strange it may sound, but I never trusted them.
As I wrote above, I could not find a cloud more suitable for me, firstly, there were no synchronization problems yet, and secondly, a lot of all sorts of services work only with Dropbox.
Go
Work files are stored on the working server, personal projects on GitLab, everything is simple here.
time Machine
I also make a backup of the entire system, excluding the Dropbox and Downloads folder by itself, so as not to take up space in vain. I backup the system through Time Machine, an excellent tool, it has helped me more than once. I do it on the same NAS, since there is such a function there. You can do it on an external HDD, of course, but it's not so convenient. You need to connect an external drive every time and start Time Machine yourself. Due to laziness, I often made such backups every few weeks. On the server, he automatically makes a backup, I don’t even notice when he does it. I work from home so I always have a fresh backup of my entire system. A copy is made several times a day, I did not count how many times and how often.
NAS
This is where all the magic happens.
Synology has a great tool, it's called Cloud Sync, I think it's clear from the name what it does.
It can synchronize many cloud systems with each other, to be more precise, it can synchronize files from a NAS server with other clouds. I think there is an overview of this program on the network. I won't go into details. I'd rather describe how I use it.
On the server, I have a disk folder called Dropbox, it is a copy of my Dropbox account, Cloud Sync does all the synchronization. If something happened to the files in Dropbox, it will happen on the server as well, it doesn't matter if delete/create. In general, classical synchronization.
Yandex disk
Further, I throw all these files on my Yandex disk, I use it as a homemade backup disk, that is, I throw the files there but do not delete anything from there, it turns out such a dump of files, but it helped out a couple of times.
Google Drive
I send only the “Photos” folder there, also in synchronization mode, I do this only for convenient viewing of photos in Google Photos and with the ability to delete photos from there and they are deleted everywhere (except for the Yandex disk by itself). I will write about the photo below, there you can generally write a separate article.
hyperbackup
But all this is not very reliable, if you accidentally delete a file, then it will be deleted everywhere and you can consider it lost. You can, of course, restore from a Yandex disk, but firstly, a backup in one place is not very reliable in itself, and the Yandex disk itself is not the service in which you can be 100% sure, although it has never had problems.
Therefore, I always tried to store files somewhere else, already with a normal backup system.
Synology also has a tool for this, it's called HyperBackup, it backs up files either to other Synology servers or to some third-party cloud solutions.
He also knows how to backup to external drives connected to the NAS-y, which is what I did until recently. But this is also not reliable, for example, if there is a fire, then the end of both the server and the HDD.
Synology C2
Here we are smoothly approaching another service, already from Synology itself. She has her own clouds for storing backups. It is sharpened specifically for HyperBackup, it makes a backup there every day, but this is already a well-thought-out backup, there are file versions, a timeline, and even clients for windows and mac os.
That's all, with file storage, I hope my files are safe.
Now let's move on to sorting files.
Regular files, books, document scans and other unimportant files, like everything else, I sort into folders with my hands. Usually there are not many of them and I rarely open them.
The most difficult thing is sorting photos and videos, I have a lot of them.
I take photos from a few dozen to a few hundred a month. I shoot with a DSLR, a drone, and sometimes on my phone. Photos are personal and for stocks. I also sometimes shoot home videos (not what you thought, just family videos, more often with my daughter). It also needs to be stored and sorted somehow, so that there is no porridge.
I have a folder in the same Dropbox called Sort Images, there are subfolders where all the photos and videos go, from there they are taken and sorted where necessary.
Sorting takes place on the NAS server, there are bash scripts that run automatically once a day and do their job. The NAS is also responsible for their launch, there is a task scheduler which is responsible for launching all scripts and other tasks. You can configure how often and when tasks will run, cron interface if easier.
Each folder has its own script. Now more about folders:
Drone - here are photos from a drone that I took for personal purposes. First, I process all the photos in lightroom, then I export the JPG to this folder. from there they end up in another Dropbox folder, "Photo".
There is a folder "Drone" and there they are already sorted by year and month. The scripts themselves create the necessary folders and rename the photos themselves according to my template, usually this is the date and time the photo was taken, I also add a random number at the end so that files with the same name do not turn out. I don't remember why setting seconds in the file name was not suitable for these purposes.
The tree looks like this: Photo/Drone/2019/05 - May/01 - May - 2019_19.25.53_37.jpg
Drone Videos - I am not shooting video with a drone yet, there is a lot to learn there, there is no time for this now, but I have already created a folder.
Image Operations - there are two folders inside, when files come across there, they are simply either compressed on the maximum side to 2000px for publication on the Internet or a flip of pictures is made, I don’t need this anymore, but so far I haven’t deleted the folder.
panoramas - panoramas, as it’s not difficult to guess, get here, I store them separately, since this is a specific type of photo, I usually take them with a drone. I also do regular panoramas, but I also do 360 panoramas and sometimes spheres, such kind of panoramas as small planets, I also do it with a drone. All photos from this folder also end up in Photo/Panoramas/2019/01 - May - 2019_19.25.53_37.jpg. Here I do not sort by months, since there are not so many panoramas.
Personal Photo - Here are the photos that I take on a SLR, usually these are family photos or travels, in general, photos that are taken as a keepsake and for myself. I also process raw photos in lightroom and then export them here.
From here they get here: Photo/2019/05 – May/01 – May – 2019_19.25.53_37.jpg
If I shot some kind of celebration or something else that would be better stored separately, then in the 2019 folder I create a folder with the name of the celebration and copy the photo there with my hands.
RAW - here are the source of the photo. I always shoot in RAW, I store all the photos in JPG, but sometimes I want to store RAW files as well, sometimes I want to process a frame differently. Usually this is nature and only the best shots get there, not all in a row.
Stock Photo - here I upload photos for stocks, which I do either on a DSLR or on a drone. Sorting is the same as on other photos, just in its own separate folder.
In the root directory of Dropbox, there is a Camera Uploads folder, this is the default folder in which the Dropbox mobile app uploads all photos and videos there. All photos of the wife from the phone are thrown off in this way. I also throw all my photos and videos from my phone here and sort them into a separate folder from here. But I do it in a different way, more convenient for me. There is such a program for Android, FolderSync, it allows you to take all the photos from your mobile, upload them to Dropbox and then delete them from your phone. There are many settings, I recommend. Videos from the phone also fall into this folder, they are also sorted, like all photos, by year and month.
I collected all the scripts myself from all sorts of instructions on the Internet, I did not find ready-made solutions. I don’t know anything about bash scripts at all, maybe there are some errors or some things could be done better, but the most important thing for me is that they do their job and do what I need.
Scripts uploaded to github:
Previously, I used Hazel for mac os to sort photos, videos, everything is easier there, all tasks are created visually, there is no need to write code, but there are two minuses. Firstly, you need to keep all the folders on the computer so that everything works well, and secondly, if I suddenly switch to windows or linux, then there are no such programs there. Tried to look for an alternative but they were all about nothing. A solution with scripts on the server is a more universal solution.
All scripts are configured to run once a day, usually at night. But if there is no time to wait and you need to somehow execute the desired script now, there are two solutions: connect via SSH to the server and execute the desired script, or go to the admin panel and manually run the desired script in the same way. All this seems to me not convenient, so I found a third solution. There is an Android program that can send ssh commands. I created several commands, each has its own button, and now if I need to sort, for example, the photos that I took from the drone, then I just press one button and the script works. The program is called SSHing, there are others similar, but for me this is the most convenient.
I also have several of my own sites, they are more for show, almost no one goes there, but it still doesn’t hurt to do a backup. I keep the sites on DigitalOcean, I installed the aaPanel panel there. There is an option to backup all files and all databases, but on the same drive.
Keeping a backup on the same disk is not the case, so I also knock there through the bash script and copy everything to my server, archiving everything in one archive with the date in the title.
I hope at least someone will help the methods that I use and with which I shared.
As you can see from the article, I love automation and I try to automate everything that is possible, I did not describe many things from the point of view of automation, since these are already other topics and other articles.
Source: habr.com