Here, people often write posts about how they store and back up their photos — as well as just files. In the last such post, I wrote a fairly long comment, did some thinking, and decided to expand it into a post. Moreover, I have changed my backup method to the cloud a bit; it might be useful for someone.
Home server, where much of what is described below happens:
What needs to be saved?
The most important and voluminous for me are the photos. Occasionally videos, but very rarely — they take up too much space and time, so I don’t like them much. I only shoot short clips that pile up in the same heap as the photos. Currently, my photo archive takes up about 1.6 terabytes and grows by around 200 gigabytes a year. Other important items are much less bulky, and there are fewer questions regarding their storage and backup. A dozen or two gigabytes can be spread across numerous free or very cheap spaces, from DVDs to flash drives and clouds.
How is it stored and backed up?
My entire photo archive currently occupies about 1.6 terabytes. The master copy is stored on a two-terabyte SSD in my home computer. I try not to keep photos on memory cards longer than necessary and transfer them to my desktop or laptop as soon as possible (when on the go). However, I don't delete them from the flash drive if there's still space. An extra copy never hurts. Once home, everything from the laptop is also transferred to the desktop.
A copy of the photo folder is made daily on my home server (with a type of mirror based on Drivepool, where important folders are set up for duplication). By the way, I still recommend Drivepool — in all the years of use, I’ve had no glitches. It just works. You just shouldn’t pay attention to its Russian interface; I sent the developers a more decent translation, but I don’t know when they’ll implement it. Meanwhile, in Russian, this program is for managing a pool.
Of course, copies can be made more often. If a lot of work is done in a day, I can manually trigger a backup task. Although now, I’m considering starting to run backups based on file changes; I want to stop keeping the desktop running 24/7 and let the server do more work. The program is GoodSync.
Until recently, files were uploaded to the OneDrive cloud from the same desktop using GoodSync. Most of the files are not personal, so I uploaded them as is, without encryption. The personal files were uploaded separately, with encryption.
OneDrive was chosen because the Office 365 Home Premium subscription for 2000 a year offered five (now six) terabytes in the cloud. Even if in chunks of one terabyte. Now, unfortunately, the deal has become a bit more expensive, but a few weeks ago there were still options for 2600-2700 a year (must check with retailers). I anticipated this when MS raised prices last year and even stopped selling subscriptions on their website, so I activated a five-year subscription in advance while there were still boxes available for 1800-2000 (I could have taken a few boxes for backup, but I didn't dare plan that far ahead).
The upload speed is maximum for my plan, 4-5 megabytes/sec., at night up to 10. I once looked at CrashPlan — it was good if it was loading at a megabyte per second.
Lifetime 5TB for 2-3$ from eBay is a very random thing. The lifespan can turn out to be very short; three months is the record so far. It’s not a good idea to back up to a place that could shut down at any moment, even for pennies.
However, now that I’ve decided to move some tasks from the desktop to the server, I’ve switched the copying to OneDrive to Duplicati. It may still be in beta, but I’ve been using it for several months and it has been quite stable so far. Since Duplicati stores its backups in archives rather than loose files, I decided to encrypt everything being uploaded with the built-in tools. After all, if something goes wrong, I’ll have to restore through Duplicati anyway. So let it encrypt everything.
Considering that my terabytes are in chunks, the cloud backup consists of several tasks. Here, the backup is being re-uploaded to the cloud. 2019 uploaded quickly — there were about fifty photos in just a couple of days; I didn’t travel much at that time, and 2018 is gradually uploading. The current upload speed is not at its maximum — it’s during the day, the channels are busy, and all that.
In the cloud, the backup folder looks like this — many zip archives, the size of the archive is adjustable when the task is created:
About once a month, I back up to an external drive that is kept in a cabinet. I connect it and manually start the task with GoodSync. Although, of course, you can set it up to start automatically when the drive is connected — but I don't always need to make a backup when I connect the drive.
Ideally, I would need another remote storage location — my own and not overly cloud-based. On my server, which is located on the provider's platform, I've had a drive set up for this purpose for a long time, but I haven't gotten around to it. However, since I've started moving everything to Duplicati, I think I'll take care of this now, after I re-upload everything to OneDrive.
How is it cataloged?
This question splits into two parts — the file system level, where cataloging is done at the level of folders, and logical cataloging based on a greater number of parameters, because the folder tree is still limited in capabilities.
Yes, I shoot in RAW. Because from RAW you can create a JPG at any time, but not vice versa. I used to shoot in RAW+JPG to have the option to quickly transfer photos to my phone and send them online (it was hard to transfer RAW to my phone). I would then delete the JPGs when copying to my desktop. But now I find my phone satisfactory in terms of photo quality (for posting online), so I've completely eliminated JPG images on my cameras. The remaining ones are either from the time when I didn't have a mirrorless camera or come from my phone.
At the file system level, it looks like this: at the top level of folders — the source. Usually the names of photographers.
One level down — themes. They are generally similar, but there can be personal themes (for example, "Dogs"), and some themes might not exist.
Next are the years. Inside each year, folders are organized by days. Within a folder, there can be separate photo sessions if the photos for the day are categorized by theme.
As a result, the path to a file might look something like this: MyTrips20182018-04-11 BerlinFrench StationP4110029.ORF
I shoot with two cameras, typically alternating between them, but occasionally I take both with me — then I dump the photos from both into one folder. The main thing is to keep the time synchronized; otherwise, I have to calculate the difference and adjust the shooting date for all files (it’s easy in Lightroom, but tedious to calculate the time difference).
There is a separate folder for photos from the phone on the second level, but if necessary, a photo can be sent to a thematic folder.
Logical cataloging over folders — Adobe Lightroom. Of course, there are quite a few programs for cataloging and processing, but Lightroom works for me, it is reasonably priced (and they even offer Photoshop in the bundle), and in the last couple of years, it has become less laggy. Though, of course, the full transition to SSD also helped.
All photos reside in one directory. The folder structure from the previous point is used as a base, with additional information like EXIF data, geotags, tags, and color markers added on top. Face recognition can also be enabled, but I don’t use it.
Based on all of the above, you can create ‘smart collections’ — dynamic samples based on certain file properties — from shooting parameters to text in comments.
All tags are stored in files, editing history is in XMP files next to the RAWs. The Lightroom catalog is backed up by Lightroom’s own tools once a week to a specific folder, from which it is then uploaded to OneDrive. Additionally, the system disk of the desktop is backed up daily to the server via Veeam Agent, as the catalog is stored on the system disk.
Is it all about photos? Are there no other types of files?
Yes, there are. The backup methods do not differ (if backup is even needed), while the cataloging methods depend on the type of content.
For the most part, folder-level sorting is sufficient, tags are not needed. Only for movies and series is a separate cataloging tool used — Plex Media Server. It also serves as a media server, as the name implies. However, there hasn’t been much progress; only about a quarter of my film library is well sorted, while the rest is lying in a folder called ‘!to sort’.
Source: habr.com
