Backup Strategy for Files at TB Scale
After a long run with the Mac as my primary machine, I just built a new desktop and switched back to Windows 10 as my main computer. I am currently setting up my backup strategy on Windows 10, with Acronis True Image 2018 for my local backups, and Backblaze for my cloud backup. I am only backing up files, I don't care about Windows or my programs, everything either saves to folders that I will be backing up to Backblaze and through ATI, or is cloud based, and thus it is basically just a local cache.
All my truly critical stuff is backed up through Backblaze, but it's simply not practical to download and unzip 2TB+ of stuff every time a hard drive or SSD bites the dust.
I have the following data in three main categories that I need to back up to my 6TB backup drive:
1. On 480GB SSD: Dropbox files and Lightroom catalog, totaling about 20GB. My Dropbox changes a lot due to phone camera uploads and is capped out at something like 17.4GB. I want a versioned backup of it, as Dropbox will totally lose its mind once in a blue moon. Anything I'm working on in Office or other programs is saved to Dropbox for immediate cloud-based backup (aside from when Dropbox loses its mind).
2. On 4TB hard drive: My Documents (older stuff), Pictures (some referenced from Lightroom, most just in folders), my iTunes Library, and a few other miscellaneous things totaling about 950GB, and likely to grow significantly in the future. The Documents and Pictures are backed up on Backblaze, I will back iTunes up to Backblaze once I finish re-backing all my stuff up (you can't inherit backup state from Mac to PC, so I have re-upload it all on a 10mbps cable upload, about 100-150GB/day). I don't touch this stuff that often, but when I do add pictures, it can be anywhere from a few hundred MB to upwards of 100GB if I went on a trip and shot RAW.
3. On 4TB hard drive: Deep Blue: This is deep storage, with all sorts of media files, media projects, software installation packages, and other miscellaneous stuff from several different computers I've had over the years, totaling about 990GB. This changes relatively rarely. I'm not sure yet if I'll ever back this stuff up to Backblaze, as it's important, but not *that* important.
The plan is to eventually add a 6 or 8TB hard drive as my picture collection grows, leaving Deep Blue on the 4TB drive, and at that point I would also split the backup, likely with an 8 or 10TB drive handling my Documents, Photos, and Music.
At first I set up ATI with a single backup for all three categories, but now I'm thinking that's totally the wrong strategy. I'm thinking I should set up separate backup sets for each with different rules, that would make separate .tib files in different folders on my backup drive.
Coming from Apple Time Machine, which is idiot proof and does everything automatically (said with the Steve Jobs accent), but also has almost no options and is pretty much take-it-or-leave-it, so I have a few questions when creating backups:
1. Is there a way to use a combination of differential and incremental backups together? If I make two backup sets, one doing differentials less frequently, and one doing incrementals more frequently, can they share the same original backup, or do they have to have two different copies of the data?
2. Is there a reason to do an original backup again, or just use differentials off of the OG backup indefinitely and keep x number of differentials? Time Machine seems to backup once, and then do a combination of incrementals and differentials, but I'm not sure what tricks HFS+ gives Apple that isn't available to ATI. I do understand the logic of not wanting to have to restore from hundreds of incremental backups, as one corrupted one kills the whole chain, but what's the harm of doing differentials off of the same OG for months or years, as long as the differential size doesn't get too huge?
3. If I have to do a new original backup, do I effectively need double the space for everything in order to do the second original backup before deleting the first? In this case, I could afford it, but at some point, I just won't have that much space. Should I do original + differential, and every so often manually re-do each backup one at a time, so that I don't need double the space of everything just for the original backups (not to mention the actual differentials, which would grow over time)?
On a related note, I have several other drives full of TV shows, movies, etc, that are not backed up at all, as they're not that important. However, I would like to have a list of the files on the drives. Is there an easy way to make a .csv, .xls, .txt., or .rtf with the file structure and names of files on a drive that I could save on my Dropbox, so if a drive went south, I'd at least know what I lost? This stuff totals 6TB+, so it's not really worth it to back it all up.
EDIT: Grammatical Issues
Steve,
Thanks for your quick and thorough response.
1. Ok, so that's out, as I don't want to have two backups of everything just in order to be able to have differential and incremental backups. Why doesn't ATI provide this functionality? It seems pretty logical to use both, or is it hard/impossible to do without HFS+ (which like everything Apple uses is relatively weird and nonstandard but has some technical advantages). The only exception for doing both might be my Dropbox backup, since it is a relatively small backup compared to the others and the size of the drives.
2. Ok, it sounds like incremental is the way to go then. So how many incrementals in the chain do most people allow? It seems very inefficient to be regularly re-doing the OG backup, as opposed to the differential/incremental method, but without a way of doing differential/incremental or cleaning up the differentials, then it sounds like incrementals with periodic full backups is the only practical way to do it with a reasonable degree of automation. The upside is that ATI with a USB 3 drive is an order of magnitude faster than TM running over USB 2, which was a total dog. It seems that I should make different rules for each set of data, plus I can manually trigger my photos backup after I add a lot of photos to the data set. I can also rely, to a certain extent, on Backblaze for whatever I have worked on in the last interval between the backup and a disk failure.
3. So my question now is, does ATI do each backup one by one, as opposed to doing multiple things at once? I.e. if I had a 5TB drive, and 3 1TB backup sets for an easy example, I couldn't back them all up again and then delete them all, as that would take 6TB plus the space for the incrementals, but if ATI did them one by one, cleaned them up, and then moved to the next one, I would be able to have them all on a 5TB drive, using 4TB plus the space for incrementals at any given point in time.
Agreed, there isn't likely to be much compression.
It's kind of like preaching to the Book of Mormon to the Tabernacle Choir posting on this forum, but backups have saved my butt several times, as just in the past couple of weeks I have discovered or had fail an SD card (Dropbox and Google Photos backups), a hard drive (was a redundant backup), and another hard drive's file system got corrupted (another redundant backup, reformatted and back in service), and that's not to mention previous failures of at least one HDD and one SSD that I can think of off the top of my head. I also remembered to back up my Parallels Virtual Machines on my Mac, as those haven't been backed up in quite a while.
Alexander, replies to your further points:
- I doubt that this is a restriction of a particular filesystem but is more a design decision by Acronis for a product aimed primarily at the home user market. Most users will decide to go with whichever backup scheme (Differential or Incremental) that meets their requirements best. I have not seen anyone asking in the forums to take a hybrid approach of being able to do both based on a single initial backup file, though suspect that some may put their hands up now this discussion is being aired!
- As above, this is really a matter of choice according to what works best for you.
Differential backups will tend to gradually grow in size over time, and can exceed the size of the original backups in some circumstances, i.e. due to Windows 10 Upgrades for such as the Fall Creator's Update which effectively replaces most of the OS resulting in a major increase in either Differential or Incremental backup sizes.
The question of how many Incremental backups in a single version chain is a matter of risk assessment as well as destination storage drive capacity. The greater the number of incremental files the potential higher risk should any one file become corrupted or damaged to cause the whole chain to be invalidated from that point forward.
I tend to use default values personally, i.e. typically 5 incrementals then a new full backup, but my full backups are relatively small at around 30 - 40 GB per OS partition, with separate backups of my user data on other partitions.
I am doing multiple different backups to different destination drives, i.e. to local, network and Cloud drive locations, complemented by using the Synology Cloud Drive feature of my Synology NAS which makes my user data available / accessible to my other computers. - Acronis True Image will only permit one backup task to be active at any one time, any further task that is started (manually or by schedule) will be queued behind the active task and will only start running when the current active task has completed.
Automatic cleanup for a task is a part of the above task action, so this will be applied and any qualifying older backups deleted once a new backup has been successful. This is shown in the ATI logs as 'Consolidation' which is a throw-back to how older versions of the product used to work, but further messages in the log will show which older file(s) have been deleted by the cleanup actions.
On the overall subject of backups and war stories about disk failures etc, it is vital that users create the Acronis bootable Rescue Media as well as creating backup version chains. The rescue media is the key method used to recover from any fatal disk drive failure, assuming you only have one computer with ATI installed. The media allows recovery in a bare-metal situation using a new raw disk drive to restore to. This media should be tested before is needed.
Steve,
Thanks for the second prompt and thorough response!
1. That's strange, as the differential/incremental model seems like the most logical way to do it by default. I am somewhat biased by the way that Apple has implemented TM, even though TM forces you to accept the "defaults" since there are basically no options.
2. My biggest concern with Differential is that there is no automated way to clean them up. I want something that can basically run on autopilot. In my case, the differentials won't get that big relative to the original backup, except for the Dropbox folder, which can have a lot of files going through it on a regular basis. I'm not backing up my OS, as there's really nothing there, and if my SSD goes south, I want to reinstall from scratch anyway. Chrome, Spotify, and Evernote keep their data in the cloud, and I'm going to back up there Library/data files for iTunes and Lightroom, everything else will be saved in a directory that gets backed up.
3. Ok, that's good, I should break them out even more, say to have a very long window between backups of my iTunes library, which hasn't changed in months or years, and shorter or longer windows depending on the type of data being backed up, with the ability to manually force a backup if I have a whole bunch of new pictures coming in.
I don't really see the need for creating bootable Rescue Media. Couldn't you just reinstall Windows from Microsoft, install ATI, and then restore files from there? Also, in my case, very little of my actual data is stored on my boot drive, almost all of it is on my HDD. I simply have too much stuff (I'm looking at you RAW files) to put onto a reasonably priced SSD, so most of my stuff has to live on my HDD, with Windows, programs, my Dropbox files, and my Lightroom catalog living on my SSD for fast access. I used to have my Mac set up this way when I had all my stuff there, and my HTPC is currently set up that way as well. Computers that don't have much of anything on them just have a modest SSD of 120-500GB for the boot drive, Dropbox, and any programs that I install.
I forgot to get back to the inventory of files on my HTPC, but I wasn't thinking ATI specifically- do you know of a utility that will do that? I should probably Google around and see if I can figure out what keywords would find something like that.
Alexander, automatic cleanup will work equally as well for both Differential and Incremental backup schemes, but this is operating on completed backup version chains (a full plus x diff or incr files with the same _bx_ value). So this can work but will always involve creating a new full backup to do so.
The Acronis bootable Rescue media would remove the need to have to do a clean install of your Windows OS on the boot drive, but if your plan is to always do a full reinstall, then yes, you can reinstall Windows, install ATI and then do any restores of your other HDD or SSD drives from that point forward. Hopefully, you won't get to a complete hardware failure scenario that will require such a full rebuild of everything but never say that it is impossible to happen! I have only lost one system from a lightning strike > power surge a long, long time ago, and have had surge protection, UPS etc in place ever since - lesson learnt painfully at the time!
On the inventory question, the simplest option would be to create a small batch file with a command like below, where the output is redirected to a file.
dir c:\ /s >j:\myfiles.txt
Doing the above on my root C:\ drive gave me a 27MB text output file!
Ok, I see. With differential backups, you're not getting the space advantage that you could if you cleaned out previous differential backups, and left the same original backup.
I ended up setting up 5 backups that are all incremental, and have various frequencies and cleanup settings. They are running now, and I'd have to say WOW, Acronis is fast. I'm sure USB 3.0 has something to do with it as well, but the largest backup is already done, so I'm probably 50-60% done in total already.
Gotcha. I have everything behind a UPS. I would do a full reinstall anyway. I'm concerned about data, not a disk image per se, as my data has moved with me from one machine to another anyway.
Thanks, I'll give that a try!
On my new machine, dumping a text file of the c:\ drive was about 29MB, on my HTPC, which had been upgraded from Windows 7, it was over 83MB!! I looked at what's chewing up space on the boot drive, and it's got 25GB of the previous Windows installation on it! Time for Disk Cleanup! The files of what I wanted, however, of the files on the d:\ and e:\ drives are under 1MB combined, as they are just data storage drives. It worked like a charm!
Alexander, if you ever have to do a clean install of windows, you should then perform a disk mode back up of the install and save it in a safe place. Then, if you ever want a clean install, just restore the saved .tib file. It would have to be quicker than a clean install.
In reply to Alexander, if you ever have… by truwrikodrorow…
FtrPilot, I see what you're saying, but by that point, most of the software has to be updated/reinstalled anyway, and its a good time to get new driver packages and whatnot, so the savings are relatively marginal, and you end up with a cleaner install by just starting over from scratch. My approach on the Mac is almost entirely the opposite, I've been restoring off of TM backups for 6 years now, but prior to that, I would reformat my XP PC every 9-12 months to deal with bit rot. I'm hoping that Windows 10 will last many years before needing to reformat, but I remember the XP days when you were lucky to get a year out of an installation before the thing was so rotten you had reformat.
With Windows 10 upgrade pattern so far bit rot should be a thing of the past. You can download the latest Windows 10 install file and do a clean install on your existing Win 10 hardware and be good to go sans drivers. If you backup all your drivers you can restore them on a clean install using third party tools like DriverExtractor, look HERE
Extract your driver files to a USB thumb drive and after you do a clean install use Window Device Manager to add back all the drivers. Fast and gets you a fresh system quickly. At that point adding back user data is pretty painless as well if, you have planned well that is!
There are other tools that will extract and then add back your driver files too. Google it!