Best strategy for backing up large amounts of files using the least amount of space and processing power?
Hi there, I have been a customer for years and have been reading everything I can on these subjects; however, I still have major problems dealing with my system.
Basically, my main problem is that I have a VERY large backup set thus have the following problems:
- I keep running out of backup space (even though I have several 1TB+ drives!)
- The backup operations (especially when consolidating) take hours, if not days (even on my super high-end computer!)
The reasons?
- I am a VERY active engineer who installs and uses hundreds of applications. (Literally!)
- I have collected thousands of files and Terabytes of data that I need protected. (And unfortunately almost all of it is in mp3/zip or some other compressed form.)
I understand that backup for me is a daunting task. However, I am backing up so much that even though I have 9 hard-drives, I keep running out of room and the backup processes (especially during consolidation) take so long that it is unusable. (See below for more detailed information.)
I am not too concerned about going back to a specific point in time (For instance, a music file will not change over time) and I use VSS (version control) and online redundancy on my project/engineering files. In fact, I do not even care if I lose even a month's worth of data! It is the years' worth of stuff that I fear.
I always thought incremental backups were a mechanism to save time and space by not performing full backups all of the time - is this incorrect? (From my reading it now seems like they are designed to allow the user to go back to a specific point in time.)
If this is true, then what backup strategy would you recommend for my needs (fastest performance and smallest size...)
Anyway, here is my current setup and related problems....
Disk/Partition backup
- Size: ~200 GB (30GB of program apps on my C: SSD and 140 GB or so of app data/downloads/etc on my Velociraptor D: drive)
- Includes: applications, downloads, system, etc...
File Backup
- Size - about over 1 Terrabyte (and I want to backup much more!)
- Content: Unfortunately mainly music and video and other data that does not benefit much from compression.
Current backup solution (for both backups...)
- Scheme: Incremental. (Create a full version every 5 incremental versions. Store no more than 1 recent version chain.)
Issues:
- Incremental backups always take a long time. I always thought that that one of the biggest points of incremental backups is that it is faster than doing a complete backup each time. Is that not the case?
- Even without defragging or anything, they are often still very large.
- Once I hit the consolidation point, the process takes at least 20 hours on the image backup. (Actually, I just want to do the equivalent of deleteing the chain and creating a new clean backup. Should I be using "consolidation" instead of "automatic cleanup."?)
- I gave up on cleanup/consolidation on the file backup after waiting over two straight days.
Thank you so much for any help at all on this! My entire life will be changed if I was not spending so many hours doing this!
Mike


- Log in to post comments

Check out some methods used by others. Click on my signature link below and read the comments under item #12. Most of what is there is written along the procedures outlined by Pal L above.
Use Acronis to regularly backup your SD and Raptor and use a mix of methods on the data files. The key is not to backup the same files over and over. If you use some the programs suggested by Pat L, and you have some duplication , then repetitive backups are not necessary. For example, I do not know how much of your data is frequently used, you could have one large disk attached and use a matching second to add only the changes to what the 2nd disk is missing. There are all kinds of variations possible but read the reference links as a starter to think along a different path. Some photographers just keep multiple duplicate disk and part of your storage is have some storage elsewhere--not with your computer or your house. Floods or fires can play havoc with what is stored in your house.
Don't forget there is a risk in the use of incrementals. These form a chain and if any of the incremental becomes flawed or corrupt, then all more recent incrementals become useless because the chain has become broken. Whereas, a differential records all the changes since the last full, it only needs the base full plus any single differential to be restorable. The differential is larger and takes longer to create but is a safer choice.
Be sure when creating a backup of your SSD, you include all partitions as part of the backup. You might find it practical to include both the SSD and Raptor in the same backup as they are both small but that is a matter of choice.
- Log in to post comments

Another vote for Pat L's suggestion of a replication-based backup for data files.
An incremental is the fastest backup mechanism. I would assume that the vast majority of your MP3s and video files do not change so that should reduce the time. However just scanning through all the files in a TB takes time especially if there are lots of files.
- Log in to post comments

Wow thanks for the the thoughtful responses! (In addition, I appreciate all of the help you guys have given in the past on this forum. For instance, prior to posting I already read several of Grover's guides.)
Can I ask a couple followup questions?
Image/Partition backup
I think I am comfortable with this one using Acronis True Image and Incremental backups (on both the Raptor and SSD drives.) My only question: in the interest of speed and disk-space, is it preferable to set it to "Create only Incremental versions after the initial full version" then turn on Automatic Consolidation, or to set it to "Create a full version after every x [5?] number of incremental versions" and turn on Automatic Cleanup? (Perhaps setting that to "store no more than 1 recent version change.)
File Backup
Wow, no wonder I hit a wall! I did not even consider using a completely different fundamental paradigm and backup application! Long ago, as I started accumulating these files I simply made manual duplicate copies. However, I turned to Acronis when I was losing track of what I did and did not copy. Soon the folder structures became unwieldy. However, all of the points you made about how none of the Acronis schemes are ideal for mu situation make sense. (While I can handle re-imaging my working partitions from time to time, I do not want to keep hitting my mass of data. And I can see why even small incremental additions are taking so much time just figuring out the delta on what has changed since the last backup.)
That said, the testimonials in "point 12" in the guides still do not seem to apply to me. What would you think about these ideas:
- Breakup my data into smaller chunks. As I am typing this I just moved a terrabyte of music manually to a different drive. Now I have that in two places and I am not even going to touch any backup tools with that anymore. I think I will do the same with some of my old static videos too. Thus my "moving target" will now be a manageable size.
- However, going forward, it just dawned on me - would this be a case where "Nonstop Backup" would actually help? Generally I am just adding files. So instead of a file backup on a schedule trying to determine what has changed, a nonstop backup will simply make the second copy in real-time. I am being too hopeful thinking it works like a RAID 1 setup? (Heck, maybe I should consider RAID 1. (I even bought a new HDD yesterday so I can cleanly begin my new backup system...))
I apologize... I am not a morning person and have to run to work right now, I will do some of this research myself so as not to be hand-held through all of this. However, I could not help asking what is on my mind since you guys know so much about this...If we end up with something unique and useful enough to others' I would be happy to return the favor and writeup a consolidated (and concise) guide to add to your collection. :) )
Thanks again for your thoughtful and very helpful responses!
Mike
- Log in to post comments

I've never used Automatic Consolidation and based on what I've read in the forums it takes a lot of time. I also view it as another operation on the archive with the accompanying risk of an error.
I strongly support partitioning of the data and that doesn't necessarily mean a disk partition. There is no need to keep backing up files that haven't changed over and over. Doing an incremental using a product like SyncBack, Karen's Replicator etc does achieve this but you still have a lot of files being scanned for changes.
My data management scheme is:
C drive contains OS and applications. Any data files on this drive are considered scratch or temporary. Backed up by TI partition image so I can restore bootable partition.
D drive - data files of various types. Backed up by SyncBackSE incremental every night.
E drive - installation of large games like Flight Simulator. Rarely if ever change, and I have the original CDs anyway. Make one backup and leave it obviously if you add to it you have to make another.
Your unchanging music and video files would be similar to my E drive files.
You can do a lot of backup by just copying files with Windows Explorer. I have setup SyncBackSE to do copies to rack-mounted or USB drives (racks much faster) and this way I can also do a verify to ensure the backup is correct. I also have Versioning turned on.
I have no comment on Non-stop backup other than to say it doesn't offer me any real benefit.
Your problem is the large amount of data you are handling and IMO the first approach should be to reduce the amount that needs to be frequently dealt with.
If you PC handles TI well, the fastest way of backing up should be an incrmental image not what TI calls a Data backup which deals with Files and Folders and makes use of the OS' file system. The imaging bypasses the filesystem to a large extent which accounts for the speed.
Normally a good PC these days should handle about 3GB of archive per minute (It used to be 1GB/min but PCs are faster now) to an internal HD. USB2 will be much slower. If you are doing a full backup of 1TB of data the 3GB rate will take about 5.5 hours. If you validate the archive it will take another 5 hrs roughly. If it is being done to a USB2 drive then the 5.5 hrs will become about 30 hrs. I mention this so you can see what dealing with a TB of data means in terms of time.
- Log in to post comments

+1 on what seekfoever mentioned. You've got to love how some people like to think about backup strategies :-)
I am sure we can keep this thread going for a while...
About non-stop backup: I use a similar tool (but not non-stop backup) for the files I want to keep a version of if I edit them, reorganize. I cannot use NSB of Acronis because it doesn't handle EFS encrypted files the way I want, but it is a fine solution if you don't have EFS encrypted files (encrypted in explorer properties).
One other thing to think about: diversified redundancy. 2 backups for the same thing on different technologies AND media
OS and application disk/partition: I use 2 imaging tools (Acronis for versions, Win 7 imaging as a backup). Each is synched back on a disk I take to the office from time to time
Frequently changing content OR content that I cannot lose. A local NSB type backup for everything including movies and key settings. Online backup with versionning (iDrive) for all files (documents, pictures, finished home videos, key settings) except the ones I can buy back if I lose them (ie: movies, music, AVI capture files for which I have tapes)
Never/rarely changing content. I replicate the files that are in the NSB but not in the online backup on the disk I take to the office.
- Log in to post comments

Michael wrote:My only question: in the interest of speed and disk-space, is it preferable to set it to "Create only Incremental versions after the initial full version" then turn on Automatic Consolidation, or to set it to "Create a full version after every x [5?] number of incremental versions" and turn on Automatic Cleanup? (Perhaps setting that to "store no more than 1 recent version change.)
In my opinion, I would avoid the use of "automatic consolidation." Consolidation is time consuming an has been reported as unpredictable and reported as needing the missing files if you want to mount or view.
My preference is the "store no more than xxxx" and that has been working for me--something similar to this using either incremental or differentials as illustrated by the image below. If you want to move large amounts of data, I would suggest you consider (if hardware permits) the use of a USB 3. My laptop has such a USB3 connector and I am using a Thermaltake ST0019U docking station which is also USB3. The docking station enables me to switch between many of my loose hard SATA drives and combined with the new faster SATA 3 drives, the faster speed is really great.
- Log in to post comments

+1 on not using auto-consolidation.
- Log in to post comments

Wow, you guys are possibly the most helpful people I have ever spoken with on a forum like this. Thanks so much!
I have little to add right now as I am at work and I want to think through this a bit more (I already read your posts a few times. ;) ) I just wanted to let you know that I am absolutely paying attention and I will get back to you in the next day or so.
You have a point on how long we are spending figuring this out, but I do so much on my computer and my data is so important to me that I want to get this right. In addition, I have made poor backup choices in the past and it has resulted in random TIB files and copies of download/document directories in several random places on old drives.
Oh as far as USB3 - when I bought the new SSD I bought a SataIII controller with an eSATA port. I also bought a new internal drive BUT with an eSATA enclosure. I am getting read/write speeds of almost 120 MB/sec on the backup drives (and over 350 MB/sec read speeds on the SSD! (that's with a big "B"!) FYI although those are fast, even the fastest non-SSD drives (like the Velociraptor) cannot keep-up with Sata3 nor USB3 speeds. (But it definitely does not hurt and is fantastic with the SSD!)
I followed your diagram for my image/partition backup (thank you for taking the time and clarifying the consolidation issue!) and with those speeds I think I am set with my partition backup. (I have a dual-boot one on the 60GBSSD and the other on the 300GB Raptor...I was going to de-couple this and move my mbr to the new SSD, but it runs so fast anyway that backing up both drives as a set is no big deal.)
Right now I need to report back on my data backup. The most important (and prevalent) data is my music. I collect and trade legal live concert recordings...I add up to a Gigabyte's worth of music a week. Furthermore, since these files are received via a slow trading process done over the past decade, the collection is essentially irreplaceable. As I mentioned, right now I am making sure I just have two manual copies of these files. (BTW - even at my new super-fast speeds, I have been using "TeraCopy" to speedup the copy/paste procedures.) When I get home I think I will look into the apps that you guys mentioned (such as SyncBack) and start using it just for now going forward...
I also think I might give Acronis NSB a try for my documents/downloads. (This goes along with your good "diversification" point.) *That* is one aspect of what I do that is typical of most users. If I don't like it, those files are of a manageable size.
LOL - once again I am spending too much time [at work!] thinking/working on this. I was only planning on writing a one-line response just now. :)
Thanks again and I will get back to you on my progress as I do it. (I'll also put my entire system and strategy in my signature...)
- Log in to post comments

OK I'll keep this post short and sweet. ;)
Here is what I have done:
- I am doing an Acronis Image backup on my 60Gig SSD and 300Gig Raptor drives. (Custom scheme: Full version every 5 Incremental versions; store nor more than 1 recent version chain.) All Docs/downloads/executables/etc are on these drives.
- In addition, I created an Acronis Nonstop backup for my documents and Pictures. (Only from now going forward.) This is redundant of step 1) but I have it going to a different drive that is always connected, and I basically just want to test this out.
- I created a new Music folder, grabbed "SyncBackPro" and set it to Fast Backup, Mirror mode, since all I will be doing is trying to mirror whatever is in my music drive. I know this is not a forum for that app and I admit I did not read everything about it yet, but that seems to fit my needs best, especially considering I am expecting large volumes of data.
How does this sound? Is there anything more/different you would recommend? ...Again I cannot thank you guys enough for your help. I do not know if you are officially associated with Acronis, but I have never been more happy and satisfied with a backup solution in my life. (LOL for now at least. ;) )
MikeY
- Log in to post comments

Michael,
None of those assisting have any association with Acronis. Just a few contributors donating their time. Employees of Acronis have their support identification as part of their forum ID.
What you suggest sounds very workable. However, one important reminder.
Doing your backups are half the job. These could be useless if you do not prove your procedures by actually do some restores. You must make sure that your restorations procedures do in fact work.
At the very least, use your Rescue CD and restore some files into a special test folder.
Also, use the Explore option which is to right click on a *.tib backup file and choose the explore options. Copy some files from this procedure.
Also use the Mount procedure and mount one of your partitions and again copy some files.
In other words, test at least those 3 procedures and if possible, even restore a full disk backup to a test disk to make sure you can do so successfully.
Far too many posters do the backups but never test if they can recover. Then a crisis comes and they find themselves unable to do the recovery. Maybe its their disks not being recognized or a corrupt backup.
Just make sure you do some testing! Good luck.
- Log in to post comments

GroverH is right on the mark. Doing the backup is only half the job and until you've confirmed you can recover the files you don't have a good backup. It is imperative you test the TI rescue CD since it is Linux, not Windows, so you are dealing with a different environment with different device drivers and sometimes they aren't a good fit.
Checking the SyncBack mirrored files is fairly easy since they are all present in the same folder structure in their native format which makes them easy to examine.
- Log in to post comments