Skip to main content

Full backup going so slow now pretty much dead stop

Thread needs solution

So ...

I have one backup scheme for my laptop itself, SSD and internal HDD. That's fine.

I have a second files and folder backup scheme for backing up most of the folders on a 'data' USB HDD (mostly Video, pictures and 3D CGI assets). This is my 'Content' backup. This is not fine.

Both backups are written onto a 6TB USB HDD. Both are a full backup, followed by a certain number of incremental backups, both once a week (on different days).

I did the first 'Content' full backup in March, at which time it was of 2.6 TB of data in a 2.19 TB .tib file that took about a day and a half. It carried on doing weekly incrementals. At first I had configured it just to go on with incrementals, but after a few I changed the configuration to do 7 incrementals, then a new full to start a new version chain. However it carried on with an extra incremental, so I then finished off that scheme and replaced it with a new scheme with the same details two days ago. It started doing the full backup to start a new version change. It's now 48 hours, and it claims still to be going. It is reporting that it has got to 1.18 TB of the now 2.7 TB data, while the .tib file it is writing is currently 823 GB. In the Task Manager Performance tab the (4 TB) USB HDD that the content is being backed up from, I'm seeing a read speed wandering around in the tens and few hundreds of KB/s with the odd bit of 1 to 2 MB/s, while the 6TB USB HDD the backup is being written to mostly showing 0 in and out with the odd moment of 4 or 8 KB/s, despite the original full and all the incremental backups all show speeds in the hundreds of MB/s.

When the backup started it was on the default low priority; a few hours later I put it up to normal, and a day ago I switched it up to high. But basically it appears to be crawling along doing a few KB/s if that.

So two questions:

1) What's gone wrong?

2) What do I do now? At the current rate it's going to be a week or more to finish. And I'm not entirely sure how to stop this full backup and re-do it, if I should. And with even a proper full backup taking a day and a half, I hate the thought of trying and retrying without knowing what's gone wrong.

0 Users found this helpful

David, it's tough to say exactly what the issue is without more troubleshooting, logs, etc.

However, I would first ask how much free space remains on the 6TB drive and how fragmented is the disk? 

Also, USB performance can be all over the place and write speeds to an external USB 7200rpm will fluctuate based on the type of data written. Large files go faster. Folders that have many small files and nested folders go slower - this is Windows / disk behavior and can be seen with simple copy and paste tests.

When the initial backup occured, it's likely the disk was relatively empty and unfragmented but is the same true now? 

Another thing... Are you using the default backup size and creating a single 2+TB .tib? I think that could also be a performance issue, especially if the drive is filling up or fragmented, compared to the initial blank slate it was the first time you backed up.

Think about the best biggest file you have in normal usage... maybe a Blu-ray iso at 24GB? And now you're creating a single 2+TB file - that's huge! Hard drives need to cache the data for writes. As space fills up on the destination disk, that caching becomes less effective. Also, a single large file can not use contiguous space on a drive that is filling up or fragmented as it's not likely that there is a 2Tb sequential block available at this point.  Using a contiguous block for a large file would be faster than random writes all over the disk.

My guess, if you had a blank, empty drive again, it would be somewhere in the ballpark of the original backup speed. Large TB spinning USB drives are not great performers, but are good for slow, cheap storage where Reads generally take place compared to writes. For the $ to performance ratio, they're great too, but realize that they are slooow for large file writes like this.

Would I stop this backup ... Not unless you have more pressing things that require a reboot or the performance of the system is compromised to the point it's unusable. Let it run it's course. Disregard the estimated time stamp too. It may just be working on a section of small files which is slowing it way down and could improve when it gets to larger data chunks. 2.7TB is a lot to backup as a full and would expect it to take several days to a USB external spinner. 

Next time, try setting the max .tib size to 50GB or something like that and make sure there is plenty of space or it's going to get even slower as the disk fills up. 

 

 

As with the other thread I started, I set to receive notifications of replies, but never received any notifications so thought there hadn't been any.

Thanks for replying above - sadly I've only just seen it when coming to the forum to ask about a different problem. At the time I actually did end up cancelling that new full backup, deleted that backup scheme, and instead of having the one file-and-folders backup of all 2.7TB of all three large top level folders on that USB drive, replaced it with three file-and-folders backups, one for each large top level folder. It's not an even split - one 685 GB for my films/videos folder, one 28GB pictures folder, and one 2TB for my 3DCGI assets folder. But the original full backups (before deleting the old all-together full and incrementals) and subsequent incrementals have all run at normal speed, even the first full backup of the 2TB folder. Now I'm just waiting to see if this time it does do full backups and start new version chains, unlike the original.

Sorry to hear you didn't get the reply!  I'm in the forum a lot anyway and don't pay notice to the emails as much these days so haven't noticed missing responses, or not.  Hopefully you receive it this time.

In the end, I think your new setup is probably better with it broken into smaller manageable chunks.  That way, if anything happens to just videos, or just pictures, or just 3DCGI assets, it has less to parse through and/or backup and/or recover just from that pool of data.  And you can play around with backup schedules, frequency, etc, based on the type of data in each backup.  As long as you're staggering the cleanup dates/times, you should be OK, but you can see if you have a period where there are 2 fulls of videos and 2 fulls of 3D CGI assets at the same time, you could still have a space problem if there 

The 3DCGI folder is still pretty big... keep in find that any cleanup will not take place until after a new full is done.  Space could get tight here on the 6TB drive if there 2 x 2TB fulls of the 2DCGI folder plus incrementals or differentials between them 

685GB Full Backup #1... plus incremental 1, 2, 3..., then new Full #2 = 1 chain) Deletion of old chain #1 could then occur if keeping only 1 chain

28GB Full Backup #1 ... plus incremental 1, 2, 3..., then new Full #2 = 1 chain) Deletion of old chain #1 could then occur if keeping only 1 chain

2TB Full Backup #1 ... plus incremental 1, 2, 3..., then new Full #2 = 1 chain) Deletion of old chain #1 could then occur if keeping only 1 chain

I have set all 4 backups (the 3 from the USB drive, and the whole PC one for the internal drives) to happen on different days (they are all weekly) and delete the old version chain once a new one is started and validated, so things are OK at the moment. My one remaining issue is with the PC backup schedule. Originally I set the backups (back then, the one PC and the one file-and-folder for all the three big top level folders on the USB HD) just as incremental only before I'd fully understood and thought it through, then changed both the schedules to one full backup followed by 7 incrementals. Unfortunately both original backups ignored the change and just kept on doing incrementals. I ended up stopping and deleting both schedules, then recreated them as 1 full and 7 incrementals, getting new full version chains starting for both, while reimporting the originals and then deleting them from within ATI. Things appeared OK with the big file-and-folder one, and it did start doing the new full after the 7 incrementals, although that was the one that slowed and slowed, going on for days that had me starting this topic, and which I ended up aborting, setting up the 3 replacement schedules, then deleting the old single one after the three new full ones had completed and been validated. I have some weeks to go before seeing if they do follow the scheme this time with a new full after the 7 incrementals. The problem is that the new schedule for the PC backup, this time created with 1 full and 7 incrementals from the beginning, has ignored what I configured when I set it up and is just going on and on doing incrementals - last week it did the 15th incremental despite being set to only do 7 before doing another full one. I can't see a way to force it to do a full one, so I guess I'll just have to stop that schedule, create another one and try again, deleting the old version chain when the new full one is completed and validated. But as that is what I did before, I'm not confident I'll get a different (i.e. what I configured) result this time. At least the full PC backup is just minutes, not many hours like the 2TB one.

It's interesting that the incremental backup on the 2TB 3DCGI folder takes around 2 1/2 hours, while the others are all just a few minutes - even the 685GB films folder incremental is only 2 1/2 minutes. Presumably that's because the film folder is a relatively small number of large files, while the 3DCGI folder, while only 3 times the size, is a LOT more than three times the number of relatively small files.