Backup chain not deleting - new full not kicking off!

Hi.. so I have a Full backup created along with 11 differential files.  My settings are as follows:

•   Create  a Full Version after every 10 differentials
•   Store no more than 1 recent version chains

So - instead of creating diff 11, I would have expected Acronis to:

•     Delete the entire chain (full and 10 diffs)
•     Create a new full version

Can you explain why this didn't occur and how I can set this up so it does work this way?  My Full is over 1.4 TB so I don't want a second chain (no space for it) and only want a single set of data on this drive.  Thx!

Dom, sorry but Acronis does not work in the way you are hoping for here - it will not delete any files until after it has successfully created a new Full backup file for a new chain.

You would need to obtain a larger backup drive that would be able to hold your complete backup version chain (Full + 10 differentials) plus a second Full backup, so probably around 4TB in size.

Hmmm...ok.. I'll have to merge a few partitions together then to make this happen.. but that is a very large "waste" of drive space given the full backup is over 1.4 TB!  There still should be an option to force override.... Novabackup does this and I've never had a problem. Thx for the info!

Yeah, I believe it's a vendor preference. Two lines of thought here... Acronis believes a full should be made before any deletion to ensure a recovery option as you could experience a failure or lack of recover option if deleted first... So, recovery option over drive space. Some other backup software allow to override this or delete first, but could end up with no receiver option as a result. O believe this is the safer method, but could be user preference based. Please submit feedback in app. If enough users feel the same way, it could be a standard feature down the road.

I have a similar a difficulty in understanding how this works. My settings, for a Differential scheme, are to "Create a Full after every 6 Differentials" plus "Delete version chains older than 7 days" The Help pages contain the following:

‘Delete version chains older than [defined period] [7days in my case]’ (available for incremental and differential methods only) - Select this option to limit the age of backup version chains. The oldest version chain will be deleted only when the most recent backup version of this chain is older than the specified period."

To me this raises the question: what determines the age of a 'version chain'? Assuming a version chain comprises (in my case) one Full plus 6 Differentials, is the date of the most recent VC equal to the date of its Full file or the date of its sixth Diff file?

The current status of my backup appears thus:

The 'Old' VC started 30th April and the 'most recent' one 8th May. Each day, the oldest Diff file is deleted and replaced by a fresh one. Taking the Help wording (above) literally I have to wait until the 8th May VC is 'older than [7] days' before the 30th April one is deleted. But at what date will the 8th May VC be deemed 'older than 7 days'? Will it be 16th May (one day after 8th plus 7) or, more likely, 22nd May (one day after the last of the sixth Diff file in the current run, plus 7 days)?

Obviously I shall watch the process daily to find out (!) but I too have capacity problems with my NAS capable of storing only two full VCs at a time and if the Acronis process tries to create a third Full file before it has deleted the oldest one the backup will fail.

I take Bobbo's point that Acronis have prioritised safety over space considerations but I need to know how I can limit the VCs to no more than TWO at a time!

I notice that in the 'Backup method' an alternative option to 'Delete version chains older than [7] days' exists in the form of 'Store no more than [ ] recent version chains' and I wonder if I would achieve my objective by simply switching to this option and setting it to 2 VCs? I am loathe to experiment with this yet because I want to see what happens with the current settings, but later..... ?

OK, I didn't realise the screenshot wouldn't show up!

Martesen,

The # of days in the cleanup is a bit misleading.  This does not mean that backups after that # of days just start getting cleaned up.  It means that once your entire chain is complete (in the case, 1 full and 6 differentials), then it will start the next version chain.  So, once the next full is complete, then, in 7 days it will cleanup the old version.

Basically, it allows for some overlap of version chains so that you can have more than one without any cleanup happening immediately.  My personal cleanup preference, is similar to Steve’s.  I believe that using the “keep no more than X” amount of version chains is the simpler and more efficient method.  If you tell it to keep 1 version chain, it will create that chain, create the next full and then delete the old chain. If you tell it to keep 2 version chains, then it will create the 3rd full and and then delete the 1st chain while retaining the 2nd chain.

Martensen,

From the screenshot, I can’t tell how many version chains you initially set either.  If you did set it to 2, then, so far it looks fine.  It would still need to complete the third full before any version chains get cleaned up in that case and then it would still be 7 days past that before cleanup takes place.

If you set it to keep only 1, but with 7 days, then the cleanup should be happening today or tomorrow as you’re about to be 7 days after the next full was last created.  May 5th + 7 days = May 12th and cleanup should take place the day after (day 8) which should take place on May 13th.

Bobbo - thanks for your input. At present, my settings remain as "Create a Full after every 6 Differentials" plus "Delete version chains older than 7 days" - I haven't stipulated a number of VCs using the 'Store no more than [ ] recent version chains' option. Although arguably, my settings should have the effect of holding no more than two VCs anyway?

The screenshot I posted shows the whole story insofar as the 30th April Full file is the oldest still retained. I am monitoring changes daily and will update the screenshot in a day or so's time, when the current VC will have finalised and the oldest one should be flushed out.

Scrutiny of the overnight changes has drawn my attention to the date AND TIME of each file and the (perhaps obvious in hindsight) realisation that to be deemed 'older than 7 days' (i.e. be at least 8 days old) it may be that a file has to not only pre-date 8 DAYS ago but it also has to be TIMED post the time of the earlier file! So just looking at the file DATES as I had been may not be enough?

I have also realised that the process of deleting the oldest VC does not as I had naively assumed comprise the simultaneous deletion of a Full plus all 6 of its subsequent Diff files, but actually occurs as the daily deletion of each Diff file, leaving the preceding Full and most recent Diff(s) intact until the last deletion. At least, I THINK that's what's happening!

Delete version chains older than 7 days is only counted from the date when a new Full backup file for the next version chain is created.

So, if your old version chain was started on 30th April and has 6 daily incremental files (created on days 1 thru' 6 of May), then the age of the old version chain starts counting after you create a new Full backup file for the next chain on 7th May, so will not be over 7 days of age, until the 14th or 15th of May, which is when a further new chain will be being created too.

The screenshot I posted shows the whole story insofar as the 30th April Full file is the oldest still retained. I am monitoring changes daily and will update the screenshot in a day or so's time, when the current VC will have finalised and the oldest one should be flushed out.

There's more to this as well.  April 30 is a full (b22_s1).  Then the next file is May 5 (b22_s5).  Where did B22 S2, S3 and S4 go? 

Were they manually moved or deleted?  If so, that can "break" a chain when expected files that were logged in the internal database have been modified outside of the application, since it still expects them to be there in order to complete the cleanup and retention requiremenets.  

You may want to run a "validation" on the backup job and then tell it to ignore each missing file when prompted (once for each file would be S2, S3 and S4 in this case) or tell it where they were moved if they still exist somewhere else.  Ultimately though, they are missing and that is likely the problem.

Otherwise, manually clean this version up with the "cleanup versions" option and tell it to clear out that old full and the remaining diffs that go with that chain.  After that, I would imagine the cleanup will run just fine for the full on May 8 and the remaining diffs in that chain, once the other criteria are all met... assuming there is no other interaction with the existing backup schedule or files outside of Acronis.  

Thx for the feedback everyone - I agree this should be an option - allow the user to determine if the entire chain should be deleted before or after a new chain gets created.  For me, I had to merge 2 partitions together and basically "waste" 1.5TB or space to keep just for the temporary chain ... kinda wasteful if you ask me!  Sure - if you delete the chain before creating the new one AND your main drive crashes and burns before the new chain is setup.. you are out of luck.  For me - this isn't a big deal because I keep another backup offline that is only a few weeks old.... so - offer the option and allow me to use 1.5TB of space please.

Please make sure to send feedback through the app with this suggestion as well - best way to get it to the developers for consideration - especially if more people do the same.

In the interim, the best we could do is manually use the "cleanup versions" to delete the entire existing backup chain first (or any older chain) to allow for more room.  Of course, this requires manual intervention since it's not an automated action at present, but it could be a work-a-round for some.  In the example below, I'd have to delete the entire 4/9 chain (the pic was from another thread where I was showing how you could clean up the tail end of an incremental, but not all that different in application).

Update: Bobbo asks (14th May): "There's more to this as well.  April 30 is a full (b22_s1).  Then the next file is May 5 (b22_s5).  Where did B22 S2, S3 and S4 go? "

My experience, observing this daily, has been that the program sequentially removes Diff files from a previous version chain as they age past the [7] days threshold, leaving the Full file in any VC until last. (See updated screenshot attached). This is logical since in theory, for Differential schemes at least, a VC is usable with just its Full and the most recent Diff file - intervening Diff files are superfluous (unless the most recent Diff file is somehow unusable.)

The screenshot also illustrates something I hadn't previously realised which is that (obvious in hindsight) the TIME on each new file is relevant - scrutiny of the date AND TIME stamps on the Diff files suggests that the answer to why TWO Diffs were deleted by the 14th May operation may lie in the times of the files – the 13th May files did NOT cause the deletion of the 5th May file because it was timed at 03:05, pre-date/timing the 5th May’s 03:11 (so 7 WHOLE 24 hour days had NOT elapsed) but the 14th May file, at 03:07 POST date/timed the 6th May’s 03:03 so the 14th May operation caused TWO earlier Diff files to be deleted.

The 15th May backup failed entirely because it tried to create the scheduled fresh Full file before anything earlier was removed - thus necessitating space for THREE simultaneous Full files, which the NAS did not have. I had to manually remove the oldest VC (using the approved process as illustrated above by Bobbo - thanks.)

In a further development (for me) I had an 'Instant Chat' with Acronis Customer Support and on their recommendation have switched my Clean-up rule to “Store no more than 1 recent version chain” as they say this is the only way to run my backups without greater storage space. The drawback of this of course is that I shall have no alternative backup if any current VC fails for some reason.

Martesen wrote:

...

My experience, observing this daily, has been that the program sequentially removes Diff files from a previous version chain as they age past the [7] days threshold, leaving the Full file in any VC until last. (See updated screenshot attached).

...

 

Do you guys confirm this? Something I was not aware of and would justify using differential backups all by itself.

The problem with clean up of incremental chains is that it is very lumpy. In tighter spaces where you'd use only one chain, when the new full is complete, the older chain is deleted and you are left with no history, forcing you to use more/shorter chains, but this comes with more fulls, etc.

My experience, observing this daily, has been that the program sequentially removes Diff files from a previous version chain as they age past the [7] days threshold, leaving the Full file in any VC until last. (See updated screenshot attached). This is logical since in theory, for Differential schemes at least, a VC is usable with just its Full and the most recent Diff file - intervening Diff files are superfluous (unless the most recent Diff file is somehow unusable.)

 

Martesen, I think you hit the nail on the head with the cleanup behavior.  I was under the impression the "Delete version chains older than X" would delete the entire chain since a chain is supposed to be the full and all backups created (inc or diff) associated with that full. 

I don't use the "older than" setting so have never really tested it.  However, with an incremental, that should mean that the entire chain would be deleted, including the original full.  Apparently, a differential chain is being handled differently in this case, but it's clearly not deleting the entire chain.  Either it's a bug with how this is being handled or the word chain needs to be adjusted or the behavior more clearly defined in the user guide.

https://www.acronis.com/en-us/support/documentation/ATI2019/#16143.html

• Delete version chains older than [defined period] (available for incremental and differential methods only) - Select this option to limit the age of backup version chains. The oldest version chain will be deleted only when the most recent backup version of this chain is older than the specified period.

With how Diffs are being handled with this setting, I am now seeing how this will cause recovery issues for others too.  There is a bug or bad behavior where differentials are removed from the "chain" - either in this automated cleanup, or the correct manual way from the GUI using the "cleanup versions" options.  As diffs are not related to any other diffs in the chain, this should be just fine.  However, other users have reported that when diffs are missing (say s2 and S3 were cleaned up correctly and they want to restore from S4) the rescue media warns that the chain is corrupt and the recovery may result in an unbootable version.  Would you be able to test this with your recovery media, but not actually continuing with the actual recovery - to see if you get this error / warning?

There was this older bug about differential chains that you describe, and I thought Acronis just did away with it by managing the clean ups of differentials like the cleanup of incrementals. Since they introduced manual cleanups, it could mean the cleanup of differentials are now behaving by clipping off intermediary differentials, I hadn't connected the dots and I didn't test it.

Pat L wrote:

There was this older bug about differential chains that you describe, and I thought Acronis just did away with it by managing the clean ups of differentials like the cleanup of incrementals. Since they introduced manual cleanups, it could mean the cleanup of differentials are now behaving by clipping off intermediary differentials, I hadn't connected the dots and I didn't test it.

Yup - the bug with diffs is still there - at least in the rescue media when it comes time to recover.  I tested this through and through (https://forum.acronis.com/comment/487291#comment-487291).  If the entire chain is there, no warnings or issues from the Windows GUI or rescue media - as long as they've been removed the correct way using automated cleanup rules or with the manual option on the backup task for "cleanup versions" in the GUI.

But, try to restore a later diff in a "chain" from rescue media where a proper cleanup has taken place of earlier diffs, and it says the chain is corrupt and may not boot, but that you can try anyway.  You then have to ignore the missing files and proceed.  In the end, the recovery worked and booted up just fine for me, but seeing that warning is not a happy feeling and it shouldn't be happening.  Instead of the diffs being completely independent of each other (as they are meant to be, and ultimately still function that way) the validation seems to be looking for the complete chain. So, the way the diffs are being cleaned up when using the "delete version chains older than X days", is likely going to cause rescue media restores of diffs to throw this scary warning too.

Another update - I've had an email exchange with an Acronis Customer Support person and pointed out that there appears to be no way of running a rolling two-extant-version-chain scheme which does not require capacity in the storage destination for THREE VCs at some stage in the back up cycle.

The point seems to have registered, insofar as the response I got, which included: "Unfortunately it is necessary to choose the 'Store no more than one 1 recent version' option if the NAS does not have enough storage space. I will consider your feedback and forward it to management and development team and make sure it is reviewed with utmost priority."

Your guess is as good as mine as to whether this achieves anything or not!

This conversation got me to try the clean up logic with differential backup. As we pointed out above, there were some issues historically and I always end up recommending choosing the "store no more than x recent..." options. The downside of the method is that it is very lumpy. After the full is done, an entire chain is deleted at least and all that history is lost. Also because of the previous bugs and the lumpiness of the "store no more..." there was no strong benefit of differential chains (aside from some marginal increased robustness, maybe).

I tested, I have to admit for the first time, a differential backup with auto-clean-up and the option of deleting older chains after a certain time period. It does work as expected.

I also tested restoring data from the cleaned-up backup with the Acronis Survival Kit set-up (which I had never used before), the recovery USB WinPE version and the Windows version, and everything worked just fine.

This is weirdly exciting as the product just works like it should

In the attached, I have a screen shot of the backup options, and screen shots of the backup directory and how it behaved after each backup so it is pretty clear which backup gets erased when.

Next, I am going to try a differential backup where the time used for retention is going to be shorter than the completion of the chain.

Good feedback and thanks for the screenshots too, Pat! When you're done with the next test, would you mind also using the cleanup versions option and removing one of the early differentials in a chain (say remove S2 through the GUI's cleanup option, but leave S3 and S4).  Then try to use recovery media to restore from S3 and see if you get a warning or not from it too?

I tried a different setting. Now we are looking at a differential backup where the time needed to complete the chain (3 days since daily backup) is longer than the age of the oldest backup to keep.

Aside from some minor deltas, the product behaves as intended. As the first chain keeps being completed, ATI starts erasing older intermediary differentials, preserving the desired history.

I have attached the illustration of the behavior with a copy of the backup settings and the contents of the backup directory over a week period.

From a UI perspective, one could have a gripe with the fact that what is deleted is a backup version (differential backup "slice") rather than a "version chain" in some cases.

Posted in wrong place

Pat L wrote:

This is weirdly exciting as the product just works like it should

It would probably be worthwhile running a similar test on ATI 2020.  I suspect only full chains will be deleted (even though a diff "chain" doesn't make much sense).