How do cleanup jobs work?
I'm trying to set up Acronis Backup Advanced on 4 Windows 2012 servers. One of these servers is used as the central management console, this one also has a tape drive attached to it but right now I'm still experimenting with disk backups.
I was trying to test how a cleanup job would work but i actually can't get it to work at all. I created a backup for one of our servers as a test and then added a (separate) cleanup job with a retention period of 1 hour. The backup job itself had a higher retention period (1 week) but I was expecting that a cleanup job with a different retention period would overrule this.
The first problem was the agent I chose to do the cleanup job. It seemed logical to me to delegate this task to the agent on the same server as the storage node and the disk-based backup folder that i would like to clean up. However if i choose this agent and execute the job it will fail in about 90% of cases with this result:
{
"owner": {
"activityId": "2FC53454-0124-4041-A146-A1944D7D854F"
},
"kb_link": "https://kb.acronis.com/errorcode?ser=0x01350016%2B0x01350016%2B0x0072808A&linetag=0x8D165E86FB81959B%2B0x8D165E86FB81959B%2B0x010851EC94F3C45A&os=Windows&product=Backup&version=12.5&build=7048&language=de-DE",
"origin": {
"module": 309,
"code": 22,
"fields": {
"$module": "service_process_vsa64_7048",
"CommandID": "9696AF0B-B4B6-4067-8F7A-355FE20DBF31"
},
"text": "TOL: Failed to execute the command. Backups bereinigen",
"linetag": "0x8D165E86FB81959B",
"suberror": {
"module": 309,
"code": 22,
"fields": {
"$module": "staging_command_vsa64_7048",
"CommandID": "9696AF0B-B4B6-4067-8F7A-355FE20DBF31"
},
"text": "TOL: Failed to execute the command. Backups bereinigen",
"linetag": "0x8D165E86FB81959B",
"suberror": {
"module": 114,
"code": 32906,
"fields": {
"$module": "staging_command_vsa64_7048"
},
"text": "Die Aktion mit einem oder mehreren Backups ist fehlgeschlagen.",
"linetag": "0x010851EC94F3C45A",
"suberror": null
}
}
},
"reason": "internalError",
"date": "2017-07-12T11:15:59+00:00",
"serCode": "0x01350016+0x01350016+0x0072808A",
"context": {
"_internal": "0:0:-1",
"operation": "unknown",
"cause_str": "Die Aktion mit einem oder mehreren Backups ist fehlgeschlagen.",
"effect_str": "TOL: Failed to execute the command. Backups bereinigen"
},
"effect": "TOL: Failed to execute the command. Backups bereinigen",
"cause": "Die Aktion mit einem oder mehreren Backups ist fehlgeschlagen.",
"troubleshoot": null
}
...the other 10% of cases it will complete successfully with an empty result and without actually doing anything.
If i choose any other agent (on another machine!) the job works (meaning it completes successfully) every time, but the result remains the same, no backups get deleted, nothing changes inside the backup folder and the backups are still accessible through Acronis.
I have read on other threads that the protocol for the activity should state what happened exactly...but whenever a cleanup job fails and i try to download the protocol i get an empty zip file. If the job completes successfully but doesn't delete anything, i'm able to download a protocol which contains exactly 2 lines...the first line states that the job was started and the second line states that it was completed successfully...
So, i'm thinking that the reason no backups get deleted must be because the retention period of the backups as defined in the backup plan is still active. However, changing the retention period in the backup plan to 1 hour did not help, so the retention period must be saved with the backup i guess.
In any case I was testing this because I already expected to run into this problem: https://kb.acronis.com/content/59728
... and this brings me to my actual question (although any inputs on problems or questions mentioned above would be appreciated): How is it possible to prevent Acronis from using too much disk space? The linked article seems to suggest that there is no way to clear up disk space once Acronis has claimed it, at least not without destroying the catalog. There's no option for disk based storage to retain at least "X" amount of free space and if cleanups don't delete data then there's no way to automate it even if the backups can somehow be (thoroughly) deleted manually.
I did tests with deduplicated storage as well but the results there were even worse because even manually deleted backups left 99% of the data untouched after deletion.
I'm sure I'm just missing some important details, I hope somebody can point me in the right direction :)
Hi Vasily,
I did understand that the settings count for different sets, which is why i set the retention of the cleanup job to just one hour with "hourly" being the only set used:
The target is defined as a storage node:
Also, I did create some additional backups yesterday and made sure to set the retention of the backup plan to 1 hour as well. Right now all my backup plans point to the same storage node:
The retention rules defined in the backup plans work, but the cleanup job does not. Creating an additional backup set older backups to "marked for deletion", but the cleanup job did not set the additional backup I created yesterday (with 1 hour retention) to "marked for deletion" nor did it delete the other backups that were "marked for deletion".
And I also understand that v12 format will reuse the space that it has "freed up" but i can't help but see this as a big potential future problem. What if one backup suddenly has twice the size it had before for some reason? Maybe there are certain conditions that could create a loop while reading a file which would blow up the backup size indefinitely...what happens in that situation? It seems to me that a single backup causing problems could blow up and destroy a catalog completely...meaning it would also render all other backups unusable. Also if the storage node points to anything other than a partition/disk that is dedicated solely for backups it would grind other programs/services to a halt if they are using the same disk.
Even if the disk filling up would not destroy other backups, how do you fix it when you are at that point? Deleting files manually seems to make things worse in most cases and i'm not sure if Acronis would even still be able to "do stuff" on that harddrive if it is completely full.
Hi,
Thank you for the clarifications, - definitely the issue with the cleanup is related to backups location. Operations with backups stored in location managed by Storage Node are very different from the ones located on plain network shares or local folders. I have forwarded this problem to our QA team for investigation, as I also failed to make it to work: the separate cleanup plans vs managed locations - this is the combination which doesn't seem to work (cleanup plan runs with success, but all expired recovery points were not removed).
Note that the issue with the archive growth is not applicable to backups stored on Storage Node - the clean up worked differently there because Storage Node does not support v12 backup format and always enforces backups to be saved in v11 format where cleanup is performed either by backup chains deletion or by consolidation.
The concern about accidental v12 backup growth is valid and in Update 2 (planned for Sep-Oct this year) we plan to add a feature to "shrink" such single-file archives to reduce the physical size of the archive if there is "free" space inside it (internal feature ID: ABR-86450). This shrink will be performed automatically after retention rules are applied and you'll be able to run it manually as well.
Thank you.
Update: clean up via separate task VS managed location works properly if the retention rules are set to clean up "By number of backups". So what doesn't work is clean up "By backup age" when archive is located in managed location.
Hi Vasily,
Thanks a lot for the help. We'll just use the retention rules defined in the backup plans themselves instead of using a dedicated job for cleanups.
It's good to know that there will be an option to shrink the backup file size for v12 backups, we might still go with v11 backups without deduplication for now because those backups can be managed more easily.
With that I should have all the information I need to finish setting up our backup strategy. I'm starting to gain more confidence again that we chose the right product :)
I greatly appreciate the efficiency and transparency with which you handle support requests here, thanks a lot Vasily!