Backups frequently failing with VSS errors, such as "deleting snapshot failed (0x8007045D)"
Hello,
I am an IT support provider in the Acronis Partner Program.
I have a test server (Dell T310 with SAS 6/ir RAID). I know it's an old machine, but it works for my purposes. I run Hyper-V and have several test systems in VMs on the server. I used to back it up using Windows Server Backup, which never failed, but it took like 14 hours a day to backup.
Since I have a few clients using Acronis Backup on their servers, I decided to look into the partner program, so I could set up my own Acronis backup system to experiment with and learn more about how to use Acronis products. Anyway, ever since I started using Acronis Backup for Virtual Host 12.5 on my server, I have been having various problems. I have worked with Acronis support in the past on one of the problems.
Here is the current situation: I am running Hyper-V Server 2016 Standard, with Server 2019 Standard, Server 2012 Standard, Windows 10 and Windows 7 as guests. Acronis for Virtual Host 12.5 is set up to back up each of the VMs, as well as the Hyper-V server itself. Sometimes, backups can run for several days without issue. It used to be just once or twice a week that a backup would fail with some sort of I/O error. Usually the error would be related to deleting snapshots. Sometimes, the error would have to do with merging process. Sometimes, it would case some sort of I/O error that would cause the whole RAID to go offline in the middle of the backup. I suspect that this particular error has now created some sort of corruption to where now I cannot backup at all, without getting the "deleting snapshot failed" error. I saw another article that suggested installing an Acronis agent in the VMs, which I am struggling with now. For some reason, it says "The system cannot find the file specified." The messages indicate that it can't find the backup location, even though it is clearly visible from File Explorer.
Getting frustrated with this, but I'm glad this is just happening on my test system and not on one of my clients' production servers!
Thanks, Rick
OK, Here is a log from one VM where it got the "deleting snapshot failed" error. The sysinfo file is 253 MB, so I'm not sure how I can upload that, since the message I get indicates there is an 8 MB limit.
Rick
Also, now, I have another VM that, when Acronis tries to back it up, it causes some kind of controller errors which eventually lock up the whole machine. This was the main problem I was having some time back (6 months to a year ago, maybe). I worked with Acronis support on this one for quite some time, then finally ended up deleting and recreating my RAID and restoring all the VMs from the Acronis backups. That seemed to fix everything for some time. Now Acronis is corrupting the RAID again.
I created a dump while the RAID errors were happening and the machine was starting to lock up.
Let me know how to proceed.
Hello Rick.
OK, Here is a log from one VM where it got the "deleting snapshot failed" error. The sysinfo file is 253 MB, so I'm not sure how I can upload that, since the message I get indicates there is an 8 MB limit.
The Input/Output error in the activity log you shared requires investigation of psc logs that are gathered with Acronis System Information. Since you have already collected it, please upload it to FTP link that I've sent you in a Private Message. Please see how to upload to Acronis FTP server https://kb.acronis.com/content/5991.
As for the lock-up issue you reported, I advise that you open a case with Acronis Support Team and share with the engineers the dumps you collected. The investigation of such kind issues requires a deeper level.
OK, the System Information file is uploading now.
Thanks,
Rick
Hello Rick.
The analysis of your logs showed disk errors:
Failed to read data from the disk.
Failed to read from sector '203,096' of hard disk '3'
Windows System Information shows LSI_SAS 11 and Disk 153 errors which indicate hardware problems with disk or controller.
You can check it when open Start->run-> issue eventvwr.msc -> open Windows logs -> System and filter to show warnings and errors.
Yeah, that's pretty much what you guys said the last time I tried investigating these errors with Acronis support. The thing is, the server runs perfectly fine and has run perfectly fine for years. These problems ONLY OCCUR when Acronis runs. As long as I don't run any Aconis backups, the server runs perfectly fine, 24/7. No other diagnostic utility shows any hardware problems or disk problems.
I guess maybe Acronis is just incompatible with this hardware and can't/won't be fixed. Maybe I should just go back to using Windows Server Backup, which still backs up the server without any errors. It just takes forever to run.
Hello Rick.
These errors are not Acronis incompatibility errors but Windows drivers' errors. You should show them to your system administrators who would understand their critical importance and potential danger for your disk (on the data on it) safety. You would not need any specific diagnostic utility for revealing them, simply open Start->run-> issue eventvwr.msc -> open Windows logs -> System and filter to show warnings and errors.
When Acronis software creates backups it reads the entire drive, that is the reason why we see these errors. Other software that you are referring to, deal with specific data sectors and do not see problems in neighbor blocks.
Thus, you should pay attention to these errors, investigate the cause (according to Windows System Information, it could be either disk or controller) and fix it until it becomes a reason of the data loss.
I am the system administrator and I do understand their importance. Again, I have never had any problems with this server until I started using Acronis Backup to back it up. The first time I went through this with Acronis support, they also insisted it was a driver error or hardware error. I believed them, so I ran every Dell diagnostic utility as well as other 3rd party utilities to try to find a hardware problem. I ran each of the RAID disks through a diagnostic utility the reads every sector on the disk, writes over it with different patterns, and the re-writes the original data, to verify the integrity of the disks. I finally deleted the RAID, re-created it, re-partitioned it, clean installed Windows Hyper-V server (going from 2012R2 to 2016 in the process), reloaded all my VMs and started over completely fresh.
Once again, the server was back to working perfectly for some time. Then I finally re-installed Acronis Backup and started backing up the server with Acronis again. And again, something happened while running Acronis backups that corrupted the VHD files and now I'm back where I started.
Now, it may well be that the drivers for the Dell SAS 6/ir controller are buggy and don't work right with Acronis. It may well be that there actually is some kind of hardware or firmware bug in the SAS 6/ir controller that causes it to not work right with Acronis. But, at the end of the day, this problem only occurs while running Acronis Backup to backup Hyper-V VMs. I know this hardware is old and no longer supported by Dell, so there is no hope of them providing newer firmware or drivers. It's down to either figuring out why Acronis causes this to happen, or giving up on running Acronis on this server. The errors we see now are the after affect of something that Acronis has done previously, while backing up the server, that caused the files to become damaged and no longer able to be backed up without I/O errors.
Think of it this way: Your hand is bleeding because you accidentally cut yourself. The bleeding is happening now, but the reason it is happening is because you previously cut yourself. The I/O errors I'm getting now are the bleeding. The reason for them is because Acronis previously caused some kind of corruption that I don't know how to fix. I know I can stop the bleeding by deleting and rebuilding my RAID and starting over fresh. But as soon as I run Acronis again, I will end up cutting myself again and the bleeding will start all over again. And you will tell me that my problem is that I am bleeding, but that won't explain why my hand keeps getting cut.
Hello Rick.
Thanks for sharing the details and your suggestions on this issue.
Looks like it requires a deeper investigation on the QA/Development level. However, I doubt if it will ever be tested with a quite old controller. Please also note that creating snapshots could be the too intensive workflow for this controller.
You will need to open a new case with Acronis Support Team. Please provide them with all the diagnostic information that you have along with a link to this forum thread.
As a temporary workaround, I suggest that you back up your Hyper-V VMs with Acronis Bootable Media.
OK, I will try the bootable media method. I was thinking about trying that. Hopefully that will work for the time being.
I know I need to upgrade my hardware to something newer, but this is just an internal server for my own testing and educational purposes. So it's hard to justify spending the money for a new server when this one still works sufficiently well for my purposes.
I will also open a case with the support team to see what they have to say. I guess I was just hoping someone could say something like "Oh, you just need to change this setting in Acronis Backup to prevent this error from happening."
Hello Rick.
Since Acronis Bootable Media is Linux-based, hope that a Linux driver for your controller will successfully accomplish creating a snapshot. Please share the outcome here.
I tried it with the bootable media I had previously created when I first installed Acronis. However, it was WinPE, not Linux. I was able to backup some of the VMs using the WinPE version. I will try creating a new bootable media with Linux, instead.