backup of hyper-v vm failing - acronis trying to backup non-existent snapshot?

Hi,

Most likely there are some non-existing links to the referenced virtual disk present in particular VM configuration. It could be some VM checkpoint leftovers or some disk which was physically deleted, but remained in VM configuration. You should contact our support team for assistance to review your current setup on the Hyper-V host and figure out the configuration problem with the specific VM.

Thank you.

This is a Hyper-V 2016 host and looking up details like this is a bit tricky in the new vmcx file. I'm working on a solution now to viewing this info. However, I can tell you there are no snapshots in Hyper-V manager for the VM. I can also tell you it's company policy not to make snapshots of domain controllers (this is a domain controller vm).

Looking in the snapshots folder for this vm I see an empty folder with a creation date that matches exactly to every failed attempt by acronis to backup the vm

Given that we didn’t create the snapshot that is apparently the problem and acronis did. How do we fix this? We have no functioning backups at this stage.

Hi,

On Hyper-V 2016 we use new backup API provided by Microsoft which relies on checkpoints creation (production checkpoints). To figure out what the problem is we need to check the diagnostic info from the host (see https://kb.acronis.com/content/58301 ) - it should be processed through our support team. Without the log files and reviewing the particular VM configuration it's hardly possible to guess what the problem is.

P.S. The temporary workaroud would be to install Agent for Windows inside the problematic VM and back it up using agent-based way instead of relying on Agent for Hyper-V (agentless backup).

Thank you.

I have this diagnostic report ready but am not comfortable uploading this kind of data to the forum, my last attempt to contact support was 2 days ago and still no reply. This forum is the only reply i get from Acronis. What’s next?

I don’t think our licence has an option to install the agent inside the vm, I’d rather not do that too

Hi Ben,

I'll send you a private message with the FTP link where you can upload the diagnostic data.

Thank you.

Hi Vasliy, I wasn't sure if you got my message but I have uploaded the items your requested late last night.  Please let me know if i missed anything

Hi Ben,

The diagnostic output which you've uploaded is from the management server machine (from Win2k12), while I'll need to check it from the Hyper-V host where the problematic VM (shown on the screen shot) resides. It can be collected via web console while selecting any Hyper-V VM from the list->Activities->Collect system information OR by running systeminfo.exe in C:\Program Files\Common Files\Acronis\AdvReport\ folder on the Hyper-V host - it will generate the outputs in the same folder. Can you please generate and upload this data for analysis?

Thank you.

Hi Vasily,

The diagnostic output I sent came from clicking on the vm that wasn't backing up, then clicking activities then collect system information. the first set of instruction from the first url you provided.

I've sent another set of diagnostics, is this what you are after?

Ben

Hi Ben,

Thank you for the provided details - it makes situation more clear. Quote from the log:

| error 0x53008b: The task has completed with errors.
| line: 0x7a272029c66c6578
| file: k:\7048\enterprise\virtualware\raw\microsoft\hyperv\hv_utils.cpp:244
| function: VirtualWareRaw::WaitForJob
| TaskDescription: Creating Virtual Machine Checkpoint
| ErrorCode: 0x8000
| ErrorDescription: Checkpoint operation for 'DC' failed. (Virtual machine ID F2EFF6F8-EA6B-4AF8-A230-FAD6A7A851C4)
| 
| 'DC' could not initiate a checkpoint operation: %%2147754996 (0x800423F4). (Virtual machine ID F2EFF6F8-EA6B-4AF8-A230-FAD6A7A851C4)
| $module: hv_srv_vsa64_7048

The bolded part is reported to Acronis Backup from the Hyper-V host itself. What happens is that production checkpoint (see also details on checkpoints concept) triggered for "DC" machine (which contains Windows Essentials) is failing. The symptoms are exactly the same as described in this thread: https://superuser.com/questions/1070464/cannot-create-checkpoint-on-hyper-v-0x800423f4

Plus check this thread: https://social.technet.microsoft.com/Forums/windowsserver/en-US/e8f602f6-44be-4558-a304-37f4ff4ca23a/creating-production-checkpoint-fails?forum=winserverhyperv

After that Acronis Backup tries to create crash-consistent checkpoint which seems to be successful, but apparently there are links to differencing disks which are no longer valid after application-consistent checkpoint failure, which causes the error you quoted originally.

Apparently if a VM is running a DC role then on Hyper-V 2016 (same on Windows 10 with Hyper-V) there are problems with creation of application-consistent (which rely on VSS) checkpoints. Or it's a problem related to fact that the VM was created off a physical machine with causes VSS failures (the 2nd thread I linked above).

The possible solutions for this particular VM include the following options:

• Edit backup plan and disable "Volume Shadow Copy (VSS) for Virtual Machines" option, which will enforce creation of crash-consistent checkpoints. It may help, unless the checkpoints failures also corrupted the resilient chang tracking (RCT) record for this machine on the host, which may still result in errors.
• Investigate the checkpoint failure with Microsoft support involved. The checkpoint creation and failure can be triggered right from Hyper-V manager to confirm that the issue can be reproduced outside Acronis.
• Install Agent for Windows inside this VM, so that it appears in the Devices list in Acronis Backup web console - it will appear in GUI twice with different icons therefore: 1 from inside agent and 1 reported from Agent for Hyper-V - for backup you'll need to select the one with agent inside (there is a tooltip if you mouse-over the icon)

Thank you.

Hi Vasily,

I can confirm both production checkpoints and standard checkpoints fail in our setup in both the hyper-v manager and acronis for backups.

Turning off VSS inside acronis does complete the backup (with warnings). how good is this backup?

Can you tell me where the reference to " k:\7048\enterprise\virtualware\raw\microsoft\hyperv\ " comes from, it's not the drive that has the problematic vm or set as any of the default locations for hyper-v, the K: drive has no 7048 folder?

Thanks

Ben

Hi Ben,

The backup captured with VSS disabled stores the system in crash-consistent state, which means that after recovery it will behave as if it was forcibly rebooted, so there is a possibility of inconsistency of data inside the restored system (for example some non-flushed disks transactions). Note that you can verify such backups by running VM from them (e.g. run a VM from backup) and checking the services inside.

The "k:\7048\enterprise\virtualware\raw\microsoft\hyperv\" is an internal Acronis path within the source codes - it should not be of concern. In the error stack quoted in my previous reply this part of code is responsible for returing the errors from the Hyper-V back into Acronis Backup interface, so the bolded text (about checkpoint creation failure) is coming from the hypervisor itself.

Thank you.

Hi Vasily,

So this hasn't lasted very long, manual backups appear to work but as soon as I turn the schedule back on backups fail with

Backup failed
Windows Error (0x800700005) Access is denied

Hi Ben,

The error is too generic to make conclusions - it's returned from the Windows itself when the backup agent tries to execute some function - it could be failure to access the backup location on network share, or failure to enable some service, or failure to execute specific process, etc. etc. Without checking the full log history to compare successful manual and failed scheduled attempts it's hard to guess anything I'm afraid.

Thank you.