Skip to main content

failed to quiesce the virtual machine

Thread needs solution

Hello,
I have a problem with Acronis for VMWare where I cannot create a backup of one of my hosts. the error I get is below. there are other errors in the event log but the error is always failing to Quiesce the VM.

I have tried the following:
1. Take a snapshot of the VM with snapshot the VM's memory OFF and Quiesce guest file system ON. I get the same error: Unable to Quiesce the VM.
2. Uninstall VM tools and reinstall.
3. Rebooting the server

Failed to perform the requested operation.
Additional info:
--------------------
Error code: 13
Module: 149
LineInfo: d1ab7fa1e56eca4e
Fields:
Message: Failed to perform the requested operation.
--------------------
Error code: 103
Module: 83
LineInfo: a859dd78cc91df40
Fields:
Message: Failed to open the virtual machine ([VRTX Datastore] xxx/xxx.vmx).
--------------------
Error code: 253
Module: 83
LineInfo: c7610e0a857bedf4
Fields:
Message: VMware error: 'Remote method call failed.'.
--------------------
Error code: 32
Module: 0
LineInfo: c7610e0a857bedf4
Fields:
Message: Awaiting task 'CreateSnapshot' has failed. Reason: An error occurred while saving the snapshot: Failed to quiesce the virtual machine..
--------------------

When running vssadmin list writers from an elevated CMD window I get the following:

Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {32ca14a0-b03e-4c25-815b-de1fc83027c1}
State: [9] Failed
Last error: Timed out

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {3c577aa8-a70e-4f34-9ff9-4ef2ff5e031a}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {b4e4b609-a557-4998-aecc-d24b4f6e188b}
State: [1] Stable
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {c0f7ac39-e6b8-48ff-b00f-99ff950a77b9}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {df4fae11-f99e-4d54-a5dc-873696644329}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {e9407545-393b-4da9-89d3-c9beb6a58b01}
State: [9] Failed
Last error: Timed out

As you can see the System Writer and the WMI Writer both time out. I have tried restarting the VSS service and rebooting the server but this doesn't help.

Errors in the event log are mostly warnings:
[ warning] [vmsvc:vmbackup] Failed to send vmbackup event to the VMX: Unknown command.
MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system

Any help would be great.

0 Users found this helpful
frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi,

The proper way to troubleshoot such problems is to look at the event logs inside the guest OS of the VM and find possible VSS-related errors there. To confirm that the problem is unrelated to VMware you can perform system state backup using native Windows Server Backup tool (one of the Windows Server features) and check how it goes. If the system state backup is successful and VSS writers are Ok after this action, the next thing is to try creating quiesced (quiesced=on, memory=off parameters) snapshot of the VM and see if it still fails. If it fails then the problem is related to VMware.

As a workaround you can disable VSS processing of the VM by using either methods described in the following KB from VMware (set disk.EnableUUID=false or add parameters into tools.conf file in VMware Tools folder):

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cm…

Thank you.
--
Best regards,
Vasily
Acronis Virtualization Program Manager

Need to see if I can revive this thread, I've basically got the same thing going on as the original poster and I'm having trouble with a solution. I've got a 2012 server VM runing on a 6.0 ESXI host and I'm using vCenter. I'm running Backup for Vmware version 9.2 and when I try to backup the VM it fails with the same errors as B_ITF posted and I get an error in the event viewer of my VM with id 8193  (Volume Shadow Copy Service error: Unexpected error calling routine RegQueryValueExW({564d7761-7265-2056-5353-2050726f7669},,...).  hr = 0x80070002, The system cannot find the file specified.) The only thing of note this 2012 VM is running is Wordpress, I had read some errors encoutered with SQL and VSS but didn't know if there were any issues with MySQL?

I ran the troubleshooitng steps and when I try to take a snapshot of the VM with memory=off, it fails, it only works with memory on. I verfieid that VSS appears to work inside the VM, I can do a system state backup just fine and making a volume shadow copy works just fine. So is my only option this work around of disabling VSS processing?

BTW: Forgot to mention, this is a VM that I moved to this new Esxi 6 server using vcenter from an Esxi 5.5 host. Also, the only other VM on this 6.0 host is another 2012 server that is running my vCenter server and it backs up with Acronis and works with memory off snapshots just fine. So it's defitnly something going on with this particulary 2012 VM.

 

Thanks,

Andy

 

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Andy,

The error in your case is related to lack of VMware Snapshot Provider registration inside the guest OS: the {564d7761-7265-2056-5353-2050726f7669} GUID corresponds VMware Snapshot Provider (see references to this GUID here). This also explains why VSS works from inside the guest OS only, since in this case system state backup will use native Microsoft VSS provider instead of VMware Snapshot Provider used during quiesced snapshot.

Some users discovered that removing this GUID key reference from Windows registry helps: see the following thread, however what actually happens is that removal of the key effectively disables VSS processing which is done through VMware Snapshot Provider in case of quiesced snapshot. Therefore I'd recommended going through VMware Tools re-installation procedure, i.e. uninstall them, then reboot the VM, then install VMware Tools, making sure you select the VSS components (they are selected by default, but still worthy to double-check), then reboot the VM again to complete the installation -> next verify the presence of VMware Snapshot Provider under "vssadmin list providers" and if it's present then the quiesced snapshot should work without the error you referenced.

Thank you.

Thanks Vasily, I'll give this a go tonight.

Update: I removed the rogue GUID key reference from the registry fixed my issue, after delting it from the registry and rebooting, Acronis was able to back up the running VM again.

Thanks!

Let me add some to this thread.

I am subcontract I.T. for numerous clients.  I have been using Acronis VM-Backup for years.  Recently, the past year, I have ran into two of my clients that Acronis cannot backup.  I am working on the second one presently, and here is what I've found.

I increased the RAID storage on my client's Host and restored 3 VM-Ware Virtual machines from an Acronis Backup Archive.  All three VMs worked fine.  I then added a 1.4GB partition to the Domain Controller.   When I set up Acronis VM-Backup to backup the three VMs, the backup failed.  I separated each VM into it's own backup task and the two that didn't have the added partition backed up fine.  The one with the added partition still did not backup.   I then deleted the added partition from the Domain Controller and ran a backup task.  No problem, it backed up fine.  So I then added a 500GB partition and tried the backup task.  It failed this time. 

Now I'm having a hard time believing that an additional storage partition would cause this issue.  I have another customer that has two 250GB partitions added to their Domain Controller with no problems backing up.  The only difference is my present client is running Windows 2012 R2 and the client where the backup is working is running Windows 2008 R2.

Does anyone have any answers for me?

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Ed,

Assuming that the symptoms for your failed VMs are the same as described in this thread (quiesced snapshot fails), I'd first of all checked for VSS-related errors in Windows Event log inside the guest OS, to see if there are any specific errors reported there after trying to create quiesced snapshot (quiesced=on, memory=off) of this VM from vSphere client. Probably there are some additional VSS providers inside the affected VMs which are causing the problem. The point is that this problem would require thorough investigation by checking the logs from both guest OS and from ESXi host side.

P.S. The possible workaround would be to disable quiescing processing in VMware Tools tools.conf file as described in http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1028881 (also mentioned in my previous comment).

Thank you.

Thanks Vasily. 

I will make those changes tonight when I can power down the VMs.

I will then post the results for everyone.

Vasily.

I performed adding the disk.EnableUUID on both servers.  One server now backs up, the other doesn't.  I'm getting the following errors.  Do you have any more ideas?

Failed to perform the requested operation.
Additional info: 
--------------------
Error code: 13
Module: 149
LineInfo: d1ab7fa1e56eca4e
Fields: 
Message: Failed to perform the requested operation.
--------------------
Error code: 103
Module: 83
LineInfo: a859dd78cc91df40
Fields: 
Message: Failed to open the virtual machine ([datastore1] Windows 2012 R2 DC/Windows 2012 R2 DC.vmx).
--------------------
Error code: 253
Module: 83
LineInfo: c7610e0a857bedf4
Fields: 
Message: VMware error: 'Remote method call failed.'.
--------------------
Error code: 32
Module: 0
LineInfo: c7610e0a857bedf4
Fields: 
Message: Awaiting task 'CreateSnapshot' has failed. Reason: An error occurred while saving the snapshot: Failed to quiesce the virtual machine..
--------------------

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Ed,

What was the value set for disk.EnableUUID ? Have you set it to true or false? If it was properly set to false, then the machine has to be power cycled (power off/power on) in order to apply the changes. To verify that the changes do work you can simply capture snapshot manually from vSphere client (quiesced=on, memory=off) - if snapshot works, then backup would also work. If this doesn't help then it makes sense to contact our support team for assistance.

Thank you.

Vasily,

I have the property set for "true".   I changed the property with the VM powered off.  Then restarted it.

Does that sound correct?

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Ed,

Setting the value to "true" would enforce VSS usage inside the guest OS of the VM when quiesced snapshot of it is triggered from vSphere client -> this is the process which might fail (and fails quite often in vSphere, especially on highly loaded VMs). The more failproof approach is to set this vaule to "false" - in this case even quiesced snapshot will be capturing crash-consistent state of the VM without triggering VSS and thus is much more likely to work. The system restored from such snapshot will boot up and act like it was forcibly powered off.

Thank you.

====== Update =====

I'm posting this so anyone else who might be having this problem has a little more information.

The servers that I'm having trouble with are all Windows 2012 R2 servers.  I've used Acronis VMware Backup software for years on Windows 2008 Servers and 2012 Servers.   It seems that the problems I am having now are only with the Servers that have Additional Drives added to the Virtual Servers.  There were 3 servers that were having troubles.  One with a 250 GB System Drive and a 1TB Additional drive.   The other two with a 500GB System Drive and 2TB Additional drive.   

I first started by determining that the total drive space of a VM must be 2TB minus 512K.  So I decreased the sizes of the additional drives from 2TB to 1.4TB.  Acronis VMware Backup failed.   So I deleted the additional drives, and Acronis VMware Backup work perfectly.  I then added a 500GB additional drive, and Acronis failed again.

I then followed these instructions:

Application-consistent quiescing of Windows 2008 and later virtual machines is available when those virtual machines are created in vSphere 4.1, 5.x and 6.0.x. Virtual machines created in vSphere 4.0 can be updated to enable application-consistent quiescing by modifying a virtual machine’s EnableUUID attribute.

To enable Windows 2008 and later virtual machine application-consistent quiescing:

  1. Open the VMware vSphere Client, and log in to vCenter Server.
  2. Select Virtual Machines and Templates and click the Virtual Machines tab.
  3. Locate the Windows 2008 or later virtual machine for which you are enabling the disk UUID attribute, and power off the virtual machine.
  4. After power-off, right-click the virtual machine, and choose Edit Settings.
  5. Click the Options tab, and select the General entry in the settings column (under the Advanced entry).
  6. Click Configuration Parameters.
  7. Click Add Row.
  8. In the Name column, type disk.EnableUUID.
  9. In the Value column, type TRUE.
  10. Click OK and click Save.
  11. Power on the virtual machine.
  12.  

The Server with the 1TB drive sucessfully backup up using Acronis VMware Backup, but the other two servers did not backup.    My next step was to set the diskUUID attribute to "FALSE".   Both servers then started to backup, and have backed up successfully since..

I hope this helps someone else who might be experiencing the same problem.   If you benefit from it, please post a short summary of your issue would be appreciated.