Recovery to different ESX Host with different disk layout
Hi,
here is the scenario:
VM to be backed up has
1 vmdk with 120GB
1 vmdk with 1TB
3 vmdk with 2TB
In the case of a failure we would like to be able to recover this to a second ESX host, but this host has a different disk layout than the original ESX host:
1 datastore with 250GB
1 ds with 1TB
1 ds with 6TB
In theory there is sufficient space but when trying out to recover VMP tries to put all vmdks into the same DS. How do we tell VMP per vmdk which recovery datastore to use? Please don't tell me, that this is not possible! Also how do we recover a partial set of vmdks from this VM?
Cheers
Hi Vasily,
it is quite disappointing that a backup and recovery software such as VMProtect does not support correct recovery of a VM to different hardware with different disk layout. Matter of fact it renders this software almost useless in this case.
I will contact support to get hold of a ABR bootable image which supports incremental archives.
About vmdk's: Sorry for not being clear. What I wanted to know, is whether it is possible to recover certain (not all!) Windows drives only, for example D:, E: or whatever. I am talking about large drives which hold a lot of data up to 2TB! Downloading a zip from the web GUI, unzipping 2TB is not a practicable solution. Any workaround for this?
Also, has the recovery been tested in different environments? We tried the recovery with limited space. For the last 9 hours VMP has been trying to write to a full datastore, filling up the error log with
VMware_VDDK: [NFC ERROR] NfcFssrvrProcessErrorMsg: received diskLib error 40968 from server: NFC_DISKLIB_ERROR
Repeat failed IO operation 'Write'.
Additional info:
--------------------
Error code: 245
Module: 83
LineInfo: c61573f663f5d7fb
Fields:
Message: Repeat failed IO operation 'Write'.
--------------------
Error code: 208
Module: 83
LineInfo: c61573f663f5d77b
Fields: code : 1
Message: VMware_VDDK: Unknown error
--------------------
So .. instead of aborting the recovery process and throwing a real error, why does VMP continue to recover... nothing?
Hi Vasily,
it is quite disappointing that a backup and recovery software such as VMProtect does not support correct recovery of a VM to different hardware with different disk layout. Matter of fact it renders this software almost useless in this case.
I will contact support to get hold of a ABR bootable image which supports incremental archives.
About vmdk's: Sorry for not being clear. What I wanted to know, is whether it is possible to recover certain (not all!) Windows drives only, for example D:, E: or whatever. I am talking about large drives which hold a lot of data up to 2TB! Downloading a zip from the web GUI, unzipping 2TB is not a practicable solution. Any workaround for this?
Concerning the partial recovery - the best way is to use Run VM from Backup functionality to mount the VM and retrieve the files from there (from your quote I understood that you need to perform File Recovery using different method than extracting to .zip) by simply copying them from that mounted VM.
Concerning the "VMware_VDDK: [NFC ERROR] NfcFssrvrProcessErrorMsg: received diskLib error 40968 from server: NFC_DISKLIB_ERROR" - it is coming from the VMware itself when we try to write data to the virtual disk which is being recovered. The recovery is performed via NBD mode (over network) and thus consuming the NFC connections. In the current logics these errors are re-tried by vmProtect since they are considered to be retryable by VMware, but looks like in your case it's going to re-try loop. To avoid such error it would make sense to deploy the appliance to the target ESXi host and perform recovery using it - in this case the recovered virtual disks will be attached to that appliance and they will be accessed in hot-plug mode avoiding NFC connections usage.
Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager
Concerning the partial recovery - the best way is to use Run VM from Backup functionality to mount the VM and retrieve the files from there (from your quote I understood that you need to perform File Recovery using different method than extracting to .zip) by simply copying them from that mounted VM.
I came across this suggestion a few times on the forum, but I am having some doubts whether this will always work. So maybe you can answer a few questions which I have about this:
1. How do we copy the files off the VM?
2. Do we have to power on the machine? If so, how do we deal with the fact that this machine may be a server and has fixed addresses and may result in conflicting addresse in the network? How do we deal with issues such as that the server may change the files which we need to recover upon start of the machine?
3. If we mount the VM from backup, where do you store the changes which happen while the machine is running? Where does the VM write to the disk? Do you alter the backup archives?
To avoid such error it would make sense to deploy the appliance to the target ESXi host and perform recovery using it - in this case the recovered virtual disks will be attached to that appliance and they will be accessed in hot-plug mode avoiding NFC connections usage.
Thanks for the suggestion, we will try this. However there are a lot workaround a user has to keep in mind when doing a recovery. It's acceptable for a version 1.0, but I don't think this is good for a product which is labeled as version 8. In other words I am losing my trust in this backup software with each pitfall and limitation we come across. :(
Your support, however, is good. Thanks for the quick turnarounds!
Hi ece,
1) The files can be copied from the mounted VM over network from within the guest OS. To avoid network conflict you can start it with disconnected network and then modify the IP settings after guest OS boots up.
2) Upon boot up you can boot into safe mode for example - this should ensure that only system services will be running and thus no production databases are changed when you log into the VM.
3) The changes made to mounted are stored on the datastore which you define in Run VM from Backup wizard. Typically these changes are not too large and a few GB will be enough (unless you copy huge files _into_ the mounted VM of course).
The setup of multiple disks spread accross different datastores is not really common one and it is typical for large environments, which was one of the reasons why that functionality had not been included into vmProtect which is more SMB solution than large enterprise one. That's why originally this feature is available in ABR only. Another reason is that virtual disk mapping adds complexity to the recovery workflow and requires additional configuration from user who must know how to re-arrange his disks properly (in vmProtect we're trying to avoid complex solutions as much as possible). From vmProtect perspective we definitely look into this problem and we should be able to include an elegant solution in the future version (likely in vmProtect 10).
Overall all feedback we get from customers is treated quite seriously and we try to include as many fixes/improvements as we can. So once again I would like to thank you for sharing your experience (even not really good one) - with each new build/version you will definitely see the improvements.
Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager
1) The files can be copied from the mounted VM over network from within the guest OS. To avoid network conflict you can start it with disconnected network and then modify the IP settings after guest OS boots up.
2) Upon boot up you can boot into safe mode for example - this should ensure that only system services will be running and thus no production databases are changed when you log into the VM.
I think you agree with me, that this is quite a hassle if you need to recover files.
Plus you can easily imagine that things can break when going this way. For example: There is a Pop3 mail collector, which also runs in safe mode. As soon as the machine powers up and has network connectivity it will start fetching mails. This may mean that mails disappear. Or imagine this machine comes up to the network and offers it services to the network - duplicate machines on the network, possible DHCP issues. The more I think about it the more I hope that we never have to go this route, but we all now murphy's laws - tomorrow we may have to suffer from these issues *knocking on wood*.
These hassles are okay when using a workstation, then there is very little impact caused by a duplicate machine in the network, but obviously VMP will more use for servers. :(
The setup of multiple disks spread accross different datastores is not really common one and it is typical for large environments,
Actually I think this is a setup also quite common in SMB enviroment. SMB machines may be upgrade using more drives, resulting in different vmdk's spread across different drives and thus datastores.
Another reason is that virtual disk mapping adds complexity to the recovery workflow and requires additional configuration from user who must know how to re-arrange his disks properly (in vmProtect we're trying to avoid complex solutions as much as possible)
Surely this adds complexity. But you could easily implement it in such a way that the current behaviour (recover all vmdks on one datastore) is the default behaviour and whoever needs to adjust it, is able to change this.
Pro: VMP is really easy to use and very simple.
Con: VMP offers very little flexibility because it is too simple.
Currently these limitations will make things very complicated when trying to recover. So until these pitfalls have been resolved I think it is the best for us to use a different software. The current version number of VMP is 8, I don't expect VMP 10 to be available within the next year.
Is there any chance to change our license from VMP to ABR?
Hi ece,
Thank you for the post - it's quite helpful and definitely worthy to consider. Concerning your last question migration from vmProtect to ABR should be possible, though I'm not sure about the process from sales perspective. I'd recommended to contact our support team who can point you into right direction from here.
Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager
Hi ece,
I've just made some additional research and we seem to have found more elegant solution for such configurations: prior to running the recovery of a complex VM there should be a blank VM with required disk configuration (empty disks) created on the target host (where you want to recover the VM to). The VM must be created with the same name as the one you want to restore. After that in the recovery wizard you can select this target host and get the name conflict detection dialogue (see the attached picture). If you choose to "Overwrite" then the volumes from the backup archive will be automatically spread among the disks of the blank VM you created.
That's how I can see to make it work in fast (within a couple of months, i.e. in vmProtect 9 rather than in vmProtect 10). There is currently a problem with this process however: currently the name conflict detection dialogue logics is quite complicated and it relies not only on VM names, but also on multiple other parameters (instanceUUID, UUID, etc.) which was done originally for better name conflicts detection. That's why if you simply create a blank VM with the same name and try to recover to this host with vmProtect 8 then you won't get the dialogue and the VM won't be overwritten.
To resolve that issue I've submitted a request to our developers to simplify the name conflict detection algorithm so that it relies only on VM names. This means that sometimes this warning will be thrown even though it should not, but the benefits from this change are much greater since it will allow us to resolve the problem raised in this thread.
Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager
That sounds good! Thanks for this possible workaround.
To get around the issue with the UUID, I guess the easiest way would be to force such a name collision by using the vmx files and copying them (or editing them in such a way that the correct datastores are referenced) manually. Correct?
Now if you can find a elegant solution to "solve" the file recovery issue - I am a happy camper :)
(BTW
Concerning the "VMware_VDDK: [NFC ERROR] NfcFssrvrProcessErrorMsg: received diskLib error 40968 from server: NFC_DISKLIB_ERROR" - it is coming from the VMware itself when we try to write data to the virtual disk which is being recovered. The recovery is performed via NBD mode (over network) and thus consuming the NFC connections. In the current logics these errors are re-tried by vmProtect since they are considered to be retryable by VMware, but looks like in your case it's going to re-try loop
You are right. It is stuck in a loop, at least the progress has not changed for the last hours. Time to abort this manually.)
That sounds good! Thanks for this possible workaround.
To get around the issue with the UUID, I guess the easiest way would be to force such a name collision by using the vmx files and copying them (or editing them in such a way that the correct datastores are referenced) manually. Correct?
Now if you can find a elegant solution to "solve" the file recovery issue - I am a happy camper :)
The checking for names is done by comparing the InstanceUUID value of the original VM (recorded in the backup archive) and InstanceUUID + Name of the VM on the target host. If name+InstanceUUID match then there is such warning thrown out which allows you to overwrite the VM. The InstanceUUID can be changed via Object Browser as described in the following KB article: http://kb.acronis.com/content/32243 (solution C).
In vmProtect 9 we're currently considering several options to cover this scenario (recovery to existing VM or allow datastores configuration) so one of them should be implemented relatively soon.
Concerning File Recovery I just got an idea of an alternative solution: still it is using mounted VM concept but ensures that there are no name/network conflicts. The mounted VM should be started without network connection (do not connect it to network) and after mounting is finished you can attach a new virtual disk to this VM and initialize it in Windows Disk Management. This disk can be used as repository for the files you want to extract from the mounted VM. After you copy all that you need - you detach the disk and it can be attached to any other permanent VM in environment so that you can view the files. Hope it helps!
Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager
A little correction to my previous post. Not only InstanceUUID + name values must match, but also UUID value is important (must match) to receive the overwrite prompt.
Bad news ;(
Another update: even after updating the UUID+InstanceUUID values the recovery didn't work for me with vmProtect 8. What happens is that it doesn't overwrite the target VM (which I freshly created), but instead removes the empty disks and create new ones - all allocated to one datastore :( . This happens when logics detects even slight mismatch in sectors alignment. In other words overwrite will occur only if recovery is going to exactly the same machine.
In other words the only workaround is the one I described in the beginning of this thread (using ABR), and we'll need to wait until the normal solution.
Bad news indeed. However, thanks for following up, this saves time for me, because I was about to try it out.
Hi Vasily,
Still there is a workaround for such cases: you can use ABR bootable media which can read Acronis vmProtect backups and perform the recovery by booting the new VM (created with empty disks of necessary sizes) with this media and mapping the volumes onto such disks. I'd recommended to contact Acronis support to get such media, since by default ABR cannot read always-incremental archive format from vmProtect and this functionality had been added just recently (not yet released in public).
could you kindly confirm that the build #37687 supports the new archive format? Your support was not able to confirm this.
Cheers,
ece
Hi ece,
The 37687 build of ABR does not support new archive format. I will send you the details in a private message shortly with further instructions.
Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager