[fixed] vSphere 6.0 critical CBT issue
Dear Acronis customers,
There was a major problem discovered by VMware which affects Changed Block Tracking (CBT) functionality on vSphere 6. It has been discovered and documented by VMware in KB2136854 .
This problem affects all VMs running on ESXi 6.0: the incremental backups of these VMs may not be recoverable. There are no viable workarounds for this problem except for either downgrading ESXi host down to 5.5 version or disabling CBT option in backup settings of Acronis Backup (Advanced) for VMware. Note that taking incremental backups will take longer with CBT option disabled, however the size of them will still be "incremental".
[Fixed]: VMware has released a patch with fix for this issue. See KB2136854 and KB2137546.
Thank you.
--
Best regards,
Vasily
Acronis Virtualization Program Manager
Andrew,
Thank you for the note - apparently you're right and any VM running on ESXi 6.0 is affected regardless of the virtual hardware version, so the only viable option is to disable CBT when doing backups.
P.S. Looks like the VMware KB article was changed a couple of days ago (on 17th of Nov) and probably I've misread the previous version while making this sticky post.
Thank you.
--
Best regards,
Vasily
Acronis Virtualization Program Manager
It looks like VMware has issued a patch to fix this issue:
Andrew, thank you for the note. I've corrected the original post and added link to respective KB.
Hi Vasily
Once the patch has been applied to ESX, can we then just re-enable CBT against the Acronis backups? Or does the ctkEnabled setting need to be disabled in the VM Properties in vSphere, a backup taken and then re-enabled?
Thanks
Jo
Hi Jo,
The VMware patch fixes wrong reporting of sectors range between 2 CBT records, therefore after applying the patch it should be enough to simply create a new full backup (into new archive for example). This will ensure that all following incremental backups will be correctly tracking the changes. There should be no need to re-enable or reset CBT from what I can see since it logically doesn't make sense (according to patch description) and plus there is no such recommendation from VMware.
Thank you.
--
Best regards,
Vasily
Acronis Virtualization Program Manager
Hi Vasily
Further to my question above, we have received the following advice from the Veeam forum:
VMware has released the patch for the latest vSphere 6.0 CBT issue. We have completed a few rounds of testing using the same lab setup that made the original issue easily reproducible, and the patch appears to resolve the issue when installed on ESX 6.0 Update 1a build 3073146. However, we found that simply installing the patch is not sufficient, and CBT reset is additionally required for all of your VMs. This is because existing CBT maps may already contain issues, which may result in inconsistent full backups in future. Having CBT reset will also make the following job run perform "full scan" incremental, thus fixing any existing inconsistencies in backups and replicas. Provided CBT reset has been performed, Active Full backup is not required. However, performing Active Full backups by itself cannot be considered as a substitute to CBT reset with this particular issue. I repeat, CBT reset is mandatory in every scenario.
Hi Jo,
I believe Veeam is referring to CBT* (so-called "CBT with star") functionality which depends on CBT and allows to retrieve a full list/map of occupied sectors for a given virtual disk and is used when capturing full backups. In Acronis implementation we don't use CBT* feature and we retrieve the list of occupied sectors by analyzing file system inside the virtual disk snapshot instead of relying on VMware API, so in our case it's not necessary to reset CBT before capturing new full backup (new full backup is still required as I mentioned in my previous comment). The subsequent incremental backups will be tracking changes between the 1st CBT record (appeared after capturing full backup) and newer CBT records which correspond VMware snapshots taken for new incremental backups.
P.S. To summarize: CBT* allows to retrieve list of all occupied sectors. CBT allows retrieve list of changed sectors between 2 snapshots (2 CBT records).
P.P.S. Resetting CBT for any machine won't do any harm, so it can be done to be 100% confident that the backups are ok.
Thank you.
--
Best regards,
Vasily
Acronis Virtualization Program Manager
Thank you for clarifying Vasily
Jo
Something still seems wrong, even after applying the patch. My backups now take much longer than before, and the "bytes transmitted" and "speed" counters are very high, as if it is doing a full backup each time. I've got it set to only keep 5 days of backups and CBT is enabled. I am seeing this issue on 2 separate appliances on 2 separate hosts. I have deleted all recovery points from my archives. Do I need to create an entirely new archive? I assume that with the appliance this means I will need to create and setup new local disks? (The appliances have a 2TB locally attached disk with is the RAID array in the host server).
Hi Andrew,
The symptoms you describe may look like CBT-related, so it's either a problem with specific VMs which have broken CBT which needs to be reset (see https://kb.acronis.com/content/45682 ) or a problem with the archive. Archive file should be re-created as I mentioned in my previous comment (you might need to pick up a new name for it). The backup logs should also show whether there are any issues with CBT - there must be a specific Information record about it.
Thank you.
--
Best regards,
Vasily
Acronis Virtualization Program Manager