Skip to main content

vmProtect Forced sector-by-sector mode

Thread needs solution

I have an ESXi 5.1 w/ vmProtect 8. We are backing up around 10 VM and there are 4 running SBS2011 w/ Exchange aware.

The backup is success but it takes long time for some machine. It show:

Forced sector-by-sector mode.
Additional info:
--------------------
Error code: 46
Module: 7
LineInfo: a5695862aaf8e75e
Fields: PartitionId : 221
Message: Forced sector-by-sector mode.
--------------------
Error code: 24
Module: 7
LineInfo: ef8b1618a4c0dd9c
Fields:
Message: MFT bitmap is corrupted.
--------------------

Or

Forced sector-by-sector mode.
Additional info:
--------------------
Error code: 46
Module: 7
LineInfo: a5695862aaf8e75e
Fields: PartitionId : 225
Message: Forced sector-by-sector mode.
--------------------
Error code: 37
Module: 7
LineInfo: 89d94b01b483de63
Fields:
Message: Index is corrupted.
--------------------

and it take 7~8 hours for a single VM

I tried to run the chkdsk c: /r and no error was found. It will have faster backup speed for few days. But in other few days, another VM happen the same problem....

Is there any patch or solution I can do?

Please let me know if you need the log file from the server, I'm willing to provide~

Thanks.

it

0 Users found this helpful

Hello Jacky and Josh,

Thank you for posting.

This issue can have several causes and you can use the solution listed under each cause:

Cause 1:
First of all there may be a problem with the file system inconsistency inside the backed up VM, so Acronis vmProtect is unable to properly read from it and therefore starts backing up the machine in sector-by-sector mode.

Solution 1:
 In order to resolve this problem you should perform file system check via embedded Windows utilities. Please run CHKDSK utility on all the logical volumes inside the virtual machine:
- Go to the Command Prompt (Start -> Run -> cmd);
- Enter the command:

chkdsk DISK: /r

(where DISK is the drive letter of the partition you need to check. Please note that checking the C: drive may require you to reboot the machine)

Cause 2:
These messages may appear if Acronis vmProtect cannot recognize the virtual disk block structure of the virtual machine due to the specifics of the blocks size.

Solution 2:
Turn off the virtual machine and extend the virtual disk size (from vSphere client) by any value, for example, by 10 MB. This action will rebuild the virtual disk structure and fix the potential block size problem.

Cause 3:
These messages may appear in rare cases when the virtual machine is under high load during the backup. In these cases there may be a situation when there is data recorded in NTFS journal which is not yet flushed to the volume which file system therefore is recognized as corrupt since Acronis products are unable to read the NTFS journal properly and require that the data is flushed to disk in order to read from it. This situation forces the sector-by-sector backup mode. The same issue is applicable to Acronis vmProtect 7, but in this version the message about switching to sector-by-sector mode was hidden and was available in debug mode only. This problem does not happen each time and depends on the particular VM system state at the snapshot time. We have a plan to fix the problem in the future versions of Acronis vmProtect and other Acronis Backup and Recovery Virtual Edition.

Solution 3:
Until the fix is available you should ignore these messages. Note that in some cases files/folders recovery from such backups won’t be possible, however it will work in most cases even despite this message (it is a random behavior). You can still recover entire VM from these backups or run it from backup (Run VM from Backup feature) for granular recovery purposes.

Please let me know if you have additional questions.

Thank you.

Josh~ Not yet, I just receive the reply from Anton and will try. Let you know if I can solve the problem~

Hello everyone!

Based on Anton's response here are my results:

Solution #1: Chkdsk reported back 100% clean.
Solution #2: Performed the disk size increase on two different VM's. The first I increased by 10MB and the second 1GB. Problem is still occuring after the size increase.

In addition one of the VM's had the 100MB Windows System Reserved Partition. I removed this to see if it would be causing the problem. Backups still failed.

Thoughts?

Hello Josh,

Thank you for your post!

In case you can run the backup, it will be created in sector by sector mode. That is why solution #3 is more suitable. However if your backup fails, please create a new thread where we will be able to discuss the issue in more detail. After you create a new thread, please attach the following logs.

Please let me know if you have any additional questions!

I am getting this sporadically on some machines. In fact one night a machine may have this issue but the next night it might not. CHKDSK always reports the volume is healthy.

The issue is that when this happens, the backup can take many hours to complete. In some cases in upwards of 10 hours or more. We have been canceling backup jobs in the morning because there have been instances of 4 or 5 backups running at the same time and our VMWare infrastructure becoming so slow that we can barely interact with it or the vSphere client.

When this issue does not occur, a backup may take 30 minutes at most. Its amazing what difference in time this causes.

We are seeing this much more frequently since the update to ESX 4.1 Update 3 ('VMware ESX 4.1.0 build-800380'). Prior to this on ESX 4.1 Update 2, this did not happen nearly as often, and all backups were always completed - if not failed by the time we came into the office in the morning.

I would like to add an example.

On a normal night a backup of our Exchange server takes about 49 minutes.

If we get this error mentioned in this thread, one day it was going on 9 hours. People could not access e-mail. I could not even interact with the mail server through vSphere. Pings were 75% packet loss. The machine was running but heavily burdened. We had to cancel the job and move it back a few hours to try to mitigate this condition from impacting early morning business hours.

When our jobs fail it is for 2 reasons.
The most common reason is:
Message: Awaiting task 'CreateSnapshot' has failed. Reason: fault.Timedout.summary.

The second most common reason is us canceling the jobs because they are taking so long due to bit-by-bit backup.

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi everyone,

I want to add an important information to this thread: we have just found one of the primary reasons why the issue may occur. It's happening due to "Shadow Copy Optimization Writer" VSS writer which modifies the snapshot after creation. The details of the workaround are described in this article: http://kb.acronis.com/content/36993 (it's applicable if you confirm that there are no problems revealed by chkdsk of course).

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

HI,

We have this issue now after upgrading from v7....

We will be looking at v-ray next year.

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi chi-ltd,

In fact this issue was present in 7th version too, but it was hidden (the message about sector-by-sector backup mode was not put into the log file). Have you tried the workaround suggested in http://kb.acronis.com/content/36993 ? From what we could see it helps in quite a lot of cases.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

The sector-by-sector mode mainly affects the operations with the guest data which is located on the logical volumes inside the VM (this data not always can be read). Switching to this mode means that files/folders exclusions feature may not work (still it may work - depends on particular case) and the time to back up may be increased. The Exchange-aware backup may also be affected though we've seen cases when it does work even despite this message. This is due to the fact that Exchange-awareness collects Exchange-specific metadata through VMware Tools rather than by reading the guest FS.

Very unimpressed that we have to a) shutdown the VM (in many cases production) and then (if fails to fix issue) b) start hacking the VSS writer settings around.... especially when most of the feedback I now get from support on any open tickets is to upgrade from v7 to v8!

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi chi-ltd,

In fact shutting down the VM is not necessary here - it's required for chkdsk, however if you have symptoms described in 3rd resolution section then the workaround (disabling processing of Microsoft Shadow Copy Optimization VSS writer) does not require reboot or services downtime and can be implemented on fly (only restart of VMware Tools service is required).

The chkdsk with /F switch (which requires reboot) makes sense if running chkdsk in read-only mode (from command prompt which doesn't require reboot or downtime) did reveal some issues which cannot be fixed on fly without reboot.

Can you please also send me in private message the tickets numbers which you have opened with support so that I can review them and advise? Most likely upgrading to v. 8 would resolve the issues that you have (where applicable), but I want to ensure that you were not misleaded in any of the support cases you submitted. I believe that there are some things we missed which caused the overal frustration.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Same Problem here!

VM Ware ESXI Host 5.1 running a SBS 2011 Server with Exchange.

Backup sector by sector round abaout 6 hours.

Checkdisk found no errors.

create vmbackup.conf with Shadow Copy Optimization Writer same problem.

Update to VMProtect to Build V8 Build 8184 same problem.

Ticket: 01761157

taken the decision to not use acronis for exchange any more, back to BE to tape and disk now...
a reboot of the VM seemed to fix the problem.
i think v8 also runs a new full backup so may get quicker next time your backup runs..

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Dominic,

I'm unable to take a look at the ticket right now unfortunately (I will take a look at it later next week if my following suggestions do not help), but since you have an SBS system this means that there is also Active Directory service running there and I have some info which might be useful. The AD VSS writer ("Active Directory Domain Services (NTDS) VSS Writer") may also be writing data into the VSS snapshot, so it may cause the same issue as Shadow Copy Optimization Writer and it would make sense to disable this AD VSS writer in vmbackup.conf file to see if this fixes the sector-by-sector backup mode. Note also that the incremental backup speed would be much faster in any case and this sector-by-sector mode doesn't affect the Exchange-awareness since the Exchange meta-data is collected prior to detection of the sector-by-sector backup mode.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Hallo,

thanks for your help, i have tested it, but this is not the right solution.
BackupTask runs sector by sector.
My main problem ist/was that the backup time was round about 7 hours.
so i looked to the vcenter server and saw that the 2 vcpu (that come with the vmprotect vm) have 100% load.
I decided to test with more cpu power, so i have changed from 2 to 8 vcpu and run the complete backup again.
now i takes around 1 hours in sector by sector mode.

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Dominic,

Thank you for the valueable findings! The CPU is used specifically for compression and encryption purposes, so in your case it looks like (due to sector-by-sector mode) that the compression/encryption was the bottleneck. Typically the bottleneck is the write or read speed (access to virtual disks and backups repository), however in some relatively rare cases when you have quite good data transmission channels the CPU on the virtual appliance may become the root cause of the slow speed.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Hello everybody,

we are experiencing the same problem with Acronis Backup for VMware (Agent) which is set up as a virtual Appliance, v9.2.10535 (latest as of today, April 5th 2016).

We are running VMware 5.5 and our host runs Dell ESXi 5.5U2-2068190

The affected VM runs UBUNTU Server 14.04 and the daily incremental backup takes 9 HOURS!! It is successful but...9 hours for an incremental backup?

Here is the log that the Acronis Appliance produces everyday:

Forced sector-by-sector mode.
Additional info:
--------------------
Error code: 46
Module: 7
LineInfo: a5695862aaf8e775
Fields: PartitionId : 221
Message: Forced sector-by-sector mode.
--------------------
Error code: 22
Module: 7
LineInfo: 8241cadbfca7061c
Fields:
Message: Block bitmap is corrupted.

Solution 3 indicates that this issue would be resolved in the future versions of the Acronis Appliance, I know this is an old thread, but we are using the LATEST version now in 2016 and as you can see, the issue is still there, although this time it happens with a Linux machine.

I have already tried to add more CPUs to the Appliance (it had 2, I added 4 more and now it has 6) but the problem remains.

I will try the Solutions 1 and 2, chkdsk within Ubuntu and increase the virtual disk size to see if these help.

Will update with the results...

Tassos

 

 

 

 

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Tassos,

The message you quote is in fact unrelated to the actual problem which is too long 9h incremental backups. Most likely there is a problem with Changed Block Tracking (CBT) usage which should be also mentioned in the log file: "Failed to enabled Changed Block Tracking" or similar messages. I'd recommended first of all trying to reset CBT for this Ubuntu machine as decribed in https://kb.acronis.com/content/45682 . Note that CBT may also become invalid if you resize the virtual disks of the machine (in this case it will also need to be reset).

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager

Hello Vasily and thank you for your reply,

Currently we do NOT use CBT in the mentioned backup. Since we are not using it, is there a need to reset it?

 

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi,

If you're not using CBT then it pretty much explains why incremental backups take so long - you should definitely enable it for this VM. Are there any reasons why you don't want that? As far as I remember there is close to none impact onto VM performance due to CBT enabling and other than that there are no side effects.

Otherwise the backup needs to analyze entire disk and transfer all the changed sectors. If you're using LVMs inside this VM then _only_ sector-by-sector backup is possible in agent-less mode, since LVMs cannot be parsed by opening raw vmdks.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager