Skip to main content

Backup fails with possible High I/O error

Thread needs solution

Hi all.

I have multiple clients where all of a sudden the backups start failing.
Backup plan includes Entire machine + AD.
For some reason I get a High I/O timeout error, even though there is nothing special running on the machine.
I have set the VSS registry timeouts for Windows 2012 R2, but that didn't resolve anything.
Sometimes, about once a week, the backup actually succeeds. I'm a little bit lost, so hope anyone here might be able to help me out.

Event 0x900001: Error code: 14
| Task 'Incremental backup' execution failed: TOL: Failed to execute the command. Backing up
| Additional info: 
| --------------------
| Error code: 22
| Module: 309
| LineInfo: 8d165e86fb81959b, e:\219\enterprise\common\tol\command\command.cpp, Tol::`anonymous-namespace'::MakeFailResult, 461
| Fields:  CommandID : 8F01AC13-F59E-4851-9204-DE1FD77E36B4, $module : service_process_vsa64_10170
| Message: TOL: Failed to execute the command. Backing up
| --------------------
| Error code: 22
| Module: 309
| LineInfo: 8d165e86fb81959b, e:\219\enterprise\common\tol\command\command.cpp, Tol::`anonymous-namespace'::MakeFailResult, 461
| Fields:  CommandID : 8F01AC13-F59E-4851-9204-DE1FD77E36B4, $module : gtob_backup_command_addon_vsa64_10170
| Message: TOL: Failed to execute the command. Backing up
| --------------------
| Error code: 252
| Module: 161
| LineInfo: 9fc3ee39511c2bd9, e:\219\enterprise\mms\managers\archive\utils\uri_utils.cpp, ArchiveManagement::AddStreamNameToErrForOnlineBackup, 606
| Fields:  account : xxx, $module : disk_bundle_vsa64_10170
| Message: Stream name: 'xxx.tibx'.
| --------------------
| Error code: 3
| Module: 329
| LineInfo: 1cd98aae889424f9, e:\219\enterprise\managers\gtob\util\impl\convert_batch_result.cpp, Gtob::Backup::ConvertBatchResult, 58
| Fields:  $module : disk_bundle_vsa64_10170
| Message: Backup has failed.
| --------------------
| Error code: 1060
| Module: 1
| LineInfo: b43e776571144df0, e:\219\processor\diskadm\da_operation.cpp, DaProcessor::OperationImpl::Execute, 204
| Fields:  $module : disk_bundle_vsa64_10170
| Message: Failed to commit operations.
| --------------------
| Error code: 33
| Module: 7
| LineInfo: 8ba4fa0bac28be9d, e:\219\core\resizer\archive3\backup_partition.cpp, `anonymous-namespace'::PartitionBackuper::Init, 330
| Fields:  $module : disk_bundle_vsa64_10170
| Message: Failed to create volume snapshot.
| --------------------
| Error code: 50243
| Module: 16
| LineInfo: 3fec04e376b8a16e, e:\219\core\fdisk\win_snapshot.cpp, win_snapshot_core::SnapStart, 1144
| Fields:  $module : disk_bundle_vsa64_10170
| Message: Failed to unlock the volume snapshot. To troubleshoot VSS issues, download and run Acronis VSS Doctor available at https://www.acronis.com/en-us/personal/vss-diagnostic-free-tool/
| --------------------
| Error code: 50306
| Module: 16
| LineInfo: 3fec04e376b89daa, e:\219\core\fdisk\win_snapshot.cpp, `anonymous-namespace'::ObtainVSSFailReasonSuberror, 180
| Fields:  $module : disk_bundle_vsa64_10170
| Message: VSS writer 'System Writer' with class ID 'E8132975-6F93-4464-A53E-1050253AE220' has timed out. This may be caused by too intensive I/O on the machine at the backup start time. Try to re-schedule the backup to some different time. To troubleshoot VSS issues, download and run Acronis VSS Doctor available at https://www.acronis.com/en-us/personal/vss-diagnostic-free-tool/
| --------------------
| Error code: 65520
| Module: 0
| LineInfo: bd28fdbd64edb8fb, e:\219\core\common\error.cpp, Common::Error::AddHResult, 317
| Fields:  code : 2147754994, $module : disk_bundle_vsa64_10170
| Message: Unknown Win32 error
| --------------------
| trace level: error

0 Users found this helpful
frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 250
Comments: 7092

Hi! 
Have you already applied Acronis VSS Doctor? Did the tool reveal any issues with VSS? https://www.acronis.com/en-us/personal/vss-diagnostic-free-tool/

Everything is green when I run the tool.
Then again last night's backup ran without problems (typical).
The server has not had a reboot or any changes since the failed backup 2 days ago.
Will run the tool again when I receive a failed backup message.

 

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 250
Comments: 7092

Ok, please keep us posted. Hope the things continue to be normal for you. 

Backup failed again with the message possible High I/O.
VSSDoctor results in all VSS writers timed out.
I have already tried to re-register the VSS writers before, but that didn't help unfortunately.
Can try it again of course, maybe run it every time before the backup starts?

[code]
cd /d %windir%\system32
net stop vss
net stop swprv
regsvr32 /s ole32.dll
regsvr32 /s oleaut32.dll
regsvr32 /s vss_ps.dll
vssvc /register
regsvr32 /s /i swprv.dll
regsvr32 /s /i eventcls.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s vssui.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml4.dll
vssvc /register
net start swprv
net start vss
[/code]
 

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 250
Comments: 7092

If you still see the message "VSS writer 'System Writer' has timed out", one of the possible solutions would be setting the VSS timeout value by editing the following registry key: 

HKLM\Software\Microsoft\Windows NT\CurrentVersion\SPP\CreateTimeout. 

Change value to 12000000 (2*10*60*1000 = 20 mins). 

The default value is 10 mins, so doubling it should give enough time to complete the snapshot.  If the key isn't present, you will need to manually create a DWORD 32-bit key "CreateTimeout". 

If still no joy, I'd recommend checking the Windows event logs for more information regarding the VSS issue.

Go to Start -> Run -> eventvwr.msc. Navigate to Windows Logs -> Application and Windows Logs -> System 

Check for error messages related to VSS service. Search for solution in Microsoft Support articles, for example see this Microsoft TechNet Article.

 

The settings I have done a long time ago:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\SPP\CreateTimeout --> 12000000
HKLM\System\CurrentControlSet\Services\VSS\Settings\IdleTimeout --> 12000000
HKLM\System\CurrentControlSet\Services\Volsnap\MinDiffAreaFileSize --> 2000

Also changed the VSSAccountControl to make sure the Acronis account has access.
No luck so far.
VSS writer results in the most common error 0x800423f2 --> the writer's timeout expired
There is a timeout issue, I just can't find how to fix it :(

 

After a successful backup on Tuesday evening, another fail on Wednesday evening.
No reboots, no changes. The VSS writers have a mind of their own.

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 0
Comments: 82

Hello, 

could you please also check if shadow copy storage has enough free space? You need to make sure that the volume where the snapshot storage is created has enough free space. 

I hope this could help you. If you have any further questions, please do not hesitate to contact us. 

 

We have exactly the same problem on Server Essentials 2016. Backup was working for months now without any issues, no changes made. Did a clean reboot, free space is sufficient, timeout registry entries were made on initial setup and confirmed that they are still present. Now since several days no backup completes (approx. 24.08.), neither scheduled nor manual.  Tried a manual Windows Backup, which works.

I have no further idea on how to solve this. VSS Diag is all green beside the failed writers after a backup Job and according Event log entries.

 

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 0
Comments: 82

ITSFE , hello

welcome to Acronis forum!

I would suggest you create a ticket with our support so that my colleagues could thoroughly analyze the logs and investigate the issue. 

If you have any questions do not hesitate to contact me. 

Thank you!

 

I have exact the same problem on a new installed Server Essentials 2016. There is nothing special on this machine.

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 250
Comments: 7092

Hello Kreuzritter,

have you tried to apply the above suggested steps (Acronis VSS Doctor and increasing timeouts)? The status of VSS writers can be checked using instructions from https://kb.acronis.com/content/36200#1_4_VSS_writers_should_be_enabled_and_in_consistent_state7 

I have tried everything. I have already reinstalled the server three times. The official support has told me that the problem is known and they are working on a solution. This was definitely the last time I bought Acronis. 

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 250
Comments: 7092

Would you mind sharing the ID of the related support ticket?