Backup fails with possible High I/O error
Hi all.
I have multiple clients where all of a sudden the backups start failing.
Backup plan includes Entire machine + AD.
For some reason I get a High I/O timeout error, even though there is nothing special running on the machine.
I have set the VSS registry timeouts for Windows 2012 R2, but that didn't resolve anything.
Sometimes, about once a week, the backup actually succeeds. I'm a little bit lost, so hope anyone here might be able to help me out.
Event 0x900001: Error code: 14
| Task 'Incremental backup' execution failed: TOL: Failed to execute the command. Backing up
| Additional info:
| --------------------
| Error code: 22
| Module: 309
| LineInfo: 8d165e86fb81959b, e:\219\enterprise\common\tol\command\command.cpp, Tol::`anonymous-namespace'::MakeFailResult, 461
| Fields: CommandID : 8F01AC13-F59E-4851-9204-DE1FD77E36B4, $module : service_process_vsa64_10170
| Message: TOL: Failed to execute the command. Backing up
| --------------------
| Error code: 22
| Module: 309
| LineInfo: 8d165e86fb81959b, e:\219\enterprise\common\tol\command\command.cpp, Tol::`anonymous-namespace'::MakeFailResult, 461
| Fields: CommandID : 8F01AC13-F59E-4851-9204-DE1FD77E36B4, $module : gtob_backup_command_addon_vsa64_10170
| Message: TOL: Failed to execute the command. Backing up
| --------------------
| Error code: 252
| Module: 161
| LineInfo: 9fc3ee39511c2bd9, e:\219\enterprise\mms\managers\archive\utils\uri_utils.cpp, ArchiveManagement::AddStreamNameToErrForOnlineBackup, 606
| Fields: account : xxx, $module : disk_bundle_vsa64_10170
| Message: Stream name: 'xxx.tibx'.
| --------------------
| Error code: 3
| Module: 329
| LineInfo: 1cd98aae889424f9, e:\219\enterprise\managers\gtob\util\impl\convert_batch_result.cpp, Gtob::Backup::ConvertBatchResult, 58
| Fields: $module : disk_bundle_vsa64_10170
| Message: Backup has failed.
| --------------------
| Error code: 1060
| Module: 1
| LineInfo: b43e776571144df0, e:\219\processor\diskadm\da_operation.cpp, DaProcessor::OperationImpl::Execute, 204
| Fields: $module : disk_bundle_vsa64_10170
| Message: Failed to commit operations.
| --------------------
| Error code: 33
| Module: 7
| LineInfo: 8ba4fa0bac28be9d, e:\219\core\resizer\archive3\backup_partition.cpp, `anonymous-namespace'::PartitionBackuper::Init, 330
| Fields: $module : disk_bundle_vsa64_10170
| Message: Failed to create volume snapshot.
| --------------------
| Error code: 50243
| Module: 16
| LineInfo: 3fec04e376b8a16e, e:\219\core\fdisk\win_snapshot.cpp, win_snapshot_core::SnapStart, 1144
| Fields: $module : disk_bundle_vsa64_10170
| Message: Failed to unlock the volume snapshot. To troubleshoot VSS issues, download and run Acronis VSS Doctor available at https://www.acronis.com/en-us/personal/vss-diagnostic-free-tool/
| --------------------
| Error code: 50306
| Module: 16
| LineInfo: 3fec04e376b89daa, e:\219\core\fdisk\win_snapshot.cpp, `anonymous-namespace'::ObtainVSSFailReasonSuberror, 180
| Fields: $module : disk_bundle_vsa64_10170
| Message: VSS writer 'System Writer' with class ID 'E8132975-6F93-4464-A53E-1050253AE220' has timed out. This may be caused by too intensive I/O on the machine at the backup start time. Try to re-schedule the backup to some different time. To troubleshoot VSS issues, download and run Acronis VSS Doctor available at https://www.acronis.com/en-us/personal/vss-diagnostic-free-tool/
| --------------------
| Error code: 65520
| Module: 0
| LineInfo: bd28fdbd64edb8fb, e:\219\core\common\error.cpp, Common::Error::AddHResult, 317
| Fields: code : 2147754994, $module : disk_bundle_vsa64_10170
| Message: Unknown Win32 error
| --------------------
| trace level: error

- Log in to post comments

Everything is green when I run the tool.
Then again last night's backup ran without problems (typical).
The server has not had a reboot or any changes since the failed backup 2 days ago.
Will run the tool again when I receive a failed backup message.
- Log in to post comments

Ok, please keep us posted. Hope the things continue to be normal for you.
- Log in to post comments

Backup failed again with the message possible High I/O.
VSSDoctor results in all VSS writers timed out.
I have already tried to re-register the VSS writers before, but that didn't help unfortunately.
Can try it again of course, maybe run it every time before the backup starts?
[code]
cd /d %windir%\system32
net stop vss
net stop swprv
regsvr32 /s ole32.dll
regsvr32 /s oleaut32.dll
regsvr32 /s vss_ps.dll
vssvc /register
regsvr32 /s /i swprv.dll
regsvr32 /s /i eventcls.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s vssui.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml4.dll
vssvc /register
net start swprv
net start vss
[/code]
- Log in to post comments

If you still see the message "VSS writer 'System Writer' has timed out", one of the possible solutions would be setting the VSS timeout value by editing the following registry key:
HKLM\Software\Microsoft\Windows NT\CurrentVersion\SPP\CreateTimeout.
Change value to 12000000 (2*10*60*1000 = 20 mins).
The default value is 10 mins, so doubling it should give enough time to complete the snapshot. If the key isn't present, you will need to manually create a DWORD 32-bit key "CreateTimeout".
If still no joy, I'd recommend checking the Windows event logs for more information regarding the VSS issue.
Go to Start -> Run -> eventvwr.msc. Navigate to Windows Logs -> Application and Windows Logs -> System
Check for error messages related to VSS service. Search for solution in Microsoft Support articles, for example see this Microsoft TechNet Article.
- Log in to post comments

The settings I have done a long time ago:
HKLM\Software\Microsoft\Windows NT\CurrentVersion\SPP\CreateTimeout --> 12000000
HKLM\System\CurrentControlSet\Services\VSS\Settings\IdleTimeout --> 12000000
HKLM\System\CurrentControlSet\Services\Volsnap\MinDiffAreaFileSize --> 2000
Also changed the VSSAccountControl to make sure the Acronis account has access.
No luck so far.
VSS writer results in the most common error 0x800423f2 --> the writer's timeout expired
There is a timeout issue, I just can't find how to fix it :(
- Log in to post comments

After a successful backup on Tuesday evening, another fail on Wednesday evening.
No reboots, no changes. The VSS writers have a mind of their own.
- Log in to post comments

Hello,
could you please also check if shadow copy storage has enough free space? You need to make sure that the volume where the snapshot storage is created has enough free space.
I hope this could help you. If you have any further questions, please do not hesitate to contact us.
- Log in to post comments

We have exactly the same problem on Server Essentials 2016. Backup was working for months now without any issues, no changes made. Did a clean reboot, free space is sufficient, timeout registry entries were made on initial setup and confirmed that they are still present. Now since several days no backup completes (approx. 24.08.), neither scheduled nor manual. Tried a manual Windows Backup, which works.
I have no further idea on how to solve this. VSS Diag is all green beside the failed writers after a backup Job and according Event log entries.
- Log in to post comments

ITSFE , hello
welcome to Acronis forum!
I would suggest you create a ticket with our support so that my colleagues could thoroughly analyze the logs and investigate the issue.
If you have any questions do not hesitate to contact me.
Thank you!
- Log in to post comments

I have exact the same problem on a new installed Server Essentials 2016. There is nothing special on this machine.
- Log in to post comments

Hello Kreuzritter,
have you tried to apply the above suggested steps (Acronis VSS Doctor and increasing timeouts)? The status of VSS writers can be checked using instructions from https://kb.acronis.com/content/36200#1_4_VSS_writers_should_be_enabled_and_in_consistent_state7
- Log in to post comments

I have tried everything. I have already reinstalled the server three times. The official support has told me that the problem is known and they are working on a solution. This was definitely the last time I bought Acronis.
- Log in to post comments

Would you mind sharing the ID of the related support ticket?
- Log in to post comments