Input/Output error since upgrading to 14330
Hi, Ever since upgrading our AB 12.5 to the latest 14330 build, all of my scheduled backups for the VM's hosted by a particular ESXi host fail with "input/output error". The backups (for all but one VM) appear to complete (i.e. the status is shown as "OK"), but the overview shows "Backup Failed" alerts for every backup plan with the input/output error for every VM in the plan on that ESXi host. The activities themselves for each VM don't show an error! If I reboot the management server computer (it's running on physical Windows Server 2012 R2 outside of the VMWare vCenter) and run a manual backup, I get no errors. An error log from one of the VM's is pasted below. Any ideas? Operating system error: Input/output error MODULE309 MESSAGEBackup failed Additional info:------------------------ Error code: 61 Module: 309 LineInfo: 0x4A8728DC8A1C9584 Fields: {"$module":"mms_bundle_glxa64_14330"} Message: Command has failed. Command=Backup plan 'SBS Backup'; ------------------------ Error code: 22 Module: 309 LineInfo: 0x8D165E86FB819597 Fields: {"$module":"mms_bundle_glxa64_14330","CommandID":"D332948D-A7A9-4E07-B76C-253DCF6E17FB"} Message: TOL: Failed to execute the command. Backup plan 'SBS Backup' ------------------------ Error code: 22 Module: 309 LineInfo: 0x8D165E86FB819597 Fields: {"$module":"mms_bundle_glxa64_14330","CommandID":"D332948D-A7A9-4E07-B76C-253DCF6E17FB"} Message: TOL: Failed to execute the command. Backup plan 'SBS Backup' ------------------------ Error code: 41 Module: 307 LineInfo: 0xE6792A5EE190DE2A Fields: {"$module":"mms_bundle_glxa64_14330"} Message: Failed to execute the command. ------------------------ Error code: 22 Module: 309 LineInfo: 0x8D165E86FB819597 Fields: {"$module":"mms_bundle_glxa64_14330","CommandID":"F30407D6-601F-11E0-9C67-FF46DFD72085"} Message: TOL: Failed to execute the command. Backup workflow ------------------------ Error code: 22 Module: 309 LineInfo: 0x8D165E86FB819597 Fields: {"$module":"mms_bundle_glxa64_14330","CommandID":"F30407D6-601F-11E0-9C67-FF46DFD72085"} Message: TOL: Failed to execute the command. Backup workflow ------------------------ Error code: 2 Module: 329 LineInfo: 0x8EDC81EA38FABAD3 Fields: {"TraceLevel":"1","$module":"mms_bundle_glxa64_14330","StepType":"2"} Message: Step 'Backup' has failed. ------------------------ Error code: 507 Module: 64 LineInfo: 0xA1D3981537C68833 Fields: {"$module":"disk_bundle_tape_off_glxa64_14330","id":"W2008.R2.SBS2011-EB97B47E-A2A9-47E4-8A49-BEACF7DC5800-503CE154-7AD5-EF5C-6CC9-0BB0AE69152EA.tibx"} Message: Failed to open the backup archive by the ID. ------------------------ Error code: 853 Module: 64 LineInfo: 0x6A1198D1B8BE2C36 Fields: {"$module":"disk_bundle_tape_off_glxa64_14330"} Message: Failed to open archive 'W2008.R2.SBS2011-EB97B47E-A2A9-47E4-8A49-BEACF7DC5800-503CE154-7AD5-EF5C-6CC9-0BB0AE69152EA.tibx'. ------------------------ Error code: 5005 Module: 667 LineInfo: 0xC3BC4C1EB2299AFE Fields: {"$module":"common_archive_addon_glxa64_14330","path":"W2008.R2.SBS2011-EB97B47E-A2A9-47E4-8A49-BEACF7DC5800-503CE154-7AD5-EF5C-6CC9-0BB0AE69152EA.tibx","function":"archive_open"} Message: Input/output error ------------------------ Error code: 7 Module: 4 LineInfo: 0x3D9EA26F1BFDF25D Fields: {"$module":"disk_bundle_tape_off_glxa64_14330","path":"/tmp/.mnt_net_smb_6F404230-F94E-4DA2-8057-F6300AE77CBC"} Message: Error occurred while opening the file. ------------------------ Error code: 65521 Module: 0 LineInfo: 0xBD28FDBD64EDB918 Fields: {"$module":"disk_bundle_tape_off_glxa64_14330","code":"5"} Message: Input/output error
Thanks Maria,
I've uploaded the system info as requested.
Neil
Hello Neil.
According to the information that you uploaded, the software cannot find the part of archive, the file name is "VMware vCenter Server Appliance-A0EE45ED-A740-41CA-85D9-6596AF5F2137-52E38EC3-5C33-5153-1A8B-C4FB99238F31A.tibx"
Please check it's availability/accessibility.
If there are other tibx files with the names like "VMware vCenter Server Appliance-A0EE45ED-A740-41CA-85D9-6596AF5F2137-52E38EC3-5C33-5153-1A8B-C4FB99238F31A-XXX.tibx" where XXX is a number, then please create an empty file and name it as VMware vCenter Server Appliance-A0EE45ED-A740-41CA-85D9-6596AF5F2137-52E38EC3-5C33-5153-1A8B-C4FB99238F31A.tibx and try to back up again.
Hi Maria,
I've deleted the "vCenter" TIBX files prior to a scheduled backup running and still having the same issue - input/output error. There were no duplicates, though it could be that the management server computer temporarily loses connection to the external USB disks. I've modified the power management settings for the USB disks/hubs so as not to go to sleep (allow the computer to turn off this device to save power = off)!
What I have found is that if I try to run a manual backup, the backups fail with the same error. It's only when I reboot the management server computer then run the backups, that the backups complete without error!
Seems that to fix the issue the management server machine needs to be running clean!
My next task is to schedule a reboot prior to the scheduled backups to see how that goes (not really wanting to reboot the server every day, but if that's the solution, then so be it.
I'll update on progress.
Neil
Hello Neil.
I've modified the power management settings for the USB disks/hubs so as not to go to sleep (allow the computer to turn off this device to save power = off)!
Yet it looks as your USB disks go into sleep mode which causes this problem. When you start the backup task manually and just after a reboot, the USB disks wake on and everything works. Please adjust the settings to prevent your hard drives from going into sleep mode during backup plan execution.
Hi Maria,
Changing the power management settings on the USB appears to have fixed the problem.
No errors on the scheduled backups since changing the setting.
Neil