[ABR11] Backup Failed - Network Disconnected

I am also having this problem, It occurs on all my scheduled or manually started backups when trying to do a daily or weekly backup. Monthly backups run just fine. I have an open case with acronis regarding this issue (its been open for a while), but its good to know its not just me!

Are you backing up to a NAS? If so what kind. I am backing up to a FreeNAS (BSD based) NAS, and Linux based D-link NAS.

One thing that I have been looking into as a cause is the way that Acronis uses file locks. Daily and weekly backups require acronis to connect and read the monthly backup to find which items have changed. Doing so causes a lock on the file (to prevent data corruption) and when it tries to write changes it immediately fails.

I have analyzed the network traffic and connections that acronis makes to the files, and it appears that it is putting a Samba "DENY_ALL" lock (Samba share modes defines the DENY_ALL as "Deny all open requests on the current file") that prevents the file from being subsequently opened and changed (support once told me that every backup requires approx 4 connections to the destination). This results in the "Error occurred while writing the file". Acronis assumes the network connection was lost, resulting in the "Network Disconnected" error, and thus a failed backup.

I disabled Oplocks on my NAS to eliminate that as a point of failure, but it made no difference whether they were enabled or not.

While testing this I did a full backup to a share on a Windows machine (NTFS) and then ran daily backups, which succeeded. To me this points to either Samba protocol differences or file system differences. (The NASes i used have either ext3 or UFS filesystems.)

This is all me speculating and reporting what i have observed. As for a solution, i guess we have to use Windows as a storage medium (not really an option, at least for me) or wait for Acronis to come up with something.

Hi, you both are not alone with this problem. One of my customer also have this problem. He cannot create differential backups using Tower of Hanoi, GFS and custome scheme. I don't now exaclty what he is using for backup destination but it may be NAS or external USB drive connected to one of machine in the network. First time support provide me info that this is known issue. Second time they provide me info to try backup on local drives to check if they are performed OK and that's all at the moment.

davidb wrote:

Are you backing up to a NAS? If so what kind. I am backing up to a FreeNAS (BSD based) NAS, and Linux based D-link NAS.

One thing that I have been looking into as a cause is the way that Acronis uses file locks. Daily and weekly backups require acronis to connect and read the monthly backup to find which items have changed. Doing so causes a lock on the file (to prevent data corruption) and when it tries to write changes it immediately fails.

I have analyzed the network traffic and connections that acronis makes to the files, and it appears that it is putting a Samba "DENY_ALL" lock (Samba share modes defines the DENY_ALL as "Deny all open requests on the current file") that prevents the file from being subsequently opened and changed (support once told me that every backup requires approx 4 connections to the destination). This results in the "Error occurred while writing the file". Acronis assumes the network connection was lost, resulting in the "Network Disconnected" error, and thus a failed backup.

I disabled Oplocks on my NAS to eliminate that as a point of failure, but it made no difference whether they were enabled or not.

While testing this I did a full backup to a share on a Windows machine (NTFS) and then ran daily backups, which succeeded. To me this points to either Samba protocol differences or file system differences. (The NASes i used have either ext3 or UFS filesystems.)

This is all me speculating and reporting what i have observed. As for a solution, i guess we have to use Windows as a storage medium (not really an option, at least for me) or wait for Acronis to come up with something.

Yes, I am backing up to a NAS. I'm using a Synology NAS to store our backups.

What is strange about this problem is that I have another server (a windows file server) that does file level backups to the same exact NAS and that one executes all its backups without a problem.

The backup that is failing usually runs 20+ minutes into the backup and then fails. The amount of time isn't identical each time but it has been between 20 and 30 minutes so far.

Something else i noticed about my backup plans that fail the daily/weekly vs the plans that work, it seems that the plans that are using a managed vault are the ones that fail.
I have the laptops we are backing up using an unmanaged vault (no storage node) and they are able to complete the daily/weekly backups (most of the time anyways). Like you said, the computers are using the same NAS but some succeed and some fail.
Support told me my case has been escalated to the expert team, if i hear anything from them i will definitely follow up here.

I have the same error when performing file backups of any kind (simple, GFS, etc). I am only backing up 2 servers, and one of them is the local server that AMS is installed on. Both of them fail with the file based backups, and with the same error listed in the first post.
The disk image backups work fine - they are currently weekly full image backups with a 2 week retention. I am using 2x iSCSI based Drobo's that have 4x 4TB partitions each set up as managed vaults (primary and secondary). These 4TB partitiions are mounted as local drives on the AMS server.
Acronis Support took my diagnostic logs and asked me use domain admin credentials for Managed Machine Service, Acronis Management Server Service, and Acronis Storage Node Service (they were using local accounts with local admin rights). I did this (and had to re-do all of my managed vaults and backup jobs from scratch as a result). It worked - for a week. Then, it started failing again with the same error.
BTW, support has been averaging 5 working days to respond.

I tried getting some help from support but it didn't prove very helpful.

Initially when I had this problem, I tried getting help from support but it went nowhere. I resorted to deleting the archives, the backup plans and completely removing Acronis from the server that I was backing up and then reinstalling Acronis. This seemed to work for a few days but then stopped again and started failing the daily and weekly backups once more.

I tested to see if it was the running backup priority (both low and high) and whether deduplicating after transferring to the vault would change anything. Neither of those things helped in any way.

What is strange about this is that I backup up another server using file level backups with more files than this one to the same NAS and the same vault so I do not believe that it has anything to do with the vault or the NAS.

I'm awaiting the end of the work day to run some disk checks on the disk to see if there are any faults.

Does your Synology NAS support iSCSI?

I had lots of problems running Acronis Storage Node with a "Windows Networking" Backend (Samba in my case). Mostly really really bad performance and permission/locking problems that don't show up in the logs as permission/locking problems.

If you can, configure your NAS to export an iSCSI-LUN, configure your Windows node to be an iSCSI-Initiator and mount the drive. This way Windows will bother about permissions, locking and the NAS will be just a dumb block-store. This worked pretty well for us.

Just be really really sure to disconnect the Initiator before reconfiguring, rebooting or disconnecting your NAS - otherwise bad things could happen to your datastore.

Yes. The NAS does support iSCSI.

I'm not comfortable with that last part though:-

--->> "Just be really really sure to disconnect the Initiator before reconfiguring, rebooting or disconnecting your NAS - otherwise bad things could happen to your datastore. " <<---

Sometimes, we have to update the NAS software and someone might inadvertently update the NAS and have it reboot without un-mounting on the backup server.

This problem is just a bit perplexing for me.

that means in normal operations you shut down your NAS without checking if users have open files?

Well you should keep in mind that shutting down the iSCSI is equivalent to unplugging the drive during operations. If you're confident your backup vault can handle it - do it. If not make sure everybody with admin access checks this before rebooting.

If you're on build 17318 of Acronis, then your iSCSI-connection is not your main data integrity problem :)

The NAS devices are for storing backups only. Our servers are backed up to the NAS devices so the NAS devices only 'come alive' at a certain point during the day.

So you have several devices backing up to the NAS and I assume you're not using a storage node.

You could try using an alternative protocol - maybe FTP. I know FTP had huge problems in ABR10, but maybe those are fixed in ABR11. You could at least give it a try.

And as a bonus, data transfer via FTP is usually a slight bit faster (and uses less CPU).

What I did was to set up a SAMBA share on the NAS and I have the server with Acronis installed on it setup the storage node using that share. All the servers backup up using that storage node.

I would prefer not to use FTP.

What is really getting me is that this is the only file backup that isn't working. All the others work perfectly. All the disk backups work perfectly as well.

Heiko Helmle wrote:

I had lots of problems running Acronis Storage Node with a "Windows Networking" Backend (Samba in my case). Mostly really really bad performance and permission/locking problems that don't show up in the logs as permission/locking problems.

Heiko Helmle wrote:

So you have several devices backing up to the NAS and I assume you're not using a storage node.

When you were running the storage node that was giving you problems, was it giving you the same (or similar) errors we are reporting here? And were they resolved by moving away from the storage node?

I honestly don't remember the specific error. I got slow and failed backups and moved to ISCSI and now all the errors are gone. There are problems with Backup Stores accessed via Samba - so a solution could be not using samba.

Thank you Heiko! I am still waiting for a response from the acronis expert team, but in the mean time i did move my backup plans away from using storage nodes and ... I am now getting daily/weekly backups! Unfortunately I'm stuck with Samba for the time being, but actually having a few backups puts my mind at ease. Seems like a pretty serious issue if the Storage Node cannot back up to a vault via samba, which if i had to guess is probably one of the most common protocols used, especially for small business.

So to recap, If you are having the "Network Disconnected" issue, try using unmanaged vaults (no storage nodes), and iSCSI or FTP if you can.

//I still plan to follow up here when the acronis team gets back to me

We won't have deduplication without a managed node though.

Same for us with QNAP TS459 PROII
Any ideas how could it be resolved?

The interesting thing is that there are around 30 different machines with several backup plans
Only one backup plan (file-based backup, GFS) fails with the same error.

Same machines exactly run Image Backup, incrementally, on the same storage - without any errors.

come on Acronis! Please help!

--------------------
Log Entry Details
--------------------
Type: Error
Date and time: 24/12/11 11:00:22
Backup plan: [None]
Task: [None]
Managed entity type: [None]
Managed entity: [None]
Machine: XXXXXXXXXXXX
Code: 262,169(0x40019)
Module: 4
Owner: acronisAgent
Message:
Error occurred while backing up.

Error code: 0x40019
$module = "C:\Program Files\Acronis\BackupAndRecovery\DiskBundle.dll"
Tag: 0xCE542E14DA203B81
Error occurred while writing the file.

Error code: 0x40003
Tag: 0xD11B4F5EC453B993
Network disconnected.

Error code: 0x40015
Tag: 0x93943C42F2BFCC44
Acronis Knowledge Base: http://kb.acronis.com/errorcode/

Event code:

--------------------

<User's blocked: http://forum.acronis.com/forum/27401&gt;

Hello all,

Thank you for your comments, and for reporting about the issue.

First of all I would like to apologies for the inconvenience you've faced. I've checked our internal resources, but we never had such issue reported. So the situation requires investigation.

Please send us the following diagnostic data: 

1. Acronis Info
2. Wireshark

This should help us to localize the cause of the problem.

Should you need anything else or have any further questions - feel free to contact us at your earliest convenience, we will be happy to help you!

Thank you.

Hello All,
I got a response from the acronis expert team that my case regarding this issue had been given to, and i would like to share the information I got. After sending in about 5 GB worth of wireshark and procmon logs (from the target computer and the storage node), this is the response i got:

I have studied the logs and system information you've provided and came to a conclusion that the backup fails because of unexpected network drops, when the computer is communicating with the network device.

From my experience, this may be caused by improper network controller workflow in specific situations. In order to address this issue, please proceed with these steps:

1) Please update the drivers for your [...] NIC.
2) Install them, reboot if necessary and then retry the backup immediately - will there be a success?
3) If not, please navigate to the Device Manager, directly to the network adapter advanced properties and then find and disable every single setting related to "offload" or "offloading".
Press OK and then retry your incremental backups again. Any change?

If there will still be no success and the behavior will not change even after driver update, please let me know the exact model of your NAS device and the firmware version it is currently being run at.

This did not resolve my Network Disconnected errors. We'll see where we go from here.

I have had this same problem since Echo Server. I did a clean install of BR10 and the problem persists. The above changes did not correct the problem.

davidb wrote:

Thank you Heiko! I am still waiting for a response from the acronis expert team, but in the mean time i did move my backup plans away from using storage nodes and ... I am now getting daily/weekly backups! Unfortunately I'm stuck with Samba for the time being, but actually having a few backups puts my mind at ease. Seems like a pretty serious issue if the Storage Node cannot back up to a vault via samba, which if i had to guess is probably one of the most common protocols used, especially for small business.

So to recap, If you are having the "Network Disconnected" issue, try using unmanaged vaults (no storage nodes), and iSCSI or FTP if you can.

//I still plan to follow up here when the acronis team gets back to me

Can anyone confirm this? I am using managed vaults and am running out of patience with this product. For some reason only full backups run.

http://forum.acronis.com/forum/29550

Hello Adrian

I wanted to follow up on my experience with using unmanaged vaults due to the network disconnected issue. After I stopped using the storage nodes, backups got a lot more consistent, however not without problems. Most of the time my daily/weekly backups would run, but after a while i would start getting an error that "the handle is invalid" and it could only resume daily/weekly backups after another Full backup had been made.
(check topics http://forum.acronis.com/forum/27602 and http://forum.acronis.com/forum/28970)
Acronis support provided me with an internal build (there were several reasons for this, not just this issue) and i have been running with it for just over a week. So far i have not gotten any "invalid handle" errors from running my backups to unmanaged vaults, although i want to run this for a full month before im convinced it is fixed. I have not tested the storage node functionality yet, i think that the cons outnumber the pros of using it for my particular situation (not using dedup, don't want to install and do the storage node dance again), so i do not know if it fixed the network disconnected issue. There are still issues with the build that support gave me, hopefully by the time they make the next public release they are resolved...

-davidb

hello. have you tried http://www.synologybackup.com http://www.synologybackup.com?

it's a remote backup service provider. this is better than a universal serial bus device and your files will always be in security.

stela c

Thank you both for your replies, we are testing disk level backups.

stela c , we are backing up to a RAID 1 volume on the local machine and replicating to another RAID array and a NAS device.

My main backups work. I backup to managed vaults on a storage node. They are not on a NAS.

I am having the same issue, however, on replication: ABR Virtual & Advanced 11.0.17318 on server 2008 R2. I have done a Clean uninstall on my AMS and created a domain admin specific login for the ABR services. I use that same user for the storage node and the vaults. Prior to the clean uninstall I had mixed local credentials and domain credentials (not good!). I called in through the Acronis Partner line and got a tech that knew what he was talking about! We removed the storage node and recreated it using the acronis_domain credentials, then I ALSO clean uninstalled my agent Machines and reinstalled them using the acronis_domain credentials. It is vital that the agents use this same domain credential. a world of issues went away when i did this!!

I now, however, get the Network Disconnected issue when the backups attempt to replicate to Tape (set for AFTER Backup). My tape is a 3 TB LTO5. it is also a managed vault on the same storage node. Some of the replication tasks work, most don't. I am noticing that every failure is on ENCRYPTED node replication. i will also note: I can replicate on a schedule prior to the backup without this error.

I am going to remove the encryption for tonight and see what happens.

If you're using a NAS make sure you turn off the TOE on the settings on your NIC. The TOE just sucks and causes major issues for backups and any large file transfers.

Under device manager open your nic and disable the following settings...

Large Send Offload V1 & V2
Receive Side Scaling
TCP Connection Offload (IPv4 & IPv6)
TCP/UDP Checksum Offload (IPv4 & IPv6)

Ryan might be right. The Veeam-guys found out some time ago, that there's a few faulty network drivers floating around that will happily pass corrupt ethernet packets to the OS though they were supposed to discard them because of the checksum mismatches.

As few application re-check what comes through a TCP connection (because it's supposed to be safe), acronis might choke on it on a later stage.

It's definitely worth trying to disable all driver offloading.