Backup Validation failed. Also, Performance optimization tips?
Hello,
I'm currently evaluating (running trial) AB Server 12. I've been testing different test servers (Windows servers, no real data, no risks) successfully.
1. Validation error:
I wanted to be sure and (thanksfully) I did one last test copying real data to a test server ( between 1 and 1.5 TB, TONS of files > 900.000 ).
Today I went ahead with this new test server: Installed the acronis backup client, removed all my previous test schedules, cancelled a running backup that had already started, and after a few mintutes I performed a manual backup (pressing the "Backup now" button).
After the backup finished, I've got an error:
|
Error Date and time May 15, 2017, 12:05:43 PM Module 309 Message Additional info: ------------------------ |
I didn't have this issue before trying with real data. (Previous backups, which included this test server and another one, were more lightweight and had other backups as 'stuffing'. The current scenario features lot of information with varying filesizes). Meanwhile, I will run this same backup again.
2. Performance optimization? Any tips or recommendations?
I've been benchmarking a lot, since We're having trouble with the ammount of time our current backup solution takes to finish the backup. I've noticed that acronis performance is similar to our current solution.
- I realized that the smaller the files are, the worse the performance is. We have the need to back up two (outdated, to be replaced) file servers that take too much time, during nights. We cannot control the nature of the files themselves, so I'm wondering what other factors should I consider?
- The previously mentioned TEST and BACKUP-TEST servers are rackmounted Xeons with 8GB RAM (more shall be added when going productive). Both have Areca 1264IL RAID Cards with 1GB cache. OSes are in RAID1, and the DATA logical disks are (also, in both servers) 6x disks RAID6 arrays. OS partitions are MBR while DATA partitions are GPT. Both servers feature intel X540-T1 10gbps NICs. As a reference said servers can sustain transfers of ~400MB/s when copying finished backups (the giant .tib files).
- The issue arises when a large amount of small files is involved.
What would you suggest to optimize our systems so the transfers/Backups go faster?
Hi,
1. The validation error indicates archive corruption which needs to be re-created. It makes sense to start backing up into a new folder in the same location. An archive is a container for backups, including full and incremental ones, so corruption of some full backup (if you created only full backups) may cause the entire archive validation failure. In the upcoming update for Acronis Backup 12 which will come along with Acronis Backup 12 Advanced release there will be a possibility to define archive name, so starting a new chain would not require re-setting the backup location (e.g. you won't need to create a new folder).
2. The best way to backup file servers is via "Entire Machine" backup, instead of using "Files/folders" selection - the disk/partition ("Entire machine") backup will be much better in terms of performance especially when the machine holds millions of small files.
Thank you.
Thanks for the reply,
The second backup, that ran before your answer also failed. So now, I'm running a third backup in a new folder. I'll update you once it finishes.
All backups (including previous backups) are done as "Entire Machine".
I've been checking resources and I'm not detecting what is capping the final performance. RAM, CPU, Disk queue. I've tested with "Normal" compression and "High" with almost no difference both in times (~20 minutes) and sizes (~35 GB), surprisingly in favor for "High" compression.
- After the first backup finished, I've copied the TIB file (600GB) from the backup test server, back to the test file server. After it finished, I renamed it and started copying it back to the Backup server. This test allowed me to check (in a simple fashion) the performance roof for I/O. After some minutes, I started the second backup. Monitoring includes SNMP traffic obtained from the XGE switch, windows resource monitor and task manager. I'll try to upload some screenshots from said test.
- Offtopic: As soon as ANY backup starts, progress is always 9%. Is this behaviour considered normal?. Also, there is a time (around 5 minutes) between pressing "Backup Now" and seeing network activity. Is this also normal?
Thanks again,
Agustin.
I'm afraid the third backup also failed. What could be the problem ?
|
Error Date and time May 16, 2017, 08:45:37 AM Module 309 Message Additional info: ------------------------ |
What should I try next ?
Regards
Hi,
Thank you for the clarifications and additional details. In general copying a single huge file would be faster than taking an image of a system which implies reading through the file system and jumping between the occupied sectors to read data for the files (skipping the empty sectors), so it's not really valid to compare these speeds. They would be comparable for example if you created a dedicated partition, then copied a single huge file to it and tried to back up this partition.
However we do work on improvements for backup speeds and the upcoming update for Acronis Backup 12 (which will come along with Acronis Backup 12 Advanced release pending for the end of May) contains a new v12 archive format support which allows us to better process large number of files in both backup and recovery processes, so you should definitely see an improvement. These performance improvements are partially included into Acronis Backup 12 Advanced Beta builds and the final RTM build will contain all of them. We're testing various systems configuration with 1GBx1000 files, 1MB x 1000k files, files with random data (non-compressable) and with aligned data sets inside to make sure that we get best possible performance for most real-life scenarios.
What concerns your percentage-related question - on the first stage there is snapshot activity performed which requires some time to get a valid snapshot (involves VSS-related activity for example), so it's normal that it stays on 9% for some time before you see the network traffic. Before 9% there are other operations included, such as pre/post commands for example.
Thank you.
Hi,
Concerning the validation failure - it could be caused with problems accessing the backup archive location after backup creation. Can you please try recoverying some data from the created archive? The archive could be still valid. Other than that you should contact our support team for assistance. This particular validation error we've seen only in specific cases (there were only a couple of them reported) and these cases were caused by specifics of the backup location used: for example in one case the USB drive used as backup storage was connected via USB3.0 and this caused issues, while connecting it via SATA fixed the problem.
Thank you.
thanks for your reply. This is very discouraging.
I'm having another serious problem: I'm using the acronis Booteable media to restore a whole disk from the previous backup.
Everything goes fine until you must specify a local destination to said backup. Please see the attached photo. Can't press OK, all options are greyed out. This is happening with Two different servers. One with previously said Areca, and the latest one (from the attached picture) with LSI.
Any clues? Thanks again
Hi,
Concerning the bootable media recovery - this issue is very likely to be related to the same root cause as the archive validation issue. I would recommended to create backup to some other location first of all. For example try to capture backup of a single disk to another local disk and validate it. If validation passes properly then further localize the issue, by moving the created backup to original backup location and trying to perform incremental backup (edit the backup plan and point to new backup location). If validation fails, then check what could be backup location specifics (some SMB settings or specific external drive settings, etc.). That's why my previous recommendation was to contact our support team for further investigation. Such issues can rarely be solved via forum.
One thing you can try from bootable media environment is to restore "Disk", rather than a "Volume" from the backup. This selection is made on the 1st step of the recovery wizard when you browse the archive path and select the recovery point in the pop-up - in the left bottom part there is a selection of what item to recover (Disks or Volumes). See attached screen shot. Another thing is to try accessing the backup archive from another location.
Thank you.
Hello,
I added another test server and performed a backup without issues. So, I deleted the test file server from the acronis management console and then uninstalled the acronis backup agent to install it again.
Finally, I changed the destination folder (another subfolder) once again and saved that as a new backup plan. This time, before I run the backup, I emptied the test file server data partition and left a single 5 GB file to speed things up.
I've got the same kind of error again:
|
Error Date and time May 16, 2017, 03:20:55 PM Module 309 Message Additional info: ------------------------ |
Before I move forward to opening a ticket I would like to be sure I correctly understood your instructions about the new folder in the same location. "Backups_TEST" is the shared Folder. Some tests backups are already there, so I created a subfolder "FileServer_TEST". It failed again, so in my latest attempt (that also failed), I created another subfolder "FileServer_TEST2" (at the same level as the first one).
Any ideas? Thanks and regards,
Agustin
Hi Agustin,
Basing on the additional details, since another server is backed up properly to the same backup location (so the problem is not in 192.168.0.47 share host specifics), it looks like the problem is related to the specifics partitions structure on the server which is being backed up (e.g. on source). Probably there are some specifics in how the partitions are organized on the disk - this can be revealed from investigation with our support team.
Thank you.