Validating backup takes all memory

Hello Radoslaw,

Let me clarify this situation for you.

Unfortunately, validation process is designed in such way - it just uses all available memory to process the data. Probably we will change the process in one of next releases.

I'm sorry for the inconvenience.

Regards,

Can we assume that no changes to this process have been made in TI 2011?

Yes, I can confirm this problem exists in 2011, but it only affects x64 versions of Windows.

Validation run separately from backup works fine, but validation immediately following backup consumes all RAM (assuming your backup file is larger than your available RAM). Due to the way ATI opens the file Windows attempts to load it in its entirety into the System File Cache. On x64 systems this cache can easily grow to fill physical RAM and it seems applications must be well behaved if they are not to bring the system to its knees!

A ticket has been raised and is currently under investigation. With luck the reply will not be the same "possibly we might fix it one day" response in this thread!

This is a serious issue. ATI does not in fact ask for a lot of memory for itself. Its method of file access causes the system fie cache to grow uncontrollably until it consumes almost the entire RAM. Even ctrl-alt-del takes many minutes to react, if it all. If the user's screen has gone into standby pressing a key on the keyboard will not bring it back for a similar time. Often even the mouse pointer is frozen. The system becomes so unresponsive people could easily assume the system has hung and reboot.

What is possibly even more worrying, I suspect (although I have not positively confirmed) that the reason for the issue may be that ATI is attempting to validate the backup from the cached image created during backup. If the backup is able to fit into the available RAM (eg. a 4GB backup with 8GB RAM) it is possible the disc copy is not being read at all!! In which case the backup has not really been validated.

Jon Gardner wrote:

What is possibly even more worrying, I suspect (although I have not positively confirmed) that the reason for the issue may be that ATI is attempting to validate the backup from the cached image created during backup. If the backup is able to fit into the available RAM (eg. a 4GB backup with 8GB RAM) it is possible the disc copy is not being read at all!! In which case the backup has not really been validated.

I beleive I have now confirmed this is exactly what happens!!

I have managed to set up a backup, where post backup validation COMPLETES before the backup has even been full written to disc!! I am adding this info to my ticket on this issue.

Jon Gardner wrote:

Jon Gardner wrote:

What is possibly even more worrying, I suspect (although I have not positively confirmed) that the reason for the issue may be that ATI is attempting to validate the backup from the cached image created during backup. If the backup is able to fit into the available RAM (eg. a 4GB backup with 8GB RAM) it is possible the disc copy is not being read at all!! In which case the backup has not really been validated.

I beleive I have now confirmed this is exactly what happens!!

I have managed to set up a backup, where post backup validation COMPLETES before the backup has even been full written to disc!! I am adding this info to my ticket on this issue.

If this is the case and the archive is not being validated from the target media then it is an extremely serious flaw. Bad media whether disk sectors or optical and related subsystem failures are indeed a cause for corrupt archives.

Yeah indeed, kind of "invalidates" the whole process! :)

Only applies to validation done immediately after a backup, so there is a workaround. And things are possibly OK if the backup is bigger than your amount of RAM (less say 500MB or so) or, for x86/32-bit systems, 1GB, but, still, as you say, extremely serious.

Note: this problem of validating from cache instead of disc applies to 32-bit systems as well as 64-bit systems, although the former are not affected by the performance problems.

Jon Gardner wrote:

Yeah indeed, kind of "invaldiates" the whole process! :)

Only applies to validation done immediately after a backup, so there is a workaround. And things are possibly OK if the backup is bigger than your amount of RAM (less say 500MB or so) or, for x86/32-bit systems, 1GB, but, still, as you say, extremely serious.

Note: this problem of validating from cache instead of disc applies to 32-bit systems as well as 64-bit systems, although the former are not affected by the performance problems.

Jon Gardner wrote:

Yeah indeed, kind of "invaldiates" the whole process! :)

Only applies to validation done immediately after a backup, so there is a workaround. And things are possibly OK if the backup is bigger than your amount of RAM (less say 500MB or so) or, for x86/32-bit systems, 1GB, but, still, as you say, extremely serious.

Note: this problem of validating from cache instead of disc applies to 32-bit systems as well as 64-bit systems, although the former are not affected by the performance problems.

Just to clarify:
With my Win64, i7-920, asus p6t, 6GB ram, if my tib files are larger that 6GB this is of no concert to me??
My C: partition images run about 20GB, full disk images (C: and D: on same disk) run about 168GB. I only do regular full backup images, no incrimental, differential, etc. to an external e-SATA 1TB drive, always keeping three most recent.

Thanks for comments ... Bob

First, although I am confident of my findings, there has yet to be a response from Acronis. I guess we should wait before being too definitive.

I should also say I have only tested this on ATI 2011. Because of the date of the original post I would guess that was on an earlier version. The problems sure sounds the same but I cannot say for sure if earlier versions have this "validation from cache" issue or if only ATI 2011 is affected.

However, if things are as they appear to be, as you are on x64:

- You should be affected by the performance issues during validation that were the original subject of this thread.

- As your backup is bigger than your RAM most probably your backup is being correctly validated. In fact this would be true I believe even if you do an incremental or differential backup as the entire backup is validated sequentially from the beginning and by the time it gets to validate the new inc./dif. validation file it should have been flushed from the cache. However, it is hard to be sure. The Windows System File Cache makes its own decisions about what to keep and what to discard. Since the performance issues are caused by Windows paging out most of the heart of the operating system in a futile attempt to cache the entire backup I would expect it to have had to flush out the end of the file some time before it is needed for validation and so it will need to be re-read from disc, which is a good thing. But, thinking about it, as Windows is attempting very hard to cache the backup file, has been told the file is going to be read sequentially, and is looking ahead to what might be needed next it is just about possible that in some situations (maybe only when the backup is a little bigger than RAM, or maybe not) it might hang on to the end of the backup that was put in the cache to be written and instead release earlier parts of the file. I.e. most of the file might be validated from disc, but the end of it might be validated out of the cache. Honestly I am not sure about this last bit. I may be worrying unnecessarily, but the point is because ATI is not correctly flushing the cache before starting validation it cannot be sure where the data is coming from!

After some more work on this it is quite possible all is OK. I may have misread some data. Thought it was instantaeous throughput when it may be an average over several seconds. In my defence(!) - the documentation does not make it clear which.

I'd hold off panicing until Acronis confirm one way or the other!

Jon Gardner wrote:

After some more work on this it is quite possible all is OK. I may have misread some data. Thought it was instantaeous throughput when it may be an average over several seconds. In my defence(!) - the documentation does not make it clear which.

I'd hold off panicing until Acronis confirm one way or the other!

Still no response to this issue from Acronis?? ... Bob

Yesterday an escalation engineer from Acronis contacted me. I'm hopeful this will now get some swift attention.

Jon

Jon,

I think you are onto somthing here.
When I do backup with validation, I occational get validation failures.
Howevere, running validation standalone will pass on a archive that failed the integrated validation.

Rolf

Following comments elsewhere I have discovered that temporarily disabling Kaspersky protection (no need to uninstall) resolves the issue here. I guess this makes it a Kaspersky issue, although I wonder if all who have seen this problem use Kaspersky or if it affects other security products.

I also wonder why standalone validation performs differently?

Jon

Jon,

I'm using Microsoft Security Essentials. I noticed .TIB files are listed as excluded file types, so I would think MSE should not affect it.

I'm thinking maybe there may be an issue with ATIH may open the file for reading before it has been completely flushed to disk.

BTW I'm using a Promise NS4300 NAS drive.

Rolf

Jon Gardner wrote:

Yesterday an escalation engineer from Acronis contacted me. I'm hopeful this will now get some swift attention.

Jon

Would this possably be safer to do as separate ATIH operations, to be certain that the Write to Disk is completed, and the cache is flushed ??

Normally I do a full disk backup using Win7 task scheduler with several actions:

A bat file - 30 second delay (to have everything up and online)
A bat file - sends email that backup is starting, date and time
A Acronis task - backup to external SATA with validation
A bat file - sends email that backup has completed, date and time
A bat file - System shutdown!

After working out a few kinks this has worked flawless for several months, with never an error, and never a validation failure. However, the subject of this thread has caused me concern! Would TWO separate Acronis tasks -- a backup task followed by another task for validation -- eliminate the possiblity of the data not being read from the target .tib during validation??

Thanks for comments --- Bob

Since it isn't clear what is happening regarding reading the archive information for validation or if there is any problem at all, it is impossible to say what might or might not circumvent the issue if any.

However, separating the tasks is more likely to be the better way to do it but if you are concerned about cached data then do a reboot between the create and validate steps.

If you really want some extra security rather than reboot Windows, reboot the TI rescue CD and validate with it. That will get rid of any Windows cached residue for sure and has the added benefit of reading the archive into RAM and calculating the checksums using the environment that is actually used when restoring a failed HD.

Seekforever wrote:

Since it isn't clear what is happening regarding reading the archive information for validation or if there is any problem at all, it is impossible to say what might or might not circumvent the issue if any.

However, separating the tasks is more likely to be the better way to do it but if you are concerned about cached data then do a reboot between the create and validate steps.

If you really want some extra security rather than reboot Windows, reboot the TI rescue CD and validate with it. That will get rid of any Windows cached residue for sure and has the added benefit of reading the archive into RAM and calculating the checksums using the environment that is actually used when restoring a failed HD.

Seekforever....
Thanks for your information and suggestions.

I am not sure what the lack of any definitive information from Acronis on this issue might mean, but perhaps we expect too much!

As to your suggestions, I understand your points – however, that all requires manual time and intervention, which I can do on occasions as a test. However, as my regular backup I guess I will leave in place my procedure described in my previous post which runs unattended after hours under windows task scheduler.

Otherwise, to use separate backup and validate tasks in my procedure needs two things:
First, some operation following the backup to insure the cache is flushed.
Second, having a file name of the .tib to give the validate task.

The trouble there is that the backup task creates a name containing “date” and “time”. I guess I can get around that by using “date” but without “time”, and being sure the windows task scheduler doesn't start the task until past midnight.

The game playing begins!!!

Of course, getting a definitive explanation for a “worm fuzzy” feeling about the cache issue discussed in this thread being NO PROBLEM would eliminate all this concern!!

Bob

-
OK, BUMMER!!
Unless I am overlooking something, I find no way to schedule a unattended validation task, because it requires manually selecting a backup file. My hope was to seperate the validation task from the backup task, rather than just checking the "validate after.." box in the backup task (see last post).

In the scheduled backup task, my .tib files are named containing the date -- as in "FileName_@date@.tib". I had figured to do the same in a subsequent validation task, to identify the just completed backup file. (Again, see previous posts.)

All my regular backups are done after hours, using Win7 task scheduler (see post #15), and works well except for the concern about cached data being used during valadate !

So, my question: Is there no way to identify the file name to a scheduled validation task, using a variable @date@?? As I said, I may well be overlooking something.

Thanks for any comments..... Bob

Just to confirm from my side: Win7 64bit, 99% RAM used as well. I just cannot understand why Acronis do not include an option to schedule a validation task separately from the backup task. Everything else works fine (or I haven't found any further flaws :)