Validation of one location slower than the other
Hey,
I have a backup plan that also does replication to a second storage location and runs also a validation within the plan.
Infos on the plan:
- backup of an entire Hyper-V machine
- agent is the Hyper-V HOST
- weekly full, daily incremental
- retention rules to keep 4 wekks full and 14 days incremental
- replication to second location with the same settings
Infos on the storage locations:
- both are QNAP NAS
- QNAP of first storage location has 4 aggreated 1GBit LAN ports
- QNAP of second storage location has 2 aggreated 1GBit LAN ports
- all devices connected to the same switch
The Backup I watch has
- one file with 12kB
- 4 files with about 20GB
. Acronis Backup 12 format
What I can see in the activity log:
- Validation starts on the second storage location
- the validation on the second storage location needs about 55 minutes
- the validation on the first storage location needs about 3 minutes
- During validation I can see about 100MB/s outgoing network traffic to the Hyper-V host.
So neither disk performance nor LAN performance differs.
- on the second storage location the traffic goes for a while via LAN port 1 with approx. 100MB/s then via LAN port 2 with thge same speed, and so on...the port change seems to be after about a minute.
Can anyone give me an explanation why the tieme differs in that amount.
It´s almost factor 20!!
It seems as if the validation of the first location transfers much less data (3 minutes, 100MB/s --> 18GB) whereas the validation task of steh second location transfers about 330GB.
The 18GB of transfer size of the first validation task fits the size of the last written tibx file quite well, but the more than 300GB does not even fit the size of the complete chain....
So whats the problem.
I can see this behaviour on all my plans: second location always much slower than first.
Regards
S.
Hallo Maria,
I know that validation is time consuming.
But as mentioned, the validation of the second location needs much more time than the size of the complete chain let expect.
As mentioned it is within one backup plan, so the agent is the same - so the assumption that it depends on the agent which runs the validation cannot explain it.
Am I really the first user who reports this?
And creating the logs, and send them to the support team, yes I can do this.
But up to know ALL support calls where I created logs (about 10 in the last 3 years) end in the suggestion
- please uninstall and do a clean reinstall
or
- please start new backup plans / backup chains
Is it worth the effort ?
Regards
S.
I did some assitional testing:
It has something to do with the backup plan !
If I create an off-host validation task that is executed on the same agent as the backup plan I have NO time difference between the two storage locations.
An off-host validation of a 120GB backup chain (5 files) with validating all backup needs about 50 minutes to validate.
If I validate only the last backup, it is finished within 3 minutes...
It seems as if the first location only validates the last backup, whereas at the replication target the complete backup is validated...
Can anyone confirm this?
S.
Hello Sven.
As mentioned it is within one backup plan, so the agent is the same - so the assumption that it depends on the agent which runs the validation cannot explain it.
It depends on your backup plan. There could be different Agents selected for performing backup, replication and validation tasks.
If I create an off-host validation task that is executed on the same agent as the backup plan I have NO time difference between the two storage locations.
Different Agents could have different access parameters to your first backup location. It results in the validation speed - it depends on the read data speed on your backup location from different Agents.
Can anyone confirm this?
We can confirm it after analyzing Process Monitor logs collected from the Agent which perform this validation.
Maria,
sorry as mentioned: all tasks are performed by the same agent.
So there is no possibility that different access parameters or anything else change the speed.
And the transfer speed is from both backup locations nearly the same (105MB/s vs. 110MB/s).
To sum up:
I am forced to create process logs over several hours (yes, the validation tasks need that long) and Acronis is not willing to try to reproduce the problem independently?
If this is correct, what is then the benefit for me as customer to have the AAP support if I have the feeling that Acronis does not believe me and that - even if I can give detailled information - I must deliver information that needs me hours to collect ?
As mentioned each support case I had in the future never got solved by this procedure...
Sorry that I am somewhat dissatified, but if I give detailled information on how to reproduce the issue I cannot understand why Acronis does not check this on its own to improve product quality.
I am working in a production system and each test gives me entries in my dashboard that I can never get rid of.
Funfact:
This is one of the upper mentioned support cases where all process logs and files I sent to Acronis end in the answer "well to cleanup teh database please reinstall the product"...that this would also end in a loss of all logs, statsistics and so on of my production system was commented with "well either you delete all or you must live with the orphans"....
Sven
Hello Sven.
Acronis is not willing to try to reproduce the problem independently
We can't reproduce your situation because we do not have your backup plan details and your environment. That is why we request these logs which will show us the time for reading (validation) task and the size of the data so that we can analyze these data.
As mentioned each support case I had in the future never got solved by this procedure...
If you do not wish to open a case, please upload your Process Monitor logs along with Acronis System Information gathered from the server with the Agent which runs all these validation tasks to the FTP link that I've sent you in a Private Message.
Hey Maria,
yes I understand that you have not my plan, not my infrastructure and so on.
But as mentioned, in the past no support case for which I created these time consuming logs lead to other "solution" than the hint to reinstall Acronis....
But come back to the current problem.
I can reproduce it clearly what happens, and I can describe you how to reproduce it:
Prepare souce and target
a) Create the backup source
I created a folder DUMMY_SRC on the HDD of a machine with about 20GB data in this folder.
This was done by copying 4 DVD ISO images in ths folder.
b) Create 2 backup targets
On a different HDD I create 2 folders STOR_LOC1 andd STOR_LOC2.
c) switched off Antivir for these folders
--> so there is no difference in speed for the 2 storage locations as they are on the same HDD
--> no negative effect of antivir
Backup job
I created a new backup plan with the following features
- agent is the machin eon which the upper mentioned folders reside
- first DUMMY_SRC is backuped to STOR_LOC1
- STOR_LOC2 is the second location
- in both locations 6 backups are held during retention rules
--> as there is no network used the LAN speed has no impact on validation speed
--> as the plan is completely performed by the machine on ehich the source and target reside, there are no performance impacts/differences
Run the plan manually
I then manually started the plan 4 times.
Result (watched the ativity log of the 4th run)
I) time needed for backup to STOR_LOC1 and STOR_LOC2 only has small differences
II) first validation needs 2:47....second validation needs 42 seconds
That is exactly the same effect that I saw and described in my first post.
Analysis
Issue I) --> small differences in backup time
What I can see in Ressource Monitor of Windows is that the first backup (DUMMY_SRC --> STOR_LOC1) first creates a snapshot via VSS, reads from that snapshot and writes the data to the tibx file.
The second backup (STOR_LOC1 --> STOR_LOC2) reads the tibx file at STOR_LOC1 and writes to STOR_LOC2.
It ois not a pure copy as the file sizes differs slightly.
So the tie difference can be explained.
Now the more interesting issue:
Issue II) --> validation time differs drastically
During validation I monitored file accesses with the Windows Ressource monitor.
The first validation is done on STOR_LOC2.
Each tibx-file is read....beginning from Validation_Test.tibx to Validation_Test-0003.tibx.
The second validation is done on STOR_LOC1.
And there.....only Validation_Test-0003.tibx is read!
So for me this is the proof of 3 things:
1) validation starts with the second storage location going to the first
2) validation at second storage location validates everything
3) validation at first storage location only validates the last backup
So my questions are
- Is my description precise enough that Acronis an reproduce this issue or must I open a support case?
- what is the reason for 1) ? This is not a big deal, but I think most of us expect the validation order to be the same order as the backup itself.
- Why ist the difference in validation ? This is a severe bug for me.....
Regards
S.
Hmmm,
honestly a bit disappointing, that there is no response anmore.
I did a detailled analysis and - in my eyes - the proof of a misbehaviour and now there is silence....
S.
Hello Sven.
We still need Acronis System Information and Process Monitor Logs collected from the Agent which performed validation. We need to analyze them for further investigation and testing purposes.
Otherwise, we can neither confirm nor disprove software misbehavior.
If you do not wish to open a case with Acronis Support Team (our engineers could organize testing in similar environment if it is required according to the log analysis), I have sent you an FTP link where you can upload the logs.
Maria,
sorry I know that it´s not you that creates these rules, but really?
I gave the support-guys a detailled description how the problem can be reproduced and I am still forced to spend more time to create logs that never had any result other than "we do not know what´s happening, please reinstall all".
That´s really weird. And I can promise:
My test system is still on "update-3" and the first will be "please update to the newest version"....so again more time for me as a customer with paid (!) support that I have to spend to get any form of support.
Really very diappointing
S.
Hello Sven.
I gave the support-guys a detailled description how the problem can be reproduced and I am still forced to spend more time to create logs that never had any result other than "we do not know what´s happening, please reinstall all".
If you have already opened a support case then you please share it's number so I could look into this situation. I also recommend that you share this forum thread with support engineers.
That´s really weird. And I can promise:
My test system is still on "update-3" and the first will be "please update to the newest version"....so again more time for me as a customer with paid (!) support that I have to spend to get any form of support.
That is up to you to decide which software build to use on your system. Unless you request any testing/reproducing/fixing you could use any version of the update. However, nobody will reproduce any misbehavior and fix peculiarities in old updates because engineers work on improving the current build.
Maria,
Well I thought the detailled description in this forum may have the result of a support engineer to reproduce it.
To shorten it :
- I created a support case with the number 04191112 and wrote a link to this thread
- I uploaded the process log ans system info to the FTP
I hope that now anyone looks into the matter and I get a response within the next few days.
Regards
Sven
Hello Sven,
thank you for sharing the details.
Your case has been escalated to Experts Team.
Hmm after one week the ticket is updated and the issue is examined by their professional.
One week between opening a case and the information that it is investigated...if I would have this delay in my job my boss ..... well...let´s wait and see how fast they are.
S.
Hello Sven.
There is a testing task created with all the details that you have carefully described, collected and shared with us. Once testing engineers recreate similar environment and reproduce (or not) the validation time issue, you will receive a reply in the case. Please note that testing tasks always require more time than most of the other support issues.
We thank you for your cooperation and attention to this issue.