B&R10 Indexing Task \ Deduplication

According to the userguide, indexing task, essentially, *deduplicate*, replacing duplicated data with its index in the deduplicated storage.
UG also contains recommendation to stagger full backups if data is partially similar for all severs, and to wait for the index task completion before starting subsequent full backup.

Thanks dev-anon, and I would love for Acronis to comment.
Acronis, help..
Does the indexing need to finish on an all archive before I can restore?
I can't believe how long this indexing takes. On one server, #4 listed above, it has been running for 16 hours and estimated time to finish is 9 days - NINE DAYS. (The indexing on the Full)

I have the 5 servers above in one policy, the "simple" where it creates 1 full then incrementals the rest of the time then consolidation for retention policy.

Please explain then indexing process. Is there one indexing process for each backup?
OR
Does the indexing process see that there have been multiple backups since the last indexing task finished and index what is waiting? (Meaning ANY remaining indexing tasks?)

In my case, I have a server right now that is going to take NINE DAYS to finish indexing the full. In the mean time, I have all these incrementals that are going to run over the 9 days. Does that mean there will be 45+ indexing tasks waiting? (5 servers, 1 incremental per server per day = 45 indexes to be done.)
On top of that, the consolidation is going to run on day 6 and I have NO IDEA what's going to happen.
And on top of THAT, I have 2 archives that the validation failed. Does that mean the really are bad? Or does that mean when all the indexing finishes, the archive is ok?

UPDATE: On the failed archives (full + 1 or 2 incrementals), I tried to restore to new VM - fail. I tried to validate the archive - fail. I tried to view the contents of the archive - fail. On each of these I was selecting the latest incremental. I tried just selecting the full, validate the archive - success, view the contents of the archive - success. I'm going to try a restore and I'll let you know.

So what this tells me is that the archive in a deduplication vault is not valid or usable until after it indexes. Is this true?

Hello Jeff and dev-anon!

First of all thank you for opening the thread and many thanks to dev-anon for his valuable comment!

I'm very sorry for you had to wait for the reply and would like to notify that in case of emergency you can contact support directly. Anyway I will be glad to provide you with the answers on your questions.

First of all please check whether you are using the latest build (it should be 11345). It is available at your account here.

Indexing task is a part of deduplication process  (1. indexing task 2. compacting task). You can find the detailed description on how it works on page 75 in the User's Guide. The indexing task may take a considerable time to complete, and it depends on various circumstances like archive size, location of the vault, etc.

I have contacted the Experts' team for additional details, and would like to assure you that restore feature is still available when indexing is running. You can export the slices from the vault as well, but deleting or starting the new backup will cancel the indexing process.

Indexing is performed for the chain of the backups starting from the latest full image. Thus it starts the task for the full, than for the Incremental_1, than Incremental_2 and so until there are not backups left in the chain.

In order to increase productivity of the task, our experts suggest the following: 

1. Place the the storage path and the database path (you are offered to choose it when creating the vault) on different location
2. Locate ASN on the separate machine with - preferably - 2Gb of RAM
3. Create shorter chains of the backups (more fulls, less incrementals)
4. Storage should have a lot of space for proper running of indexing task (about 350Gb if possible).
   It is required due to the cloning of the chain: when indexing is in process, the chain is cloned for granting the safety of the backups, so it may take up to "chain+10%" space

According to your description it looks like you are running our of free space, thus indexing task often restarts and it takes a lot of time for it to be finished. I would appreciate if you could kindly follow the below suggestions to test whether it speeds up the process. I entirely understand how inconvenient it may be, but we are currently working on improving the situation as well as on the documentation regarding the recommended settings and configuration for the Deduplication.

I would also advise you to check chapter 2.13.6 in the User's Guide. It contains all the information regarding deduplication feature, and you may find it useful.

As per validation and recovery issue: I'm positive its not connected with the deduplication, so we will need to investigate it separately.

As far as I understood you have already tried validating and mounting the image. Since it didn't work the backups are corrupted for sure. There are several possible reasons: 

1. The source location is corrupted.
2. The target location is corrupted
3. There is a bad connection between the source machine and the vault.

In order to avoid it I would advise you to perform the following steps: 

1. Run the chkdsk command on the source machine
   Start -> Run -> cmd
   chkdsk /r /f
   Please note that this operation will require a reboot.
2. Run the same chkdsk command on the machine holding the vault
3. If possible, please test the backup to any other location - this will help us to localize the issue.

This information will help us to localize and - if the issue was with source or target location - solve the problem.

Should the issue persist, we will need to take a closer look at the issue, so could you please kindly send us the following information? 

1. Acronis Info from the source machine
2. Acronis report from the source, target and holding the vault machines

I would appreciate if you could kindly keep us updated and if you could let us know whether something is still confusing you or you have any additional questions - we will be glad to help!

I will keep tracking this thread for news from you!

Thank you!

Some comments, questions, etc...

As I understand it (per your explanation and user guide), "indexing" IS deduplication. In order to effectively deduplicate, you must have available free space that is equal to the entire archive backup chain + 10%. IF NOT, indexing will either not run OR will run but take a long time.
----------------------
You said:
In order to increase productivity of the task, our experts suggest the following:
1) Place the the storage path and the database path (you are offered to choose it when creating the vault) on different location
2) Locate ASN on the separate machine with - preferably - 2Gb of RAM
3) Create shorter chains of the backups (more fulls, less incrementals)
----------------------
Here is more info to help you:
I'm using B&R10 Advanced Server Virtual Edition and my AMS, ALS, and SN are all on 1 physical server - Windows 2003 Standard with 2GB of RAM. The drives are 10K RPM drives in a RAID 5 configuration. These drives are also configured as 1 logical drive with 2 partitions, C: (35GB) and D: (648GB). D: is where the Acronis vault with dedup is located. On 'D:' I have 2 folders, D:\Acronis Backups\ServersBackup (storage path) and D:\Acronis Backups\ServersBackupDatabase (database path).
So #1) They are different locations, technically, but you may be talking about different disks. I don't have that option, most people probably do not and from your developers viewpoint, they should take that into consideration.
#2) As I mentioned above, my server has 2GB of RAM.
#3) Unfortunately, I don't have the space for a short chain. That is something I will try to address.

What I have done:
I have deleted all my archives, policies and vault. I have recreated the vault and I'm currently backing up server #5 only (from above). From past testing, server #5 seems to be the one that takes up the most archive space. I will let the backup finish and the indexing. I will then slowly add in servers to see how it responds.

What I hope Acronis works on:
Let's be honest, the indexing task is PAINFULLY slow. If the indexing task is currently running, and a backup job runs, somewhere the indexing tasks get queued, because it runs 1 indexing task per backup job. If these tasks get queued up and there are backup jobs that are running or 'waiting' performance becomes so bad, the backup software is almost unusable. If I were a developer, I would create a way for the indexing task to (a) run faster, and (b) indexing any backup jobs that have completed up to the point the next index is to start.
Example, backup job 1 completes and the indexing task starts. While the index task is running, backup jobs 2, 3, 4 complete and backup job 5 is running. The indexing task finishes and restarts and indexes jobs 2, 3, AND 4 - (rather than just index backup job 2 like it does now). This would eliminate the need for 1 indexing task per backup job and make it more efficient.

I know my suggestions will probably not make it to the correct people, but I am the real world user and sharing my experience. I believe in your product, but often times it feels like I'm the test subject. I don't mind helping, but it becomes frustrating at times.

Hopefully you have more information to help me and I will keep you informed of how the backup proceeds from this point.

Ok, I got in this morning and the backup of server #5 from above finished in half the time it previously took because no other processes or tasks were running. That's the good news.
The bad news:
The indexing has failed because it doesn't have enough free space in the vault. My vault has 648.6GB of total space. Server #5 backup took 363.4GB of space and I have 285.2 free space.
This leads me to the following comments:
The deduplication, or indexing task, is inefficient. For starters, the indexing (deduplication) task runs once for every backup job. When other tasks are running or waiting, the indexing will take twice as long - minimum. So, you have backup jobs that run while the indexing task is running and the indexing task never stops as it gets queued up from all the jobs that run. So my example in the above post of how to fix it should be a top priority for the developers. Also, in a real world scenario, the requirements of 110% of the archive chain is unrealistic. I realize that I could have more space for backups, but let's think about this: Using deduplication, ANYONE who uses this feature only has LESS THAN HALF the space they think they do. Not only that, but the reporting of space in the vault is misleading. (See attached picture). While the reported numbers are accurate, it does not take into account the deduplication needs. In my case, I have 4 remaining servers that I cannot backup because of deduplication requirements.

I am VERY frustrated and Acronis needs to hire me as a consultant.
I AM your real world scenario, and while the program works, it's falling short of expectation in terms of "realistic" expectations and efficiency.

Now I know you're going to apologize and be polite so I am going to thank you in advance for that, but what I would appreciate more is to know that the main developers get my suggestions. I will have no way of truly knowing that happens.

For me personally, I will have to look at getting more storage space so I can make YOUR program work properly. Lots of money and time wasted. If there is anything good from this, I know your product very well now. Unfortunately, I don't work for Acronis so I can't help with my knowledge.

Dear Jeff,

Thank you for your detailed description and sincere feedback, we really appreciate that! I completely understand how inconvenient and upsetting current state of the problem is, and would like to assure you that not only we are carefully reading and taking under consideration your suggestions and concerns, but Development team is notified as well, and I believe that we will receive news from them soon and even have them posted here.

I completely agree that deduplication is too slow, and - according to the Experts team - we are working on improving the situation. Still Indexing task will be a resource-intensive process, and it is difficult to guarantee that we will be able to speed up it significantly (though that it our aim at the moment).

You suggestion regarding indexing task running at the background of the backup process is really valuable. Unfortunately we won't be able to implement it for current update due to the Scheduler limitations: it will not start the second task while the first one is in process. Still indexing task doesn't affect backup creation: should one start the backup process, indexing gets canceled and it is not started till the backup process is finished. And here comes the tough part: 

due to the deduplication process design, indexing task starts not for the last slice created, but for the entire chain. For instance: 

Step 1: You have a full backup created. Indexing task is running for that full backup.

Step 2: You have a full and an incremental_1 backup created. Indexing task runs first for the full backup, than for the incremental_1.

Step 3: You have a full, incremental_1 and incremental_2 created. Indexing task runs for the full backup, for the incremetal_1 and onlt after that for the incremental_2 backup.

Thus not the last backup is deduplicated, but the entire chain you have in the storage.

Hopefully we will get this improved as well - and I have forwarded the request to the development team for clarification, so I will keep you posted.

I will also appreciate if you could kindly share your feedback and concerns further on - your opinion is valuable for us, as you have correctly mentioned- you are our "real world scenario", for you we are designing software and we need to make it meet your expectations.

Thank you for your patience and cooperation!

Maybe I wasn't as clear as I could have been.

Indexing:
What it appears (to me) is happening is that the indexing task runs one time per backup. Example, Let's say I have 1 backup policy for 3 servers. It is scheduled to create 3 Fulls, one for each server. My understanding is that the indexing will start after the 1st full is completed on the 1st server. While the 1st indexing task is running, the other 2 fulls start AND finish. Now the 1st indexing finishes. Then indexing automatically starts again to index the 2nd full. When it finishes, indexing runs again for the 3rd full. That's what it appears is happening.

What I am suggesting is, when the 1st indexing task is finished, it checks to see how many backup jobs have finished since the last index start, so that the next index task will index ALL the jobs that FINISHED DURING THE FIRST INDEX, not just the next backup.

Additionally, the space needed for deduplication is just too much. In my case, you are cutting the available storage space in half. Remember, in a virtual environment, they are running on SAN's. You have to make the assumption from a development viewpoint, that users are going to have large servers, like myself.

I have the same issue with the index task. I've been fighting with it for over a month now. I've upgraded hardware and followed the best practice in the manual. Here is what I have now:

ABR10 Advanced Server and Workstation with dedup and univ. restore options - 3 servers 2 workstations

The full suite is installed on one server. The vault is on a iSCSI SAN with 12TB space in RAID6 (8x2TB drives). Currently this vault is the only thing on this SAN. The vault db is on a local RAID0 volume 385GB total and currently has 376GB free; the db is currently at 9.3GB.

I ran one full backup of my main file server (1.5TB) and am letting the indexing finish before proceeding with adding the other servers and workstations. So far the indexing has been running for 2 days 14 hours, is 12% complete and the GUI says it has another 19 days 1 hour to go.

So, I started perfmon to see what is going on. I see that the RAID0 db volume is getting hit the most with a sustained I/O rate of about 130 /sec and they are almost all read I/O; this puts the volume at about 98% time utilization. Only every great once in a while maybe once a hour or so there is a quick burst of write I/O for 2-3 seconds and that's it. The vault volume has a low I/O rate of about 3 reads per sec and no writes. The disk read queue is at 1 continuously for the db volume, and near zero for the vault volume. The disk write queue is zero most of the time.

This is an HP Storage Server 2003 x32, with 2.66GHz Dual Core Xeon and 3.37GB RAM (not all 4GB can be utilized on x32 platforms). CPU utilization is averaging 6% with occasional spikes to 23% or so. RAM is at 77% utilization.

I've opened a ticket with Acronis support and they promised to contact me. I also contacted my Acronis account manager Chris Lee as I'm a VAR and if this is not resolved will not be purchasing anymore deduping licenses. For clients that do require deduping I'll may be forced to look at other solutions. We'll see how Acronis comes through. I just had this client spend about $15k on hardware and software and the software part is not working as promised. I definitely feel that Acronis has let me down and acknowledge it was my fault for just trusting that deduping would work properly instead of testing it myself. It's a new feature for Acronis and I should have tested it before selling it. But that simply comes from my confidence in Acronis' past products. I've been selling their products to clients for three years and it has worked well. This current iteration, at least the deduping portion of it, is falling flat. We'll see how this story ends as they promised to call me today to gather more information on this issue. I'll keep everyone posted.

Still haven't heard back from Acronis support even though they promised to call me yesterday. I suppose they're busy responding to all the complaints about dedup not working.

Still haven't heard back from them . . .

Hello all,

Thank you for bringing up this issue.

I would like to let you know, that we have another topic where we discuss indexing/deduplication issues. It is available here. You can also ask our Development team directly, we have requested their assistance.

Roman,

I am terribly sorry for the delay. I have replied to your post here, if you can get back to me I will do my best to assist you.

Please let me know if you have additional questions.

Thank you.