B&R 10 Build 11105 tasks lock and bad index performance on centralized vault with dedupe

After 4 hours we are at 7% of the same 60GB on disk or 180 GB Backup

Hello Raoul,

Thank you for your interest in [[http://www.acronis.com/enterprise/ | Acronis Corporate Products]]

An indexing task requires a computer of a high performance. It looks like the operation runs very slow, the program is not freezing actually.

I would like to share some recommendations on how to set a deduplication task, please create a new task based on these recommendations and check the performance:

1) The computer with deduplicating vault or Storage Node should be of a high performance, please disable another running applications if it is possible.

2) Please avoid to place the deduplicating vault onto network devices.

3) It is preferable to place a deduplicating vault onto a speedy hard drive (IDE hard drives are not recommended in this case).

4) While creating a deduplicating vault you should specify a local folder on the storage server to create a vault-specific database. This database will store the metadata required for cataloguing the archives and performing deduplication. It is recommended to place this local folder onto another speedy hard drive.

Please note that the first deduplication requires more time than the subsequent ones.

Please let us know the results.

Thank you.

Hi Oleg,

1 year old amd 64 bit processor, running windows 2008 and 4gb memory + raid 5 should do this job fast enough on a backup of 168 gb i suppose not?

Raoul

I have the same performance issue ...
I'm trying demo version of ABR 10 advanced server ( build 10.0.11105 ) on a dual core pentium E2160 - 1.80 GHz. with 4Gb RAM. Storage node is on the same PC and use a dedicated 160GB HD.
Backup speed is not bad, but the deduplication process work ad about 1GB/hour, so the subsequent backup process ( next work day ) start with deduplication process active, becaming much more slow than first backup.
StorageServer.exe works at 10% max of cpu, even changing priority.
Is the something i've missed ?

Paolo

I'm experiencing the same issues using build 11345. Storage node is on the same machine 2TB drive with a seperate fast drive for the dedup database. currently at 76% after 3 day and 19 hours of indexing 118GB. I've tried contacting support but they waste my time. can anyone help me please?

I am having the same problem. I have had an indexing task running for 3 days now and it's only 18% with an estimated 2 weeks to complete!!! All of my jobs are in a waiting period while this thing runs. Completely Unacceptable!

Hello all,

Thank you for posting, I will be happy to help.

I am having the same issue and have set up the storage node per the recommendations in this thread.

Where I am now:
I have 5 virtual servers total. Currently the 3 smallest (in size) are being backed up. After creating the storage node and centralize managed vault with deduplication, I backed up 2 servers, full backups, and it ran VERY fast. 20 min for one, 40 min the other, Great. The indexing started, I ran a full backup on the 3rd server, about the same size as the one that took 40 min, and it has been running for 15 hours and is at 59%. Unacceptable. The indexing has been running for 15 hours and 50 min and is 40%.
I am praying that once the indexing\deduplication finishes that it gets faster. I still have 2 servers to go and they are large in size, one of them 1TB and another 700GB.
Any of you have any more info to offer?
Thanks

I am experienceing the same problem. indexing have a severe hit on backup performance, and indexing takes very long.
running the latest version 10.0.11639 of advanced server 10 with universal restore and deduplication.

since i am going to aquire a new machine for backup i wanted to determine what is the bottleneck for the indexing task. But i am unable to determine the reason for indexing to take so long. So i also don't know what type of hardware to purchase.

cpu a quad xeon 2.13 is almost never above 50% on any core.
ram is around 2GB usage out of 6GB
diskIO is at about 3MB/sec.
network IO is around 150Mb with occasional peek at about 300 Mb (a file transfer between the machines reaches easily 850Mbits)

what exactly is limiting the indexing ? where would my money be best spent to improve acronis performance

RonnyA

The indexing task IS the deduplication process. The Acronis software works pretty well for me, but the deduplication is a wasted feature as it is not efficient. It is so bad I don't use it anymore since my original testing. To answer your question, the bottleneck for indexing is the task itself and has little to do with hardware or anything else. My server specs are less\older than yours and I'm working just fine without the dedup.

If you have anymore questions, let me know - I'm very familiar with this software through all the testing and bugs I've worked thru. If it relates to me in any way, I can probably tell you about it.

Thanks for your speedy reply :)

I suspect i may have to diverse windows machines for deduplication to make sense. userdata and applications are not uniform, and there are 3-4 windows OS versions in use. I was hopeing to leverage deduplication to take more full backups.

It's a pity since i use deduplication for my linux machines in a linux tool called backuppc, where it works very good. And the space savings are huge when you have a long history with many full backups..

I need speedy backups in order to take backups of all machines during the night. excluding deduplication, would there be performance differences between a managed and a unmanaged vault ? (should perhaps ask this as a different question/thread)

Ronny Aasen

Anton Ostapets wrote:

Hello Ronny,

What is your disk subsystem configuration and how it bound with your vault storage and database?

RonnyA wrote:

what exactly is limiting the indexing ?

Your disk subsystem is limiting it. You should place the database to a separate HDD array with not only fast transfer rate but with extremely fast disk response time too (e.g. enterprise SSD drives or even RAID0-arrays builded on SSD).

Even still, its very slow. the indexing task runs once for every backup. So expect it to always be running if you have a lot of backups. Don't get me wrong, dedup does work, it's just not very efficient and can be very slow. You also need 110% of your backup archive size in available free space or the indexing will fail. I had this problem on a very larger server. Same thing goes for consolidation of archives, 110% rule.

RonnyA wrote:

Thanks for your speedy reply :)

I suspect i may have to diverse windows machines for deduplication to make sense. userdata and applications are not uniform, and there are 3-4 windows OS versions in use. I was hopeing to leverage deduplication to take more full backups.

It's a pity since i use deduplication for my linux machines in a linux tool called backuppc, where it works very good. And the space savings are huge when you have a long history with many full backups..

I need speedy backups in order to take backups of all machines during the night. excluding deduplication, would there be performance differences between a managed and a unmanaged vault ? (should perhaps ask this as a different question/thread)

Ronny Aasen

Not that I have seen. I use both with separate acronis products and have no issues. I use managed with my virtual edition product and unmanaged with my standalone.

Anton Ostapets wrote:

Hello Ronny,

What is your disk subsystem configuration and how it bound with your vault storage and database?

RonnyA wrote:

what exactly is limiting the indexing ?

Your disk subsystem is limiting it. You should place the database to a separate HDD array with not only fast transfer rate but with extremely fast disk response time too (e.g. enterprise SSD drives or even RAID0-arrays builded on SSD).

thanks for all your replies.
I have tried a raid1 set of spinning disks, and a raid1 set of ssd's not much difference. but the raid is a onboard el-cheapo. so that might be the reason for that. the strange thing is that the difference between spinning and ssd was non exsistant. i will try raid10 or 50 in the future.

i wish this was better documented by acronis. what steps to take to tune acronis for faster backups.

I know they recomend 1 full backup and then let the indexing task finish, before adding more machine. But one cant in the real world have no backups for days while the indexing does it job. especialy if you have a corrupt vault or for some reason need to start on a clean sheet.

Ronny Aasen

the database was on a raid1 of 2x ssd's it's not on a raid1 of 2x sata drives.
the vault is on a open-e NAS with 16 500GB disks in raid 50 (gigabit network between),

when observing the storage server process with only the indexing task running. it's consistently around 3MB/sec I feel this is kinda low. It's almost like acronis writes many tiny blocks all the time. ?

if i understand the indexing correctly, it reads the vault, transfer it across the network. runs calculations on data, and writes results into the database.
unfortunatly. there is not much one can do to improve read speed on the nas except more or faster disc's and both of those are quite expencive and disruptive.

RonnyA

I too have been struggling with slow indexing. My vault DB is on a Raid5 array of (5) 10K SAS disks with an LSI serveraid controller (IBM MR10i). This is a powerful raid controller but I just realized that without the optional battery backup on the controller which I did not initially order, the logical disks are set to WriteThrough caching. I just added the battery option and changed the logical drive to WriteBack. Now indexing appears to be running at least 10 times faster!

Acronis must be making tons of tiny writes to the vault DB...

The backup storage is on a NAS device (Netgear Readynas 3200). The purpose of my post was to emphasize to anyone with indexing performance woes the importance of the DB disk controller settings. The difference in performace between WriteThrough and WriteBack caching and it's impact on Acronis Vault Indexing speed is dramatic.

Recreating the database, while possible, does take a huge time. I tried it once but gave up after about 2 days. after all i need the system to do backups as well. It's definitiftly not something you want to start with when you need a restore right away.

When doing deduplication vaults...

Is it better to have multiple vaults with similar systems inside (eg only windows2003 sp2) ?
or one vault with all systems inside ?

i guess if one could put the databases on different drives it might make sense to have multiple ? but with the database on the same drive the concurrently running indexes would probably slow the thing to down even furter ?

Ronny

I'd like to hear from one of the engineers at acronis their recommendations on best practices regarding dedup vault creation, size, scalablility, as well as hardware recommendations and configurations. Number of client servers or total storage volume per storage node recommendations.

I agree with RonnyA on having multiple vaults dedicated to similar systems. More, smaller vaults are going to be easier to deal with that fewer, larger vaults. If a vault blows up, less data is lost, re-indexing is faster, ect.

It is also possible to have multiple storage nodes if you have a lot of vaults. Likewise, the management server can be seperate from the storage nodes which I'm beginning to think might be a good idea. The management server has an SQL installation that eats RAM, CPU and Disk IO. If you have a single machine that is the management server and storage node, it's doing double duty and accessing things like the task list and event log become sluggish when the storage node function is indexing a vault on the same box.

I have the same issue with the index task. I've been fighting with it for over a month now. I've upgraded hardware and followed the best practice in the manual. Here is what I have now:

ABR10 Advanced Server and Workstation with dedup and univ. restore options - 3 servers 2 workstations

The full suite is installed on one server. The vault is on a iSCSI SAN with 12TB space in RAID6 (8x2TB drives). Currently this vault is the only thing on this SAN. The vault db is on a local RAID0 volume 385GB total and currently has 376GB free; the db is currently at 9.3GB.

I ran one full backup of my main file server (1.5TB) and am letting the indexing finish before proceeding with adding the other servers and workstations. So far the indexing has been running for 2 days 14 hours, is 12% complete and the GUI says it has another 19 days 1 hour to go.

So, I started perfmon to see what is going on. I see that the RAID0 db volume is getting hit the most with a sustained I/O rate of about 130 /sec and they are almost all read I/O; this puts the volume at about 98% time utilization. Only every great once in a while maybe once a hour or so there is a quick burst of write I/O for 2-3 seconds and that's it. The vault volume has a low I/O rate of about 3 reads per sec and no writes. The disk read queue is at 1 continuously for the db volume, and near zero for the vault volume. The disk write queue is zero most of the time.

This is an HP Storage Server 2003 x32, with 2.66GHz Dual Core Xeon and 3.37GB RAM (not all 4GB can be utilized on x32 platforms). CPU utilization is averaging 6% with occasional spikes to 23% or so. RAM is at 77% utilization.

I've opened a ticket with Acronis support and they promised to contact me. I also contacted my Acronis account manager Chris Lee as I'm a VAR and if this is not resolved will not be purchasing anymore deduping licenses. For clients that do require deduping I'll may be forced to look at other solutions. We'll see how Acronis comes through. I just had this client spend about $15k on hardware and software and the software part is not working as promised. I definitely feel that Acronis has let me down and acknowledge it was my fault for just trusting that deduping would work properly instead of testing it myself. It's a new feature for Acronis and I should have tested it before selling it. But that simply comes from my confidence in Acronis' past products. I've been selling their products to clients for three years and it has worked well. This current iteration, at least the deduping portion of it, is falling flat. We'll see how this story ends as they promised to call me today to gather more information on this issue. I'll keep everyone posted.

sad to hear such stories. guess there is no "buying your way" out of this issue. if it does not work without issues with your hardware. it's never going to work for my lesser hardware.

Also beeing a VAR. and made a support request with acronis. But I did not have the resources to dedicate such large hardware costs to test a product we pay for. And need to have operational. So i simply had to run without deduplication. and could not work with acronis support to test the issue. hope acronis looks into this seriously anyway tho. And come up with a working deduplication solution.

currently i use only unmanaged vaults.

RonnyA

I still used a managed vault but don't enable the dedup option. I'm not sure what benefit using a managed vault over an unmanaged one yields in this scenario but it works. I haven't tried using an unmanaged vault but I imagine you don't get to use the GUI to see and manage the backup chains. I use the GFS backup scheme with weekly diffs, daily inc, and monthly full. The GUI lets me see what the chain looks like and I can run validation tasks from any point in the chain. Can you do the same with an unmanaged vault?

Also, Acronis never called me back yesterday as promised. Not even an email to explain themselves.

Yes i do see the backups in the vault. much in the same way i do with managed vaults. but i also notice the existance of some wierd entries, that are not easily explainable. But i belive they occure when i change a backup job.

I also use GFS scheme

I wanted to try a unmanaged vault, since i there can look in the file storage and identify the files.
I also have a installation with managed vault to compare...

nothing more fun then testing on production setups :s

RonnyA

I use a Managed Vault without dedup, since as we all have experienced, is not usable in the real world. My backup scheme is "Simple" consolidating the backups after 4 days. What I have noticed is after about 2 months, the consolidation (or "cleanup") task takes longer and longer, as do the backups. I'm using the Virtual Edition so I don't know if it's because of that, but I have to delete my archives and recreate them. Well, I don't HAVE to, but if I want it to be more quick I do.

I also have noticed the cleanup taking longer and longer. But the compacting task is what takes most time of all. i hve left it running for about 15 days before stopping it, but i have never yet seen it finish... Not sure what it's doing but it takes a lot of time.

also i notice that when i index is running, something eats up 100% ram on the server. while no process in the list uses remotely close to 12GB of ram the performance tab of the task manger shows close to 99% usage. and this drops to normal levels when the indexing finishes. or i restart the storage-node server. Does anyone else notice this RAM usage ?
(windows server 2008 r2 x64)

perhaps it's some kind of OS buffer, but then it's suprising that the disk thruput should be soo exceptionally poor.. :s

RonnyA

I experiance the same RAM consumption. I added additional RAM and no matter how much (16GB at this point), it all gets consumed by some phantom process that's not listed in the performance monitor. The addition of RAM made no difference to the indexing speed.

Haven't been running Acronis long enough for the compacting task to be an issue.

Take a look at CA Arcserve Backup and Arcserve D2D for an alternative.

I have a buffalo terastation III being used as a centralized vault attached to a management server/storage node. Dedupe is enabled currently on the vault. Indexing performance is horrendously slow.

As I understand it, there is now way to speed this up except by disabling dedupe on the centralized vault. Is this correct?

Hello,

First of all, thanks to everyone for your attention to Acronis software, I want to ensure we're constantly working on making it better and all your feedback is appreciated.

With regards to indexing issues I have the following comments:

1. Issues with indexing task not making any progress for a long time or just running too slow should be investigated for each particular case - Storage Node logs should be analyzed, system information should be collected, etc. , so I recommend to create Support requests for each such case - this will also help engineering to test some particular situations/environments to find out the reasons and provide solutions/fixes.
You should also take into account that indexing is in fact a permanent process running on a Storage Node in case there are subsequent backups performed to it, each of them queuing to be indexed after the previous backup indexing had finished.

2. As for general advises on how to get better performance of deduplication - as it was stated in this thread, some of them are reflected in product's Help/User Guide. The more detailed information on best practices along with a guideline for choosing hardware for Storage Node will be published this month in a separate white paper which we are currently preparing.

3. Indexing and compacting tasks can run in parallel, but their speed is slowed down due to each of them competitively accessing deduplication database. Also indexing task locks the archive that is being indexed and hence other operations will wait until the archive is unlocked, but operations on other archives in the same vault are possible.

4. Compacting task is deleting data from physical storage that has no references in deduplication database. F.e. if some backup was deleted (manually or by retention rules), all the references included in this backup get deleted from the database and then during compacting the data itself that has no references in the database gets deleted from the data store.

5. For upcoming releases we are working on improving the general deduplication performance.

Thanks.

--
Best Regards,
Alexey Ruslyakov,
Acronis Program Manager

thank your for your comments alexey. :)

could you also comment on the memory consumption issue me and some others in this thread have experienced ? is this 100% ram usage "normal" ?

Also i would like a clearification on the indexing speed. where is the border between "too slow" and "what should be expected" when it comes to indexing performance ? It's a bit hard to know if we have a real issue. or if we are just wasting time pursuing the issue
This will ofcourse depend on the usecase. But currently we don't even have a ballpark figure, of a normal indexing run to work with

kind regards
RonnyA

Hello,

As for the memory consumption - Storage Node is a 32-bit application and this means it cannot address more than 2 GB of RAM. To identify this "phantom" process you may try to use Sysinternals Process Explorer.

The average indexing performance we have in our labs is about 20 GB per hour on SATA 7200 RPM drives having a vault database located on a separate hard drive. So I can tell that having indexing speed 10 times slower is considered to be an issue.

--
Best Regards,
Alexey Ruslyakov,
Acronis Program Manager

Just FYI:
Is this to be expected from a centralized vault with one full backup that is approximately 200gbytes?
Please see attached file.

Just an FYI, I opened a support case for my Indexing performing too slow and when I was called about the issue I was assured that someone will call me later that day to get more information and they did not call. This was about mid last week. Today is Wednesday and still no one has contacted me. Unfortunately, this is about par for my experience with Acronis support. I've given up and have written off dedup as a lost investment. Even more unfortunate is that due to the poor experience with Support I'm forced to explore other product options for a new client also requesting a comprehensive backup solution. All I'm asking for is for you to follow up on your promises. We're a professional IT consulting company and do business with other professional organizations. Empty (but at least polite, I'll give you that) promises for action buys you time but in the end your customer feels doubly wronged: the product doesn't work as advertised and support is politely dismissive.

Roman Hanjiev wrote:

Just an FYI, I opened a support case for my Indexing performing too slow and when I was called about the issue I was assured that someone will call me later that day to get more information and they did not call. This was about mid last week. Today is Wednesday and still no one has contacted me. Unfortunately, this is about par for my experience with Acronis support. I've given up and have written off dedup as a lost investment. Even more unfortunate is that due to the poor experience with Support I'm forced to explore other product options for a new client also requesting a comprehensive backup solution. All I'm asking for is for you to follow up on your promises. We're a professional IT consulting company and do business with other professional organizations. Empty (but at least polite, I'll give you that) promises for action buys you time but in the end your customer feels doubly wronged: the product doesn't work as advertised and support is politely dismissive.

Completely agree.
I've given them the benefit of doubt and their product "just works ok" for me, but I've had to spend enormous amounts of time trying to balance the limitations of their product with real world scenarios. It seems their niche should be home users and VERY small businesses. Anything other than that and people are simply disappointed. I endorsed their product to my boss, we bought, and I've worked timelessly to not look bad. So all is well, BUT..I won't make that mistake twice. Thankfully dedup was a free add-on promo when I bought the Virtual Edition software. It's unusable, period.

Alexey wrote:

Hello,

As for the memory consumption - Storage Node is a 32-bit application and this means it cannot address more than 2 GB of RAM. To identify this "phantom" process you may try to use Sysinternals Process Explorer.

The average indexing performance we have in our labs is about 20 GB per hour on SATA 7200 RPM drives having a vault database located on a separate hard drive. So I can tell that having indexing speed 10 times slower is considered to be an issue.

--
Best Regards,
Alexey Ruslyakov,
Acronis Program Manager

I have looked at this memory consumption with many different tools, including process explorer, but I have not found what uses it. the ram is just lost.

ram-1.png shows the ram consumption after a day.
ram-2.png shows what happen when i stop and restart the storage-node service. (i have tried restarting most of the other services as well, but it's only storage-node that have the effect of releasing the memory)

in ram-1.png you can also see that there have been a compacting task running for 1day 22h and is at 28%. yesterday (when it had been running for 23h) it was at 25%, with this rate it would have taken a while to finish. I stopped it before restarting the storage node, but it did not in itself release the RAM usage. also the ram usage was this high before the compacting task started. So that task is unrelated to the ram usage.

in ram-1.png you can also see down in the right hand corner 7 processes running. but that's not correct. there was no tasks running except compacting. the running tasks indicator there is just never realtime updated.

RonnyA

more images of the memory comsumption of the index runs.

ram-3.png shows shortly after a index run starts. it goes quickly up to 45% and holds for a long time.
ram-4.png shows the slow progress of the index run. the ram usage grows slowly but steadily.
ram-5.png index progress ram usage grown to over 8GB
ram-6.png index run finished took about 3 hours. but ram usage is still high. a second index run would consume the rest of the memory at this time. Restarting the storage node service free's all the memory, as usual.

Also i do not know how much data the index run have processed in those 3 hour's to compare the speed of your normal 20GB/hour.
the disk in question was a 128GB disk with 28GB in use. but it was a incremental backup so the backup data size was 5.393GB and the uniqe data was 3.299GB not sure what value to use to determine the speed of the index run.

Hope you can use these to find the issue. let me know if you want a remote desktop or something to look closer at the machine. I have allready sent support 3 or 4 acronisinfo dumps so you should have those on file.

RonnyA

Hello all,

Thank you very much for all your comments and feedback.

Roman,

I am terribly sorry for what has happened. I was not able to find any cases for your registered e-mail account at Acronis relating to deduplication issues, if you can send me a private message with the case number or any other information that can help us find this ticket, I will forward this to our Management team who will handle it with high priority. Once again, please accept my apologies for the inconvenience.

Jeff,

I am very sorry that deduplication did not work for you as expected, we constantly try to improve our programs. I was not able to find any cases related to deduplication unfortunately. You always more then welcome to contact our support and we will do our best to resolve the issue.

Please let me know if there is anything else I can do for you.

Thank you.

That's just it Anton, the support is not good. I appreciate the post, but it goes along with what Roman Hanjiev stated above - you are polite yet dismissive. I know you guys are trying to improve your product, but it's at our expense. No thank you.

It may be the Acronis SQL instance that is consuming so much RAM? Or the aggregate of SQL, StorageServer, etc.?

Anton, thanks for the offer, but no thanks. I've spent too much time on dedup and must move on.

Roman Hanjiev wrote:

It may be the Acronis SQL instance that is consuming so much RAM? Or the aggregate of SQL, StorageServer, etc.?

I would belive this should be reflected in the taskmanager, or process explorer ? there is nothing even close to using these amounts of ram.

I see the same thing on my machine - uses all 12gb ram in server 2008/64, but looking in taskmgr, nothing accounts for it.

Killing the index task, drops the memory usage back to normal levels.

20gb per hour is completely unrealistic in today's environment. My backup set is 1.5tb - at that 20gb/hr rate, it would take 3 days to make a pass on my vault ??!?

That's really just ludicrous. Even small shops are backing up hundreds of gigabytes. Even 500 gigabytes would extend indexing time past 24 hours - and that doesn't account for the compacting time.

Really??

Then perhaps it's the system cache? The OS will cache files in RAM for faster retrieval if requested again. This will be reflected in the Task Manager, under the Performance tab, at the bottom is a section called Physical Memory with lines for Total, Cached, Available, Free.

looks like cached shows like available ram in the graph

1. installation: a week uptime have 100% ram usage
total : 12279
cache : 105
available : 110
free : 5
usage 99%

2. installation, (recently rebooted, have still some available ram)
total : 12279
cached : 2024
available : 2808
free : 780
usage 9.27GB (78%)

Ronny

I just found this thread. First I posted my story there. But now I guess I have to follow these two threads because I experiance all the same problems.

http://forum.acronis.com/forum/11543

Hello!

Does anybody have any news about this problem. Its driving me crazy and its slowing everything down...
German Acronis Support seems not to understand....

Greetings!

not to my knowledge no. We have just stopped to use deducplication until there is a working solution.

RonnyA

I am going to stop using dedup.

After indexing finishes, can take days, if you delete/cleanup some of the backups, indexing starts all over again. You can waste a huge amount of time trying to make this work. It is unworkable with 3 WinXP workstations in one dedup vault.

The methods used need to be re designed and optimized. Current approach is deeply flawed. Deeper database expertise is needed and the answer may lie in vastly different methods. List processing is probably a viable approach.

There needs to be a review process before things like this get released into the market place.

We also quit trying to use dedup over a year ago. It made the question of restoring during a disaster very questionable seeing as it never quite worked to begin with.

Does the integrity of the indexes impact the ability of one's ability to restore or only affect how file dedup [single file copies] is executed? I wish the architecture was documented. If support and doc's stopped calling indexing deduplication, that would be helpful.

From a recovery point of view, one should store the installation files and licensing text files on a SAN is that is utilized. Put SAN management software there too.

I think that all of this could be done very fast with a UNIX shell script, not using SQL. Been there, done that [10 - 100 million records]. For table and list manipulation, one should not use SQL. Someone with an extensive APL programming background would see the merit of that [matrix manipulation, applying operators across matrices and not fiddling with records and fields. At Sprint I could do somethings in a shell script very quickly that took hours with SQL.

Have anyone tested this again with the new acronis 11 ? are there any improvements ?