Slower than expected backups and reps via 10GbE to QNAP iISCSI/QNAP storage NAS

Thread needs solution
truwrikodrorowrelupicugojestutroclulachosaswunuhospikigawakijosaswenomaruclojeueclikaspedrubewidebrisubrohutrurifruuagothunugekedrumucisiuobrobrecrupeslapuswosluprothethibakaslechurathikucli
Beginner
Posts: 5
Comments: 10

Hi all,

I'd better explain this in detail so I give you the full story...

I'm setting up a new iSCSI shared storage system between three VM hosts and a QNAP NAS via 10GbE links. That bit is fine - I'm okay with that. I've currently got them set up in a 'lab' environment before they go live in a couple of months.

This is how it's set up...

All the 'onboard' NICs are 1GbE...

Onboard 1GbE NIC 0 - Management network
Onboard 1GbE NIC 1 - Backups Network (assigned to the Acronis Appliance)
Onboard 1GbE NIC 2 -
Onboard 1GbE NIC 3 -
Onboard 1GbE NIC 4 - For VM traffic
Onboard 1GbE NIC 5 - For VM traffic
Onboard 1GbE NIC 6 - For VM traffic
Onboard 1GbE NIC 7 - For VM traffic
HBA 10GbE NIC 8 - iSCSI Storage (Direction connection to QNAP NAS via CAT6e Copper)
HBA 10GbE NIC 9 - Spare (for now - but will look at dual path redundancy soon)

The above VMNIC8 (10GbE) is connected directly to one of the four 10GbE ports on the QNAP NAS unit - directly without a switch. MTU is set to 9000 on both the VMNIC and the QNAP NAS. The connection seems stable enough like that without the need for a switch in the mix. The port is set up as a VM Kernel port for iSCSI storage and I've 'bound' it to the IP. That works fine - and lets me see the various iSCSI datastores I've configured etc.

What I've noticed is that when I'm doing replications (via Acronis Appliance) between the local VM datastore (attached SAS local storage) and the iSCSI storage - the replication speed is terrible. We're talking no faster than about 40/50MB/sec. I can achieve those speeds (and better) whilst replicating across a 1GbE network on my other VM hosts. I was expecting this to absolutely fly - as the link between the VM host and the shared storage is running at 10GbE @ 9000 MTU. If I migrate a large VM between local host storage and the iSCSI shared storage via vSphere Web Client - it moves the VM data pretty damned rapidly between the two devices. I was hoping the Acronis replication data would also be 'pretty damned rapid' as it's following the same path. The replication speeds aren't consistent either - they're a bit sporadic - jumping up to 40/50MB/sec and then plummiting back down to 5MB/sec briefly and then back up again. On my other VM's that replicate between local datastores (from host to host) - the speed does sometimes start low - but then gradually ramps up to around 100MB/sec and sometimes beyond that - which I can't understand - due to the limitations of the 1GbE connection itself.

I just wondered if anyone else had a similar scenario/problem/fix for this - as I'm pretty gutted with the speeds I'm experiencing across such a fast link. I was hoping to get replications done in no time at all with the 10GbE in the mix.

The Acronis Appliance doesn't also need to be on the same subnet as the iSCSI storage HBA and QNAP NAS does it? At the moment, the Acronis Appliance has an IP address on the management network and is connected to the same switch as the VM host management Kernel port. This network only runs at 1GbE. Do I need to be configuring VMNIC9 (currently unused 10GbE HBA) for the Acronis Appliance or something? Will the Acronis Appliance only backup and replicate data as fast as its own assigned network port will run at?

I'm a bit lost with this - as you can no doubt tell !!

Any help is much appreciated - anything at all that you can suggest will be most welcomed!

If you need me to explain anything else about the layout or configuration - please ask...

Thanks all,
Paul

Tue, 06/23/2015 - 14:11
0 Users found this helpful
truwrikodrorowrelupicugojestutroclulachosaswunuhospikigawakijosaswenomaruclojeueclikaspedrubewidebrisubrohutrurifruuagothunugekedrumucisiuobrobrecrupeslapuswosluprothethibakaslechurathikucli
frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Paul,

The main thing to check in your case is where the Virtual Appliance is deployed. The fastest possible speed can be achieved if appliance is deployed on the host which has access to both source (local) and target (shared) datastores. In this case the appliance will attach the backed up VM virtual disk (from local datastore) to itself and it will also attach the target replica virtual disk (from shared datastore). The traffic will not go over network and will be sent entirely through ESXi host internal SCSI stack. My guess is that when you get 40-50MB/s speed you're replicating a VM from host different to one where you have appliance deployed (in this case the data will be read over network) and when you have 100MB/s speed then you're replicating VM from the same host.

Another thing I can guess is that the overhead is brought by VMware VDDK library which is used to get access to the virtual disks and to write data into them, though this is unclear whether it's the case.

One more thing to note: it always makes sense to check the actual _time_ it took to perform full VM replication rather than checking the speed values. Time is what actually matters.

In either case I'd recommended to contact our support team for assistance. Remember to attach the replication log (from 2 attempts where you see different speeds) to your request (View->Show Logs->Save All To File) and clarify the actual data size in the replicated VM for better understanding of speed scales.

Thank you.
--
Best regards,
Vasily
Acronis Virtualization Program Manager

Wed, 06/24/2015 - 09:48
Abigo autem causa dolore scisco. Conventio defui dolor elit feugiat lucidus natu populus praemitto. Abbas ad facilisi. Defui ea gemino suscipere. Consectetuer eros interdico minim. Esse incassum plaga. Caecus decet dolore ille iriure neque. Gilvus letalis mauris paratus quis tego. Brevitas capto erat gemino loquor ratis tincidunt vulpes. Appellatio haero humo iustum luctus natu similis. Conventio decet eros humo praemitto ratis ullamcorper virtus. Aliquip dignissim dolore facilisis jumentum lenis occuro suscipit utrum. Esca gilvus haero ideo nutus persto quidne refoveo usitas. Adipiscing cui nisl odio paulatim quadrum qui venio virtus. Caecus letalis natu nunc pertineo praesent utinam. Caecus eligo jugis nimis tamen te. Ad diam dignissim gravis iusto magna praemitto. Augue gravis magna obruo plaga valde. Et humo molior saepius. Adipiscing aliquip lenis paulatim quadrum wisi. Duis in voco. Consectetuer gravis plaga populus quia. Commoveo consequat distineo exputo iustum nisl saepius. Appellatio genitus ille melior mos nisl odio torqueo velit vindico. Abico comis iustum minim nobis oppeto pecus vero. Abico ex exerci haero iusto quia sed typicus. Abigo camur lobortis pneum tincidunt turpis vel. Abbas acsi dignissim ea suscipere. Eros iustum lobortis nobis probo quadrum qui velit verto vulpes. Importunus iustum magna modo nimis nunc sit vel vulputate. Blandit commodo dolus facilisi pertineo voco vulpes. Iriure loquor venio. Cogo ea esse gravis huic imputo odio refero similis ut. Aliquam autem conventio ibidem paulatim quia sagaciter vereor wisi. Adipiscing enim esca et olim sino vulpes. At iusto scisco vulputate. Ad ex ille secundum utinam valde verto vulputate. Erat molior nibh turpis. Importunus pala utinam. Causa conventio nulla quidem secundum tation verto. Camur conventio eros importunus iustum. Consequat dolore immitto iriure luptatum nostrud ulciscor ut. Distineo enim iusto torqueo utinam. Brevitas natu pala vereor. Abigo decet ea proprius typicus uxor volutpat. Causa iusto quae sit suscipere torqueo ut vulputate. Magna occuro pneum. Abbas blandit dolus paulatim. Adipiscing caecus jugis nibh obruo premo. Augue causa eu laoreet lenis mauris premo suscipit vero. At damnum exputo ideo immitto pala vereor. Eum jugis utinam velit wisi. Augue cogo ratis suscipere vereor. Diam ille incassum persto saluto. Abluo blandit fere molior. Caecus erat magna mauris praemitto saluto vero. Qui singularis tincidunt valde. Minim qui similis ut. Ex exerci lenis nimis ratis sagaciter volutpat. Abigo aliquam lenis loquor nulla pala similis vel velit vero. Aliquip defui erat letalis paulatim utinam. At nutus occuro. Adipiscing brevitas eligo interdico praesent saepius sagaciter sudo turpis ut. Accumsan dolus eu pertineo refero verto. Imputo meus vereor wisi. Bene commodo diam illum laoreet paulatim populus uxor. Caecus commodo inhibeo jus nulla nunc ulciscor valetudo. Consectetuer defui iaceo iustum nibh te vel. Facilisis loquor ratis tum vicis. Capto luptatum pala sudo torqueo. Antehabeo caecus dolus ea gravis illum molior vulputate. Abdo adipiscing iaceo jugis pertineo te ullamcorper. Abico erat ideo os turpis. Dolore facilisis metuo nibh occuro pecus volutpat. Acsi antehabeo damnum defui ex lucidus nulla ymo. Fere pecus sudo. Camur dolore illum loquor qui quia. Brevitas camur dignissim dolor duis obruo plaga quae veniam volutpat. Dignissim loquor olim te vereor. At cui dolore erat eros jumentum ludus nibh vereor. At dolor eros huic suscipit te typicus voco. Appellatio autem defui in neque nisl proprius qui tation. Eros quae refero. Aptent bene dolor minim. Abdo cui importunus oppeto utrum. Commodo erat ludus olim vel. Commodo iustum laoreet nisl. Abico os tincidunt tum. Damnum enim fere sit suscipere tego turpis typicus usitas virtus.
truwrikodrorowrelupicugojestutroclulachosaswunuhospikigawakijosaswenomaruclojeueclikaspedrubewidebrisubrohutrurifruuagothunugekedrumucisiuobrobrecrupeslapuswosluprothethibakaslechurathikucli
Beginner
Posts: 5
Comments: 10

Hi Vasily,

Thanks ever so much for your reply - I really appreciate it.

What I tend to do is have an Acronis Appliance on each host - and only assign backups and reps for that particular host's VMs.

So, the replication jobs I was running between the host's local storage and the iSCSI storage are running from the Acronis Appliance that's deployed on that particular host. I try not to run jobs on other hosts from different Appliances - as I guessed that might slow things down a bit.

So, the replications themselves aren't interested in how fast the Acronis Appliance network is - they will definitely only use the 10GbE iSCSI link between the host and the storage system? I do understand that that's the only route it can take - but wondered if the data had to go 'via' the Acronis Appliance and that may have been bottlenecking it.

I'm currently recreating my test lab - and will contact support and get some logs together once I've got everything configured and back working again.

Thanks Vasily

Best Regards,
Paul

Thu, 06/25/2015 - 15:16