Skip to main content

Simultaneous ESXi backup issue in vmProtect 8 appliance - leads to no free space on appliance disks

Thread needs solution

Looking for a vmware backup solution.

Initially, Acronis vmProtect looks nice.

There not too much VM's (less than 20), 4 ESXi hosts.

Received a lot of "backup failure" messages today, not even able to log in.
Code: 27066484 (0x019D0074)
Line: 785B8E25EFF8E862
Message: An error has occurred while executing command 'login'.

============
Code: 27066387 (0x019D0013)
Line: 16D8D0EAB81B7522
Message: A key holder error occurred.

============
Code: 16285701 (0x00F88005)
Line: C25B1498C08C3530
Message: Failed to store object access information.

============
Code: 7667714 (0x00750002)
Line: 23C7366F69C73E79
target: /tmp/root/.Acronis/CredStorage/acronis_wcs_wcs_authorization_7f429eaf-a59d-4c14-9b8e-c21a35160087/
cred: 0
Message: Failed to write credential '{0}' to the set of user credentials.

============
Code: 262161 (0x00040011)
Line: 3D9EA26F1BFDF04F
path: /tmp/root/.Acronis/CredStorage/acronis_wcs_wcs_authorization_7f429eaf-a59d-4c14-9b8e-c21a35160087/0
Message: The specified file does not exist.

============
Code: 65521 (0x0000FFF1)
Line: BD28FDBD64EDB8D0
code: 2
Message: No such file or directory

0 Users found this helpful

Ok, more "news"

After restart (correct one, used acronis console GUI) the appliance is stacked in a reboot cycle. It boots, starting GUI and then self-rebooting in a few seconds - infinite loop.

Wonderful!

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Maxim,

The symptoms you describe are typical for cases when appliance disks become overfilled. The solution (to revert back things to normal) is described in the following KB article: http://kb.acronis.com/content/32819 .

The most likely reason for such behavior (I made this conclusion when you mentioned _4_ ESXi hosts) is that you have scheduled the backup of ESXi configuration of all these hosts for approximately the same time. What happens is that first there is .tar.gz archive created under /tmp folder on appliance, then this archive is transformed into .tib format which is placed onto the actual target backup repository which you specified in the task properties. The size of these .tar.gz files is around 300-500MB so simultaneous backup of all 4 ESXi hosts will lead to lack of free space and subsequent crash of the appliance (which has 2GB disk).

To prevent the issue from future occurences please avoid scheduling ESXi configuration backups for the same time - make at least 10-15 minutes interval in schedule. We are going to forbid simultaneous execution of ESXi configuration backup tasks in the 9th version which we will soon publish as Beta.

If my assumptions are not correct, please send me a private message with the link to this thread so that I can give you further instructions.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Just chatted with the support. The experience is awful.

Indian (?) guy (his name is Sree) advised me to delete virtual appliance / to lost all configs. This is the only way to "repair" everything.

For sure, it is completely unacceptable.

Ok, fortunately it is just a Linux box.

I mounted appliance disks in another Linux VM and realised that second disk is 100% full. The reason is simple - unbelievably, but ESXi hosts backup is initially made in local /tmp directory (!!!). 4 ESXi hosts, /tmp is full, vmProtect is dead and not able to start properly.

It means that Acronis not testing their software properly, in addition to worse-less support. The vmProtect appliance is suicidal as it fills the local disk (which is very small) and then starting to kill itself by zeroing configuration files.

It is really happening?

p.s. this is how it looks like in a /tmp directory:

-rw-r--r-- 1 root root 525383680 Jun 3 11:04 esxibackup_3ABA129B-CE7B-4B0C-9205-5128D48E1176.tar
-rw-r--r-- 1 root root 473063424 Jun 3 11:04 esxibackup_4EBEC5B2-B6D5-490A-AD25-DFC7294B11A9.tar
-rw-r--r-- 1 root root 464388096 Jun 3 11:04 esxibackup_5A362D54-73B9-474F-B693-D63D5D5918A8.tar
-rw-r--r-- 1 root root 75104256 Jun 3 11:04 esxibackup_A2E4B40C-613C-42B0-B8B4-0B4DDE0350C7.tar
-rw-r--r-- 1 root root 515921920 Jun 3 11:04 esxibackup_E5C944FF-5C9D-4D82-96B9-56864B30DB79.tar

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Maxim,

Looks like my assumptions from previous post were correct and the problem was properly indentified - simultaneous ESXi configuration backup of 4 hosts at the same time. You should delete these /tmp/esxibackup_*** files (#rm -rf /tmp/esxibackup_*) and after that replace the /etc/lighttpd/lighttpd.conf file with the one linked in http://kb.acronis.com/content/32819 article (direct link: https://kb.acronis.com/system/files/content/2012/06/32819/lighttpd.zip ). Then after reboot if you schedule the ESXi configuration backup tasks with at least 10-15 minutes interval there should be no problem.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Hi Vasily,

You are absolutely correct, I already identified this issue by myself - it is related with ESXi hosts backup.
The issue is that (probably) some config files are already damaged additionally and the appliance is not starting properly even with those files removed manually.

The forum is much more helpful than your online support indeed. As I mentioned earlier, the only advise was given to delete the appliance and install a new one.

Thank you!

Vasily,

Will do that now.
The forum support is excellent (completely opposite to "chat-based" one).

Thank you once again.

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Maxim,

Thank you for the notes! In any case please let me know (either in this thread or via PM) how the instructions worked for you.

I definitely agree that such issue may indeed look really frustrating when one simple misconfiguration step leads to multiple problems and we will fix it in the upcoming version for sure to avoid such problems in future.

P.S. In my experience I've seen such problem (the same reasons for appliance failure) once or twice in the past in reports from our customers, so we will definitely fix it soon.
P.P.S. FYI: I have modified the topic subject to be more related to the issue described in it.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Short question.

The new and old lighttpd.conf are different only in one option - certificate.pem

Which one is correct? Should I really replace the config file?

oraclevmdb:/mnt/1/etc/lighttpd # diff lighttpd.conf.old lighttpd.conf
39c39
< ssl.pemfile = "/bin/certificate.pem"
---
> ssl.pemfile = "/var/ipcert.pem"

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi,

The links are both correct actually. Typically after appliance gets into such state (overfilled) the /etc/lighttpd.conf becomes empty (0 size), so replacement is necessary. However in your case if /etc/lighttpd.conf is not empty then you may not need to replace it.

After cleaning up the space the appliance should come up back normally.

If you still get the same error on connect as mentioned in your original post then you should delete the following folder contents:

#rm -rf /tmp/root/.Acronis/CredStorage/*

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Vasily,

Still the VM reboots (infinite loop).

Probably, some other config files are damaged.

Is there any way to backup all "tasks" manually (some config?) and to reinstall appliance? Probably it could be the easiest way.

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Maxim,

Yes its possible to migrate the tasks and their scheduling my copying the /var/lib/Acronis/vmProtect/ + /var/lib/Acronis/Scheduler/ folders from old appliance to the new one. I have described this process in details here: http://forum.acronis.com/forum/34725#comment-107881

After such migration all you need to do on new appliance is to add licenses under Configure->Licenses (the same ones that you used before).

If that still doesn't help, can you please send me a private message for further guidance?

Edit: FYI: in the 9th version of vmProtect we will have export/import vmProtect configuration feature to protect appliance from failures and ease the migration process.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Vasily,

I deleted the "/tmp/root/.Acronis/CredStorage/*"

Now I've got error message "Acronis Managed Machine Service is stopped" (screenshot is here):

http://gyazo.com/e6c76aac066fbfe63e04c47038d19655

...

It is really nice to hear about version 9, I'd really prefer to buy Linux-based appliance product than something like Veeam (tightly-linked to Windows), but it should be stable and reliable solution.

It would be nice not only to provide manual configuration backup / restore, but scheduled / automated one as well.

Is there any timeframe regarding new version?

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Maxim,

I've sent you a private message with further instructions. That should be the fastest way to resolve the issue. After I figure it out I will post the final outcome in this thread.

Edit: FYI: The vmProtect 9 beta will start really soon, though I cannot publish the exact date right now :(
Also vmProtect configuration backup will be a manual one (quite similar to Internet routers configuration export/import approach) since typically that's something that should not change constantly (i.e. it must work in "set-and-forget" way).

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

Well, "since typically that's something that should not change constantly" - I could not agree with that as backup tasks could be added / removed / changed quite frequently (ok, vmProtect is aimed to SMB and medium-sized businesses but still up to 100 vms is quite common for many modern companies), bearing in mind Miscrosoft licensing policy (unlimited VM's per physical host) and best practices ("separate every single service")

frestogaslorastaswastavewroviwroclolacorashibushurutraciwrubrishabenichikucrijorejenufrilomuwrigaslowrikejawrachosleratiswurelaseriprouobrunoviswosuthitribrepakotritopislivadrauibretisetewrapenuwrapi
Posts: 22
Comments: 3800

Hi Maxim,

Yes, definitely there are cases when scheduling for exporting/importing of backup application configuration is important and we may add it later if we see demand for this feature from our customers. Though there are multiple ways to achieve the same goal. For example to cover the case you described it is sufficient/more convenient to select entire resource pool/host for VM backup (on the 1st step of the backup wizard) - in this case it is considered as a dynamic group and all new VMs appearing in this pool/host will be backed up by this task.

Thank you.
--
Best regards,
Vasily
Acronis vmProtect Program Manager

The issue is fixed, Acronis support is excellent (besides the initial chat-based experience).

The procedure to backup and restore tasks / jobs described correctly and we've got no any issues at all since we did it (takes less than 10 minutes).

Thank you!