VM Appliance. Frequent failed processes. Operating system error: Cannot allocate memory
Hi,
Over the last few weeks I've been seeing frequent "Operating system error: Cannot allocate memory" failures from one of the VM "acronis agent" appliances (Agent for VMware build 15.0.30984) on one of our VMware ESXi 7.0.3 hosts (VMware ESXi, 7.0.3, 20328353).
MESSAGE
Operating system error: Cannot allocate memory
Additional info:
------------------------
Error code: 20250646
Fields: {"$module":"mms_bundle_glxa64_30984","CommandID":"D332948D-A7A9-4E07-B76C-253DCF6E17FB"}
Message: TOL: Failed to execute the command. Backup plan 'Servers (daily)'
------------------------
Error code: 20250646
Fields: {"$module":"mms_bundle_glxa64_30984","CommandID":"D332948D-A7A9-4E07-B76C-253DCF6E17FB"}
Message: TOL: Failed to execute the command. Backup plan 'Servers (daily)'
------------------------
Error code: 20119593
Fields: {"$module":"mms_bundle_glxa64_30984"}
Message: Failed to execute the command.
------------------------
Error code: 20250677
Fields: {"$module":"mms_bundle_glxa64_30984","FailCount":4}
Message: 4 activities have not succeeded. One of them is displayed.
------------------------
Error code: 20250646
Fields: {"$module":"mms_bundle_glxa64_30984","CommandID":"F30407D6-601F-11E0-9C67-FF46DFD72085"}
Message: TOL: Failed to execute the command. Backup workflow
------------------------
Error code: 20250646
Fields: {"$module":"mms_bundle_glxa64_30984","CommandID":"F30407D6-601F-11E0-9C67-FF46DFD72085"}
Message: TOL: Failed to execute the command. Backup workflow
------------------------
Error code: 20250646
Fields: {"$module":"mms_bundle_glxa64_30984","CommandID":"4504F8D4-2727-42AB-BB4F-A42EDBB790A0"}
Message: TOL: Failed to execute the command. Tol::IsolateCommand
------------------------
Error code: 39649282
Fields: {"$module":"mms_bundle_glxa64_30984","cmdline":"[/bin/service_process,-mode,fork,--pipe,local_srv_proc_pipe_35_1672700403_2C611AB4-96A7-48FB-A010-C206ADBBBE79_1054,--parent_svc,35,--parent_svc_homedir,/var/lib/Acronis/BackupAndRecovery/MMS,--addons_dir,/bin/,--addons_filter,python,--priority,2,--security,1,--parent_log_port,24101]","libcCall":"fork"}
Message: Cannot launch child process due to platform API error.
------------------------
Error code: 65521
Fields: {"$module":"mms_bundle_glxa64_30984","code":12}
Message: Cannot allocate memory
------------------------
The issue is happening more often than not.
I've reinstalled the agent but that hasn't fixed anything. A restart of the agent usually fixes the problem, but only temporarily.
Our other ESXi 7.0.3 host isn't affected.
Any ideas?
Hi Manuel,
Thought I was the only one.
I've this morning seen a similar but different message.
"Failed to complete the operation because the process ran out of RAM."
Not convinced anyone from Acronis actually visits this forum, but we can only hope!
Neil
Neil,
I tried with 32GB Ram with same troubles.
Support Acronis said Memory leak detected.
The ticket was forwarded to RnD (TTASK-53816).
On my side, the error appears by activating "Application Backup" -> "Microsoft SQL Server" in the plan.
When the option is disabled, I no longer have a memory leak.
I created an add-on plan with an agent for Windows SQL Server VMs.
Hello Neil and Manuel, hope you are doing well.
I hope you're doing well.
The error message "Operating system error: Cannot allocate memory" could indicate that the Acronis Agent for VMware is running out of memory while performing its operations, but it's a generic error code and further troubleshooting/log collection is needed.
To troubleshoot this issue, you can try the following:
-
Ensure that the VM running the Acronis Agent for VMware has enough resources allocated to it, including memory and CPU. You can try increasing the allocated memory to the VM to see if that resolves the issue. ( about 1 GB RAM per 1 TB of archive size is required for proper backup operation. Increase RAM on the Virtual Appliance to resolve the issue )
-
Check for any resource-intensive processes running on the VM that may be causing the memory issues. You can try stopping or disabling any unnecessary processes to free up memory.
-
Try to re-schedule the backups to another time ( for example an hour where the device isn't in production or overloaded )
If none of the above solutions work, please keep in touch with our support for further assistance in troubleshooting the issue ( https://kb.acronis.com/content/8153 ). In this kind of situations further logs are needed to know the root cause and apply a solution.
Thanks in advance!
Hello,
Thank you Jose for you anwser.
But all was check, re-check and rebuild with my team and level 2 Acronis support.
I think the problem was solved last 2 weeks.
I waited a bit before posting the solution.
In fact, the problem is a VMware trouble.
The server with troubles is a Dell PET640 with VMware-VMvisor-Installer-7.0.0.update03-20328353.x86_64-Dell_Customized-A09 and VMware ESXi-7.0U3i-20842708-standard (Build 20842708) (updated later).
With these versions, the memories errors are recurrently.
Recently, Dell publish VMware-VMvisor-Installer-7.0.0.update03-20842708.x86_64-Dell_Customized-A10.
There are 2 weeks, I decided to use this version using the VMware command "esxcli software profile install" and not "esxcli software profile update".
Be carefull, using "install" instead "update" delete all additionnals VIBs not prensent in the Customized-A10 file. You need to reinstall your additionnals VIBs after reboot.
I did not use the last VMware updates.
The server stayed on the Dell Cutomized-A10 version.
Since that day, there is no more memory error.
Sincerely
Manuel Pelayo wrote:Hello,
Thank you Jose for you anwser.
But all was check, re-check and rebuild with my team and level 2 Acronis support.I think the problem was solved last 2 weeks.
I waited a bit before posting the solution.In fact, the problem is a VMware trouble.
The server with troubles is a Dell PET640 with VMware-VMvisor-Installer-7.0.0.update03-20328353.x86_64-Dell_Customized-A09 and VMware ESXi-7.0U3i-20842708-standard (Build 20842708) (updated later).
With these versions, the memories errors are recurrently.Recently, Dell publish VMware-VMvisor-Installer-7.0.0.update03-20842708.x86_64-Dell_Customized-A10.
There are 2 weeks, I decided to use this version using the VMware command "esxcli software profile install" and not "esxcli software profile update".
Be carefull, using "install" instead "update" delete all additionnals VIBs not prensent in the Customized-A10 file. You need to reinstall your additionnals VIBs after reboot.
I did not use the last VMware updates.
The server stayed on the Dell Cutomized-A10 version.Since that day, there is no more memory error.
Sincerely
Hello Manuel.
I am glad the issue was resolved.
Thank you for posting the results of the troubleshooting steps and the recommendations. I am sure it will help the other users.
Wish you a nice day in advance!
Hi,
Well, after 3 weeks, the memory errors come back :(
I don't understand.
I will reopen the Acronis ticket in order to restart analyses.
Hello Manuel.
Please let me know if you face any issues while contacting our support after.
Thanks.