what is local disk size limit in Acronis Backup Agent Appliance

Hi Aldous,

Can you please clarify which exact operation returns "there is not enough disk space" error? Note that 'fdisk' utility cannot be used for management of >2TB volumes so outputs from it are not of concern. The >2TB disks are initialized as GPT ones and are formatted into ext4 file system through appliance local GUI, so your initial configuration steps seem to be right, while the error might be caused by something else (need to understand the context of the operation first).

Thank you.

Hi Vasily,

I got the "there is not enough disk space" when a scheduled backup plan was executed. I noticed that the management console showing was enough free space for any VM in the backup plan. Anyway, after I rebooted the agent appliance and unmount/remount the local storage, it's working now. 

Regards,

Aldous

Hi Vasily,

unfortunately, I got following error today, "database or disk is full" and a backup stuck at 100%. 

***************************************

MESSAGE

Failed to check out the resource with ID '/vmwesx/host-44673/vm-214/[NAS04B_SAS_2] bwxsp-03.mgmt.comwave/bwxsp-03.mgmt.comwave.vmdk' from the repository.

Additional info:

------------------------
Error code: 60
Module: 469
LineInfo: 0x2C7A0B848A167CD0
Fields: {"$module":"mms_bundle_glxa64_10330"}
Message: Failed to check out the resource with ID '/vmwesx/host-44673/vm-214/[NAS04B_SAS_2] bwxsp-03.mgmt.comwave/bwxsp-03.mgmt.comwave.vmdk' from the repository.
------------------------
Error code: 11
Module: 249
LineInfo: 0x30AD4F7563758C53
Fields: {"$module":"mms_bundle_glxa64_10330"}
Message: ASYNC: Exception has occurred during work execution.
------------------------
Error code: 4
Module: 249
LineInfo: 0x236AA8C5161DFBA3
Fields: {"$module":"dml_glxa64_10330"}
Message: ASYNC: Execution flow control job.
------------------------
Error code: 4
Module: 297
LineInfo: 0xF3558890AD00AE78
Fields: {"$module":"dml_sqlite_addon_glxa64_10330"}
Message: Statement 'COMMIT' cannot be executed. Error code: 13.
------------------------
Error code: 17
Module: 297
LineInfo: 0xF3558890AD00AE78
Fields: {"$module":"dml_sqlite_addon_glxa64_10330"}
Message: An error occured while using database '/var/lib/F4CEEE47-042C-4828-95A0-DE44EC267A28.db3'.
------------------------
Error code: 13
Module: 0
LineInfo: 0xF3558890AD00AE78
Fields: {"$module":"dml_sqlite_addon_glxa64_10330"}
Message: database or disk is full

***************************************

Regards,

Aldous

turn out I found the Appliance / disk went out of space. When I looked further, I found most of the space were filled by "Activity" files in "/var/lib/Acronis/BackupAndRecovery/MMS/Activity". What can I do with to free up the disk and stop from happening? The '/' disk of this appliance already has 20 GB instead of the default 5 GB and 17 GB used by the "Activity" folder. 

Hi,

The activities logs are rotated within 1 year period and normally contain only text files which are very small in size. 17GB is definitely not normal size for these logs unless appliance was running for 10+ years executing backups every 5 minutes. It makes sense to contact our support team and analyze why there is such huge size for these logs by getting them exported from appliance. You can connect to appliance via WinSCP (as described in "Connecting using WinSCP utility" section in https://kb.acronis.com/content/61859) and copy the contents of the /var/lib/Acronis/BackupAndRecovery/MMS/Activity folder for further analysis with our support team. These files can be safely deleted from appliance permanently and it won't affect other operations.

Thank you.

I already contacted support and i was told to update the VM's vmdk files because the support want the logs on the appliance but unable to tell the exact location. Anyway, it's good to now that the logs are rotated within 1 year. It will be nice if Acronis can build  a internal monitoring to 1. reserve some space for working (so that it won't crash any process); 2. giving some warning more specific instead just saying out of space; 3. set the log retention or purge it as needed. 

thanks,

Aldous

I am getting this same problem all the time.  A backup of guests won't complete with the disk is full error and then I have to delete the Appliance and redeploy.

It's incredibly annoying.

Jason.

Aldous, Jason,

Having appliance overfilled with logs is not normal situation and to properly address the issue we need to understand which particular logs are consuming so much space, e.g. it should be investigated with our support team involved.

There are multiple Acronis modules which write independent logs and all have rotation logics included when the logs increase some threshold (10MB for "snapapi" module logs for example), then they are archived into .zip automatically and get permanently deleted after 1 year. Packing text files in .zip achieves very good compression ratio, so the logs are not taking much space under normal circumstances.

In your cases there could be some faulty module which violates the designed retention logics and this has to be properly analyzed (reveal which particular module is causing the issue).

Thank you.

Hi Vasily, 

FYI... I've uploaded the whole VM files (acronis backup agent appliance) to the cftp.acronis.com as per requested from your support. In case you are interested to look at it, the case number is 03527724 and the zipped file name is Case_03527724.zip. Thanks,

Aldous

Hi Aldous,

The logs are growing due to unexpected garbage error message which appears when appliance guest OS (Linux) is trying to initialize the local disks on the SCSI controllers. There seems to be some specific in the locally attached storage disk in your case (which is >2TB and probably has some other specifics) which in turn causes such behavior. Faulty lines spammed in the logs:

2018-10-23T02:23:17:642-04:00 139779079350016 C0000001B: Assertion failed: !"Unknown scsi host type", file e:/232/enterprise/virtualware/raw/vmware/scsi/impl/host.cpp, line 105
2018-10-23T02:23:17:644-04:00 139779079350016 C0000001B: /bin/libDiskBundle.so          (                                           + 0x84cf13)  [0x7f2121908f13]
/bin/libDiskBundle.so          (           

These are the problematic messages which are overfilling the logs under /var/lib/Acronis/BackupAndRecovery/MMS/Activity/ folder. These logs can be safely removed completely (all files) as a temporary measure.

For futher troubleshooting we need to understand the type of SCSI controllers discovered by the appliance guest OS. This can be done via the following command executed from local appliance UI (from command line interface):

cat /sys/class/scsi_host/hostX/proc_name

Where X varies from 0 to [number of SCSI host adapters], in my below example it's from 0 to 3:

The expected (by Acronis engine) SCSI host adapters classes names are bolded below:

LsiLogic, "mptspi"
LsiLogicSas, "mptsas"
BusLogic, "BusLogic",
AtaPiix, "ata_piix",
Paravirtual, "vmw_pvscsi"

If the class name for SCSI controller on appliance does not match one of the expected ones this results in error quoted above and it leads to unexpected growth of the log files.

Please check the classes names in your environment (it cannot be checked by us, since it's tied not to appliance VM itself which I retrieved the logs from, but rather to specifics of the particular disks attached in real environment) and send this information to our support engineer.

P.S. I will also coordinate these troubleshooting steps with our support team internally.

Thank you.

please see attached output from the appliance. It's 100% the same as yours

Hi Vasily, 

the local attached disk in the appliance, in fact, a vmdk. Therefore, it's using the same scsi controller as the appliance. I didn't customize any VM's setting other than adding an extra 3 TB HDD as local storage

Regards,

Aldous

Hi Aldous,

Thank you for the details. The issue is definitely related to locally attached storage used, however our first theory (specifics of SCSI host adapter) didn't confirm, so there should be extended logging added for further troubleshooting. This can be done by our support team (our maintenance developers). Please continue working with our support engineers - I've passed them all required technical details related to the case.

Thank you.

sure, thank you