Snapshots with quiescing fails directly after starting the snapshot

A new customer who I’m working for, has Avamar from EMC as their backup solution for VMware and all other products.

By chechking the activities of the backups, I’ve seen that both Exchange Server VM backups fails every night.

After having a look at the Avamar logs, i found an error indicating some problems with quiescing:

avamlog3

avvcbimage Error <17775>: Snapshot ‘Avamar-1430951406e32d32fac65d442458b882c72e09fdd7bd3b0a79’ creation for VM <PATH_TO_VM> task creation encountered a quiesce problem (Log #2

So first of all I tried to make a manual snapshot with the quiesce flag from the vCenter:

snapshot

Directly after the snapshot started, it failed again with the following message:

snapshot2

An error occurred while quiescing the virtual machine. See the virtual machine’s event log for Details.

For me this was now clear that this had to be a vmware related problem. As the error message recommended, I checked the event log of the virtual machine witch is normally located in the VM Folder on the datastore. Scrolling around on the log, I’ve found e interesting error message:

avamlog

ToolsBackup: not enough empty nodes (needed 8, found 7)

For some reason the VM is saying that the SCSI Controller has no more empty nodes. By searching in the vmware knowledge base, I have found a KB article where this Problem was explained: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1037071

When you use quiescing for snapshooting a VM, you can’t add more than 8 disks to a SCSI controller. Normally you can add up to 16 disk to one controller.

It seems like when you snapshot the disks with quiesce option, vSphere adds an additional disk for each disk that is attached. This must be the reason.

So the only way to resolve this problem, was to add an additional SCSI Controller and attached 4 of the disks to the second controller. After this, the VM was snapshoted without any issues:

snapshot3