One or other VM failing during Commvault Backup

[Noor2122]

With Commvault backup, we are facing this issue where during backup one or other VM is failing and when we try to power on the VM it throughs an error like,

"/vmfs/volumes/57d9728f-6r5de0ec-0000-5cb9019c7168/xxxxxxx/xxxxxxx-000001.vmdk" : failed to open (The system cannot find the file specified).

Disk consolidation didn't help here as on checking we couldn't find any snapshot disk in VM directory.

Now to power on this VM, we have to manually point the VM disk back to base disk.

Note: We are using Vsphere 6.0 u2 and this is an intermittent issue and not limited to any particular VM....ESXi logs are also not showing anything except above kind of entries. 

As per my understanding, ideally once the backup is completed by the back up software it should release the control of the base disk and attach it to VM but seems that is not happening here.

Any help will be appreciated...

[vXav]

So the backup makes the VM crash and then you try to restart it?

What do the events of the VM look like at the time of the backup?

I assume it's the same behaviour as Veeam so:

- Create VM snapshot

- (Nothing while Comvault takes the backup)

- Remove snapshot

- disk consolidation

When the VM is working, are you able to take a snapshot and consolidate successfully?

Can you see anything interesting in the logs/events of your commvault server?

[Noor2122]

We can't take a snapshot during issue because VM disk is pointing to a snapshot disk that doesn't exit so without re-pointing the disk to base disk we can't do anything...

Once the VM is working then tried to manually take/remove snapshot with or without disk quiescing option but there is no issue at all.

We couldn't find any specific event in esxi host logs and even VMware support guy is unable to find the reason of this issue and suggested to check with backup software support.

As commvault backup tool is managed by a different team so so far have no clue from that side however we requested the team to open a ticket far the same with commvaut.

[WhiteForEver]

Commvault is very recommending to use latest VM Hardware version.

Is your VM updated to the latest one?

[Noor2122]

Yes all the VMs having HW version 11, which is greatest for  ESXi 6 u2...As per our conversation with VMware support, they couldn't find anything specific to the issue in provided host logs and suggested us to contact them during the issue (without making any changes our-self)....their impression about this issue is, its something where due to some reason the VMX file is not updating correctly during Commvault backup and they need to check it during the issue.

Thanks

[RAJ_RAJ]

Hi ,

Please check on the proxy server (vmware) , vm edit setting disk details (vmname-0000X.vmdk ) , if there is available you have remove manually . Some cases the snapshot won't release automatically there is no permanent solution for this issue (common in all vmware backup solutions).

Please follow the below

- Power off VM

- Create VM snapshot

- Remove snapshot (Remove Snaphot all option)

- disk consolidation

- Check on the proxy server base disk with snapshot attached or not if attached remove manually

- Power on the VM

[johnw230873]

Agree with RAJ just like to add that the proxy server is the backup proxy server and if disk consolidation doesn't work try a disk migration to a different LUN, or a clone of the virtual machine.

We have had to do this in the pass with Commvault on the odd occasion.

[Noor2122]

but this case is different...the case you are referring can be simply fixed by just what you mentioned or by doing the storage vMotion of the VM in question and then the disk consolidation would complete successfully.

In this case the Commvault backup is happening over network so there is nothing like disk hot-add happening here, the weird part of the issue is sometimes during backup one or other VM geting powered off with the error like the snapshot disk to which the VM is pointing is missing and then to power on this VM we have to manually remove the missing snapshot' disk entry and repoint it to base disk.

As per my understanding, this is something where during backup operation, snapshot task failing for some reason and then VM configuration file (*.vmx) is not updating correctly or i can say not repointing to base disk and so the VM is crashing.

We are working with VMware support to find the root cause of this case and as suggested investigate about it with them when next time come across this issue.

Btw thanks for your response...

[RAJ_RAJ]

Hi ,

Are you sure your vm backup configuration is correct ?

Advanced Configuration - Virtual Server Agent for VMware

You may refer with commvault also  , there may be chances for bug

[Santosh401]

1. Check Commvault Logs: The first step should always be to check the logs provided by Commvault for any error messages or warnings related to the VM backups. This can give you specific clues as to what might be going wrong.

2. Review Resource Allocation: Ensure that the VMs and the backup infrastructure have sufficient resources (CPU, memory, and disk I/O) to handle the backup process. Resource contention or overallocation can cause backups to fail or timeout.

3. Network Stability and Configuration: Check the network configuration and stability between the Commvault Media Agent and the VMs being backed up. Network issues can disrupt communication and cause failures. Ensure that any network throttling or bandwidth limitations are appropriately configured to handle backup data transfer loads.

Parablu is a company that offers data management and security solutions, with a particular focus on secure data backup and recovery. Their products are designed to help businesses protect their data while ensuring compliance with data protection regulations.