vm snapshots created via veeam sometimes cause host disconnect

[ufo8mydog]

Hi everyone,

Heres a strange issue - we're looking at veeam for backups. We're running veeam in the virtual appliance mode since the environment is NFS.

Sometimes (not all the time) after a vcenter initiated snapshot is made and removed as part of the backup process the ESX 4.0 U1 host will disconnect from vcenter and will then take 5-10 minutes to reconnect. This doesn't cause a HA event and all services remain normal during the disconnected status.

When this happens the removal of the snap does not seem to complete and I have to do it manually; any other virtual machines on that host cannot be contacted by veeam until the ESX host reconnects and so those backups fail too.

Has anyone ran into this issue?

[athlon_crazy]

Have you try to increase SC memory & resources allocation on ESX host? That's what suggested by VizionCore last time when I'm having similar issue. It's may due to yor SC resources bottleneck.

1) SC Memory = ESX -> Configuration -> Memory -> Edit

2) Resource Allocation = ESX -> Configuration -> Resource Allocation -> System Resources Reservation -> Edit

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

[ufo8mydog]

Thanks for the tip athlon_crazy; the SC memory is set to 800MB already (the maximum). And the CPU and memory shares both have a limit of 'unlimited'.

[athlon_crazy]

Lucky, I still managed to get my last email with VizionCore as below :

Thank you for contacting Vizioncore Support Team.

The COS is the Console Operating System or the Service Console.. This is where the ESX processes such as taking/deleting snapshots and backups take place. Having more resources available alleviate resource contention.

To increase the amount of physical ESX host RAM that is allocated to the service console, a change is required in the VI Client.

Click on the ESX host in the inventory, select the configuration tab and then choose memory. From here, the link labeled "Properties" can be clicked and the amount of RAM modified from the default of 272MB up to 800MB. This change requires a RESTART of the ESX host to take effect.

To increase the CPU reservation that the service console has over CPU 0 in a ESX host, the VI Client is also used.

Select the ESX host in VI Client inventory, select the configuration tab and then "System Resource Allocation" from the software pane. By clicking on the "Edit" link the CPU reservation slider can be adjusted up to 1500 MHz

The System CPU0 reservation part solved our problem.

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

[Doug_Hazelman]

There's a thread in the Veeam forums about a similar issue. The original poster did not indicate if the issue is resolved but the Veeam product manager suggested there may be too many SOAP connections open to the host, for instance Virtual Infrastructure Client "console" connections to VM's running on the host. You can check the thread here: http://www.veeam.com/forums/viewtopic.php?f=2&t=2469&p=10196&hilit=host+disconnect#p10196

Doug Hazelman

Systems Engineer

Veeam Software

[ufo8mydog]

Hi Doug,

The host disconnects only during the snapshot removal phase; someone has suggested that it happens because the snap is very big but the backups usually only take an hour so the snaps don't grow that big.

I set system CPU reservation to 1500Mhz (although the maximum is unlimited anyway) but I'd like to find the root cause of the issue before I kick off more backups.

[Doug_Hazelman]

OK, understood. Have you tried the "Safe Snapshot Removal" option under Advanced properties for the job? I'm not sure if it will help if the snapshots aren't too large and the VM remains responsive. Also, I do recommend zipping your log files and sending them into Veeam Support.

Thanks,

-doug

[ufo8mydog]

Hi Doug,

Yes, I've sent my logs to veeam support, I will see what happens there.

According to gostev safe snapshot removal is no longer required from ESX3.5U2 onwards, when helper snapshots were introduced;

http://www.veeam.com/forums/viewtopic.php?f=2&t=2279#p9598

[nimos001]

Did you ever find a resolution for this? We have similar problems outselves.