VCB could not renew disk lease

[powers]

I'm running a HP MSA1000 SAN, Virtual center 2.5 with VCB on the same box, and esx 3.02

When trying to manually run vcbmounter, I get the error

"Could not renew disk lease. Aborting operation"

during the backup of the vmdk file. Happens with all the VMs.

I've tried changing the leasetimeoutSecs to 180 in the BLKLISTSVC section of /etc/vmware/hosts/config.xml as per other people's posts, and have restarted the esx host to no avail.

Anyone have any ideas or anything I can try?

Vmware tech support seems to have abandoned my SR ticket without the issue being resolved. 😐

[bigdee]

We had a similar problem. vcbmounter was able to create the snapshots on the VM but was not able to access the disk and aborted with the error "Could not renew disk lease".

In our case it was a problem with permissions in VirtualCenter. The VCB-User needs permissions at the "Hosts & Clusters" Level.

That solved our problem.

CU

[powers]

Hi,

I've tried using the Administrator account and another user that has Administrator level permissions, but it still doesn't work.

[dconvery]

powers -

run vcbmounter with the "-L 6" switch and post the log.

Dave

[powers]

Hi Dave,

I've attached the log file.

Thanks for your help :smileygrin:

Eloise.

[dconvery]

Make sure you have enough free space on C:\mnt2. Make sure that you delete any snapshots after a failure. Other than that, I don't see anything obvious. The only thing that I can think of is that maybe the MSA is overworked and you have a lot of SCSI disk locks cusing the problem. I assume by the disk path that you have created a 610G VMFS LUN. If you have more that 15-20 VMs running on the LUN, it could cause contention.

If it is actually copying the logs and vmx files during the operation, which the log indicates, then the connection from the VCB proxy to the VMFS is good.

Anyone else have ideas??

Dave

[powers]

I'll try adding some more storage space on the vcb box.

Its got 60G free and I'm only trying to back up a 10G VM, but maybe it needs more for some reason......

Thanks for checking the logs Dave

[dconvery]

Well - 60GB free is plenty for a 10GB VM. Do you have some free SAN space on the MSA? can you create a smaller LUN and svmotion a VM to the new LUN and try a backup?

[powers]

Finally got a response from vmware support (after no responses for 2 1/2 weeks)

During backup procedure, VCB proxy periodically asks ESX host of disk lease extension.

If this request delayed/lost due to poor network condition or increased load on VC (which acts as a broker between the VCB proxy and ESX host), the host declares disk lease timeout and fails the mounting operation.

So when running vcbmounter and connecting to the esx server directly, the backup worked fine. Looks like VC was the problem... (for whatever reason.. its not under much load at all)

[dconvery]

OK...now you know where to look. I would check speed/duplex on the network connections of the VC server. I see many many times in a 10/100 that the nic negotiates to full duplex and the switch negotiates to half. You should force the port speed and duplex on both the switch and NIC. Is it doing anything OTHER than VC? Is there a backup client running other jobs at the same time? This causes CPU to spike while processing files. Is the database on the VC server or a separate server? Is it SQL express or MSDE?

You're right - the VC should not be so over taxed that it cannot process disk leases. I have worked with VC servers with 75-80 ESX servers associated and have had no issue with disk leases. Don't let them close the case until the problem is REALLY resolved. They just narrowed down the cause....

Dave

[powers]

You were spot on with the network diagnosis. Gigabit card was running at 100mb fd. Got it at 1000mb fd (driver upgrade & cable swap)

However I still have the disk lease issue. CPU load was around 5 - 10% while vcbmounter was running.

The box runs SQL server 2000 and symantec (shudder) corporate antivirus server (it distributes virus def updates to around 30 pcs every few days)

No backups or anything else running on it when I'm running vcbmounter...... Its not doing much of anything that would tax it.

[dconvery]

It doesn't sound like SAV is bogging it down, but you can try to stop the SAV services to see if that makes a difference. Also, make sure DNS is rock-solid. Everything must resolve long, short, forward and reverse.

[AmarPatel]

Hi,

Did you manage to find a solution this issue? I'm having a similar issue:

http://communities.vmware.com/message/1831791#1831791

Thanks

Amar