Extremely high latency when migrating from local datastore to shared datastore.

[BorjaGonzalez]

Hi guys i hope you to help me. Sorry for my english btw, i'm not native.

Let's begin!

I have:

1 vCenter
1 Host
1 Distributed Switch (with one pg for management network/IPstorage of the esxi)
1 Standard Switch (Empty)
1 FreeNAS to provide iSCSI LUNs
1 Microsoft to provide iSCSI LUNs

When i try to migrate VMs between shared datastores or from shared datastore to local, everything is fine. The problem became when i try to migrate VMs from local to Shared datastore. All the datastores go down ( All paths down ) and go up again and i recieve this error:

" Error caused by file /vmfs/volumes/volumenID/VMDirectory/Disk.vmdk "

When i try to migrate VMs from local to FreeNAS iSCSI datastore it fails inmediatly.
When i do the same from local to Microsoft iSCSI datastore it takes a loooooong time to migrate the VM, give me the same All paths down and uplinks down error but don't fail in the migration.

I'll give you some screenshots to see the errors.

Thanks a lot!

EDIT: I have notice extremely high latency when i try to migrate from local to shared datastores. 2000ms average with peaks of 50.000 (See my reply below for more info)

[Nick_Andreev]

How is iSCSI configured on the ESXi host? Do you just have one iSCSI VMkernel port, which is on the same subnet as the FreeNAS appliance?

Can you show properties of the datastores you share from FreeNAS and Windows box. Go to datastore properties, click on Manage Paths and send a screenshot.

[BorjaGonzalez]

Hi @Nick_Andreev , thanks a lot for your reply and your help.

Exactly, i have one VMkernel Port in the VDS for Management and for iSCSI, for both. (See screenshot) which is in the same subnet as the FreeNAS and the Microsoft providers.

[pterlisten]

Hi,

sounds like a problem with your iSCSI targets. Do you have already VMs running on the iSCSI datastores?

[BorjaGonzalez]

Thanks for the reply Patrick.

No there are not VMs running in the datastores. That's a lab enviroment and they are all empty less where the VM to migrate reside. And the VM is always powered off.

[BorjaGonzalez]

Hi Guys i have discovered the problem source (i guess). The problem is the latency.

When i try to migrate a VM from local to shared datastore i have 2000 ms!! (Average) with peaks of 40.000 - 50.000. That's cause it fails i guess, but when i try to migrate a VM from shared to local i only have 20 ms (max)

I have done these test from Local datastore to Microsoft iSCSI LUN.
Local to FreeNAS fails so fast that i can't see the latency but imagine something similar.

I give you some screenshots to see the graphics.

What can i do to fix this? Any ideas?

Thanks a lot to everyone!

[Nick_Andreev]

What are you using as underlying storage? They all are sitting on the same disks?

[BorjaGonzalez]

Microsoft VM Provider
     1 System Disk (C:)
     1 Disk (D:)
          1 LUN (Provided to ESXi and where i did the tests and migrations)

[Nick_Andreev]

I'll make a wild guess and say that VAAI may have something to do with this. Can you try and disable it and see if it makes any difference?

Refer to the following KB below, section "Disabling VAAI using the vSphere Client"

VMware KB: Disabling the VAAI functionality in ESXi/ESX

[BorjaGonzalez]

The LUNs that i use to test it don't support VAAI anyway so i don't think that's the cause of the problem.
I will try anyway, wait for my reply!

Thank you.

[Nick_Andreev]

You're using FreeNAS, I think they introduced VAAI support in one of the later releases.

You can see if VAAI is enabled by going to the host storage settings and checking the Hardware Acceleration column in the list of datastores.

[BorjaGonzalez]

Same results Nick, VAAI isn't the problem
I don't know what can be but i was working around that for 2 weeks and still with that problems, but little by little is going better to fix it!

More ideas?

[Nick_Andreev]

Microsoft VM Provider

     1 System Disk (C:)

     1 Disk (D:)

          1 LUN (Provided to ESXi and where i did the tests and migrations)

Is this a Windows host and ESXi running in VMware Workstation or something along these lines?

[BorjaGonzalez]

ESXI (Physical)
     - VM ESXi
          1 Local datastore
          1 LUN (Provided from windows VM)
     - VM Windows

          1 System Disk (C:)

          1 Disk (D:)

               1 LUN (Provided to ESXi)

[BorjaGonzalez · Accepted solution]

Finally i have found the solution!! The problem was that i have been using E1000E vmnics instead of vmxnet3. I have configured a vmxnet3 adapter and boom! 20 ms in all the migrations!!

Thanks a lot to everyone for the help, specially to Nick_Andreev‌ !

[Nick_Andreev]

Great to hear you have solved the problem. And thanks for letting us know what it was.