VMware Cloud Community
Slingsh0t
Enthusiast
Enthusiast

VXLAN cluster preparation issue

Guys I have successfully deployed vCloud Director 5.1 along with the vShield Manager and have deployed a few vApps and VCDNI networks etc...  All seems to be OK there.

The issue I'm having is I want to test VXLAN.  When I go to prepare a cluster, I see a new portgroup appear on the DVswitch "vxw-vmknicPg-dvs.....blah" however no vmkernel port is created on the port group.  From what I've read, a vmk port should be created and its on this interface that DHCP waits for an address.  I'm unable to manually create a vmk port on the newly created VXLAN port group as I get the following error message:

  • Call "HostNetworkSystem.AddVirtualNic" for object "networkSystem-1944" on vCenter Server "PVPVC01P201.poc.pvp.lon5.telecity.net" failed.
    Operation failed, diagnostics report: Sysinfo error on operation returned status : Would block. Please see the VMkernel log for detailed error information

In the end I need to unprepare the cluster to remove the portgroup.  I have rebooted the hosts, vSM and vCenter.  This has made no difference.

27 Replies
firestartah
Virtuoso
Virtuoso

Hi

Normally the vmkernel for the vxlan is only completed once you have done all the configurations for the DVS. Have a look at this VMware blog article for an idea of the steps you normally follow

http://blogs.vmware.com/vcloud/2012/10/vcloud-director-5-1-vxlan-configuration.html

Gregg

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful". Gregg http://thesaffageek.co.uk
Reply
0 Kudos
Slingsh0t
Enthusiast
Enthusiast

Thanks for the reply.  I have read that article before (throughout my endless googling to try and identify a cause) but the first steps are the ones that aren't working for me?

Steps 1 - 4 are exactly what I'm trying to do and it fails on Step 4 where it says "New dvPort Groups and vmknic interfaces are added and automatically created on the Distributed Switch associated to the VXLAN".  The portgroups are created but the vmknic interfaces aren't.

I have enabled Jumbo frames on the switching infrastructure and confingured the DVswitch MTU to 1600 before attempting to prepare the cluster.

Any other ideas or is there something I'm missing?

Reply
0 Kudos
firestartah
Virtuoso
Virtuoso

Hi

Do you have spare vmnics for the auto creation of the vmk to utilise? I have seen it before with vCAC and vCO where if there wasn't a spare/unallocated vmnic then the creation would fail

Gregg

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful". Gregg http://thesaffageek.co.uk
Reply
0 Kudos
Slingsh0t
Enthusiast
Enthusiast

In the latest test, I isolated a single ESXi host in its own cluster and created a new empty DVswitch.  I bound two physical vmnics to the dvswitch and then tried to prepare this one-host-cluster and the same thing happened.  The portgroup was created but no vmk port?

Reply
0 Kudos
jun0
Contributor
Contributor

I am facing the same issue.

The portgroup was created but no vmk port was created.

Reply
0 Kudos
findogg
Contributor
Contributor

Ditto...

Same issue here exactly.

The port group was created on the distributed switch, but NO vmkernel ports.

Any advice would be greatly appreciated!!

Thanks.

Reply
0 Kudos
b0rbb
Enthusiast
Enthusiast

We have the same issue as well.  We do have an SR with VMware on this, but with no luck on any of the suggested resolutions.

What seems to be the problem is the during the preparation process, the hosts will try to pull the VIB from vCenter Server, which in turn pulls the VIB from vShield Manager @ https://vsm-ip/bin/vdn/vibs/5.1/vxlan.zip.  I can download this zip file directly on the vCenter Server with a browser, so no network issues there, but the eam.log file in C:\ProgramData\VMware\VMware VirtualCenter\Logs shows the following error upon any attempt to download the VIB:

host-4 | VibTask.java | 148 | Failed to complete VIB task:
java.lang.IllegalStateException: Client version not specified

We get this over and over again with any attempt to prep the hosts with VXLAN.  Oddly enough, however, if I upload the aforementioned vxlan.zip file to a host and install it manually via ESXCLI, any further attempts to prep the hosts will add the necessary VMK to hosts with the VIB already installed.

This is also with all the hosts, vCenter Server, AND vShield Manager on the samve IPv4 subnet.  We've tried turning off the Windows Firewall completely on the vCenter Server, and enabled the firewall rule for outgoing traffic to TCP/80 and TCP/443 on the hosts with no luck. 

To anyone else with this issue, do you also see the above error in your eam.log file on the vCenter Server?

Reply
0 Kudos
sachinthakkar
Enthusiast
Enthusiast

Do you have VUM installed in your environment? Is port 80 opened between VC & ESX?

Reply
0 Kudos
sachinthakkar
Enthusiast
Enthusiast

@findogg: What vShield Manager version are you running? Can you try to move the host to maintenance mode and back? If that creates a vmknic, this may be a known issue. Let me know what you find and I'd be glad to help

Reply
0 Kudos
b0rbb
Enthusiast
Enthusiast

We actually ended up getting our issue resolved with support & some of the product folks (not sure if you're the same Sachin that was there too, haha).

Long story short, it appeared that the upgrade from vShield Manager 5.1.1 to 5.1.2 changed how the VXLAN deployment works as far as utilizing VUM or not for the VIB installs to the ESXi hosts.  5.1.1 would utilize VUM for pushing the VIBs to the hosts, while 5.1.2 changed this behavior to instead bypass VUM.  Ironically, the reason for this change was to fix this very issue (VIBs not getting installed/VMKnics not getting created), yet changing a MOB config to essentially go back to the 5.1.1 behavior of going through VUM fixed this for us.  Go figure.

Our only concern is that we haven't received the necessary documentation from support on where that actual MOB config was located at in vCenter for changing this behavior back to the 5.1.1 style behavior of going through VUM. We were told we were going to get at least some kind of rudimentary documentation for that prior to our ticket being closed, but the ticket was closed without any documentation sent to us.

Reply
0 Kudos
12074n12
Contributor
Contributor

Hi,

Can I know what did you actually do?

I am facing exactly the same issue.

1.jpg

I am using vSM 5.1.1-848085.

I have tried to unprepare the cluster, reboot the host, but still face the same issue.

Thanks

Regards

Reply
0 Kudos
sachinthakkar
Enthusiast
Enthusiast

On the VXLAN vmknic DVPG, the vShield manager must create the vmknic. We don't allow user creation as this will affect the teaming mechanisms used by VXLAN. If the VSM creates the vmknic, it should go through. I'd try to initiate the action from there (either through a prep operation or the REST API)

Reply
0 Kudos
12074n12
Contributor
Contributor

Hi sachinthakkar,

Yes, I know  initiate the action from there (either through a prep operation or the REST API)hat VSM should auto-create vmknic durinc cluster preparation.  But in my case, it didnt, so I try to make it manually from vsphere client. Then it prompt with the error as shown above.

Can I know what do you mean by "initiate the action from there"? By there, you refer to VSM, right?

How do we do this?

Really appreciate your help.

Thanks

Regards

Reply
0 Kudos
rksam
Contributor
Contributor

I am running into a similar issue, though not identical.  Sorry if this is the wrong thread, but please feel free to move to the correct thread.

I have prepared everything required for vxlan and when i ping from guest-VM, it is not working.  I see that the packets are not seen in the vmk1

(vtep) interface.  The mcast groups are setup on the external routers.  The ping from vtep to vtep is working fine.

Here is the setup.

server-1, guest-vm-1, vtep-1(vmk1)

server-2, guest-vm-2, vtep-2(vmk1)

I can ping from vmk1 to vmk1 from vtep-1 to vtep-2.

But when i send a ping packet from guest-vm-1 to guest-vm-2 it is 'destination host unreachable'.

On doing tcpdump-uw on vmk1 in server-1, i dont see the arp packets from guest.  Should i see this?

What am i missing?

Reply
0 Kudos
ealaqqad
Enthusiast
Enthusiast

Hi,

Just as b0rbb mentioned, your best bet is to get vCloud Networking & Security 5.1.2. I have documented this problem and resolution at my blog including a link to where you can get the just released vCNS 5.1.2 and release notes at: vCloud Networking & Security 5.1.1 create dvPort Groups, but fails to create vmknic interfaces

Please start using vCloud Networking & Security 5.1.2 for any future implementation, & upgrade your current one.

Regards,

Eiad Al-Aqqad

B: http://www.Virtualizationteam.com

B: http://www.TSMGuru.com

Regards, Eiad Al-Aqqad Technology Consultant @ VMware b: http://www.VirtualizationTeam.com b: http://www.TSMGuru.com
Reply
0 Kudos
simdevmat
Contributor
Contributor

Hi,

After upgrading to 5.1.2a I am still having the same issue. Has the upgrade fix this for everyone else?

Thanks

Simon

Reply
0 Kudos
ealaqqad
Enthusiast
Enthusiast

Hi Simon,

I have noticed that in some cases after the upgrade you will need to follow the below proces, which I just added to my original post:

- Remove the original VXLAN configuration.

- Restart the vCNS web service (Exact commands were listed in my post vCloud Networking & Security 5.1.1 create dvPort Groups, but fails to create vmknic interfaces)

- Re-Add vCenter to vCNS

- Add the VXLAN Configuration again.

This seems for me to fix the problem in my lab and at a couple of customers.

Good luck!

Regards,

Eiad Al-Aqqad

B: http://www.Virtualizationteam.com

B: http://www.TSMGuru.com

Regards, Eiad Al-Aqqad Technology Consultant @ VMware b: http://www.VirtualizationTeam.com b: http://www.TSMGuru.com
Reply
0 Kudos
ITMan01
Contributor
Contributor

Hi ealaqqad,

What do you mean re-add vCenter to VCSN? Do you mean as below, just reenter the password and click OK? or is there any other steps?

2013-3-15 12-39-06.png

Reply
0 Kudos
ealaqqad
Enthusiast
Enthusiast

Hi ITMan01,

That's right. That is the screen where I meant to re-add vCenter to. You might as well try to add both vCenter & vShield to your vCD.

Regards,

Eiad Al-Aqqad

B: http://www.Virtualizationteam.com

B: http://www.TSMGuru.com

Regards, Eiad Al-Aqqad Technology Consultant @ VMware b: http://www.VirtualizationTeam.com b: http://www.TSMGuru.com
Reply
0 Kudos