Skip navigation
2012

Kevin Barrass's Personal Blog

November 2012 Previous month Next month

In this Blog Post I will describe how I have configured VXLAN's over a Multicast enabled Layer 3 network. I will show the router configs and the associated multicast routes created and the host VXLAN mappings.

 

This lab is a physical lab rather than a virtual one on VMware Workstation. I hope to cover how to do this on a virtual lab in a later Blog Post. The lab is based on two hosts each with 2 Nics, a small PC for vCenter, vSphere Client and shared storage. The vShield Manager "VSM" is deployed onto the one of the two ESXi hosts. The lab is based on vSphere 5.1 and vCNS 5.1.

The network is based on two Cisco routers and a Switch for vSphere PC as shown in the diagram below:

 

vxlan over l3 lab.png

 

On the network I have deployed PIM Sparse-Mode "PIM-SM" with R1 as the rendezvous point for both routers R1 and R2. I have used PIM-SM as apposed to sparse-dense mode which seems to be the recommendation. My reasons for this is that I have more experience of PIM-SM so this seems a good starting point for me to learn VMwares implementation of VXLAN.

 

As the previous blog I will not be deploying a VXLAN gateway just yet and will be concentrating on just VXLAN itself. The VSM and VXLAN preparation is identical to my previous 2 part Blog Post "Simple VXLAN lab on Workstation viewing traffic with Wireshark".

The only difference here is that each ESXi hosts vmk1 interface is now in a different layer 3 and layer 2 segment.

Host ESXi1 VXLAN interface vmk1 is in subnet 192.168.150.0/24

Host ESXi2 VXLAN interface vmk1 is in subnet 192.168.136.0/24

Each physical router acts as a DHCP server for each ESXi host.

 

Preparation for VXLAN is now as per below:

 

prepared cluster.png

Below are the configs I used for routers R1 and R2, a simple network based on two routers in a single PIM-SM domain both running OSPF in a single area 0.

 

Router "R1" config:

 

!
hostname R1
!
ip multicast-routing
!
interface Loopback0
description PIM RP
ip address 172.16.1.1 255.255.255.255
ip pim sparse-mode
!
interface FastEthernet0/0
description Link to Host esxi1
ip address 192.168.150.254 255.255.255.0
no ip proxy-arp
ip pim sparse-mode
duplex auto
speed auto
!
interface FastEthernet0/1
description Link to R2-Fa0-1
ip address 10.1.0.1 255.255.0.0
ip pim sparse-mode
duplex auto
speed auto
!
router ospf 1
log-adjacency-changes
passive-interface FastEthernet0/0
network 10.0.0.0 0.255.255.255 area 0
network 172.16.1.1 0.0.0.0 area 0
network 192.168.0.0 0.0.255.255 area 0
!
ip pim rp-address 172.16.1.1
!
ip access-list standard VXLAN-1-BOUNDARY
deny   224.1.1.50
permit 224.0.0.0 15.255.255.255
!

 

Router "R2" config:

 

!
hostname R2
!
ip multicast-routing
!
interface FastEthernet0/0
description Link to Host esxi2
ip address 192.168.136.254 255.255.255.0
no ip proxy-arp
ip pim sparse-mode
duplex auto
speed auto
!
interface FastEthernet0/1
description Link to R1-Fa0-1
ip address 10.1.0.2 255.255.0.0
ip pim sparse-mode
duplex auto
speed auto
!
interface FastEthernet1/0
ip address 10.3.0.1 255.255.0.0
ip pim sparse-mode
duplex auto
speed auto
!
router ospf 1
log-adjacency-changes
passive-interface FastEthernet0/0
network 10.0.0.0 0.255.255.255 area 0
network 172.16.1.1 0.0.0.0 area 0
network 192.168.0.0 0.0.255.255 area 0
!
ip pim rp-address 172.16.1.1
!
ip access-list standard VXLAN-1-BOUNDARY
deny   224.1.1.50
permit 224.0.0.0 15.255.255.255
!

 

As in the previous lab I have two OpenBSD VMs deployed, VM01 on host ESXi1 and VM02 on host ESXi2. I have then created a single VXLAN named vxlan-01 with a VNI of 5000 using the multicast group 224.1.1.50.

 

VM's VM01 and VM02 are in subnet 172.16.0.0/24, VM01 with the IP 172.16.0.2 and VM02 with the IP 172.16.0.3.

 

With the VM's deployed and their vNics a member of vxlan-01 as expected there are no hosts joined to the multicast group 224.1.1.50 and there are no active multicast sources for the group 224.1.1.50.

 

VMs Powered Off mroute.png

VMs Powered Off IGMP.png

Now we will power on both VM's VM01 and VM02. As soon as the VM's are powered on even before the guest OS of the VM's has booted up each host now joins the multicast group 224.1.1.50 through IGMP version 2.

 

We now have multicast routes in place for the two hosts that have joined the multicast group 224.1.1.50.

 

VMs Powered On mroute ASM only.png

We can also see the IGMP membership report for group 224.1.1.50 on each router.

VMs Powered On IGMP.png

 

At this point as no packets such as broadcast, unknown unicast or ARP have been send from VM's VM01 and VM02 and therefore nothing has had to be encapsulated in the multicast group 224.1.1.50 by either host ESXi1 or ESXi2 so no multicast sources for group 224.1.1.50 are registered with the rendezvous point.

 

Now a ping session is started from VM01 on Host ESXi1 to VM02 on host ESX2. At this point host ESXi01 encapsulated the ARP request packet into a multicast packet and transmits it on the group 224.1.1.50 with a VXLAN header for VNI 5000. The router R1 will register the source 192.168.150.128 for the group 224.1.1.50 with the rendezvous point. The host ESXi2 will receive the Multicast packet for group 224.1.1.50 and VNI 5000, decapsulates it and send onto the recipient VM whilst adding the source VM Mac address, host and VXLAN mapping into its VXLAN mapping table. Router R2 will then send a source specific join towards the host ESXi1 for the group 224.1.1.50 (192.168.150.137,224.1.1.50). When the router R1 is recieving duplicate packets one from Shared tree and one from the now formed shortest-path-tree, the router R2 will switch over to the shortest-path-tree.

 

We now have the below multicast routes in place showing host ESXi1 as a source for group 224.1.1.50.

 

VMs Powered On mroute SSM source.png

Only the original ARP request is passed over multicast the rest of the ICMP session is passed over Unicast between the host's encapsulated in a VXLAN packet for VNI 5000.

 

On the previous Blog Post I put up on VXLAN, the hosts VXLAN mapping table had an outer MAC that matched the recipient hosts vmk1 MAC address.

In this use case the outer MAC now has the outer MAC of the 1st hop router i.e. R1 fa0/0 or R2 fa0/0 as shown below, I presume this MAC address is learned when a host receives a VXLAN frame from another host as the source MAC address will be that of the egress router and alleviates the need for proxy-arp on the routers or a separate kernel default route for the VXLAN network.

 

ESXi VXLAN mapping L3.png

Anyway, this is a short Blog Post just to hopefully describe basically how VXLAN can be used over a Layer 3 multicast enabled network.

In a future Blog Post I will look at the VXLAN gateway "vCNS Edge" and how it can be used to connect from on VXLAN to another or from the "real world" into a VXLAN. I will also cover NAT and firewall services on the vCNS Edge and how you can use the Edge CLI to aid fault finding.

 

The above is based on my understanding of both PIM-SM and VXLAN so may well be wrong, then again may hopefully be right

 

Thanks for reading.

 

Kevin Barrass

My 1st lab I'm blogging shows how you can setup a simple virtual lab to create VXLAN's on then use wireshark to view the VXLAN traffic and peek inside the VXLAN traffic by decoding the packets.

 

On my lab I have a single VM for vCenter 5.1, 2 ESXi 5.1 hosts and a shared storage virtual appliance.

The vCenter and each host as well as storage appliance has a vNic in the VMNet1(host only) Workstation network. Each host also has a vNic in the VMNet4(host only) Workstation network that is DHCP enabled for VXLAN transport. My lasptop has the IP address 192.168.142.1 and has vSphere 5.1 client and Wireshark installed. I have created a single DC with one cluster that I have deployed the vShield Manager "VSM" into. I have also created two VM's for testing. I've used OpenBSD as they have a small footprint and can run various services for testing network traffic.

 

Please note figure 1.0 in Part two of this blog post on how the lab was built on VMware Workstation.

 

Once the VSM has booted up and had its IP, Subnet, default gateway and DNS configure from the VSM CLI I registered it with my vCenter Server.

 

register VSM.png

 

Once the VSM plugin has registered with vCenter Client you navigate to the DC and click the "Network Virtualization" tab. Then click on preperation and edit for the segment ID. Enter some test values in this case a multicast group range of 224.1.1.50-224.1.1.60 and segment ID pool of 5000-5100.

 

create segment ID.png

 

Once the segment ID is added click on connectivity>edit and select the cluster that you want to deploy VXLAN over. This assumes I have already created the vNetwork Distributed switch version 5.1 accross all hosts in the cluster.

 

select cluster.png

 

At this point VXLAN will be deployed onto each host in the cluster, a new VMkernel interface for VXLAN will be created on each hosts in my case vmk1. The vmk1 interfaces will be assigned an IP address using DHCP from the VMware workstation VMNet4 IP address pool.

 

At this point the Cluster is prepared for VXLAN and should have a status of Normal.

 

prepared.png

 

We now create the Scope for the VXLAN's in this case a single cluster. Select add then give the scope a name, descipton and select the cluster(s) for this scope.

 

add scope.png

 

Great, VXLAN is now deployed and ready for us to create our 1st VXLAN. Each VXLAN wil be mapped to a multicast group, if we assigned just two multicast addresses to our range and created 3 VXLANs you would have 2 virtual wires mapped to a single multicast group. This is not a problem as far as I know but you would need to be mindful of mutliple VXLANs broadcast domains being on the same multicast tree and would make pruning each broadcast domain back harder due to a single multicast group for multiple VXLAN's so might need carefull thought and planning.

 

So now to create our 1st VXLAN, simply click on Networks then the green cross for add. Give the VXLAN a name, description and scope which in this case has to be scope-01. Then click ok. The VXLAN is then created and an associated Portgroup is created. The VXLAN is given a VNI "VXLAN Network Identifier" of 5000 and mapped to the multicast group 224.1.1.50.

 

create virtual wire.png

Now we will add each of our two OpenBSD VM's into the VXLAN,

 

assign nic to vxlan.png

 

Now run a continous ping from VM01 "172.16.1.2" to VM02 "172.16.1.3". Then open the packet capture tool Wireshark and choose the interface for VMNet4 which is the virtual network on Workstation we are transporting the VXLAN traffic over.

 

Now start the packet capture whilst the ping session is still running from VM01 to VM02. It is important to make sure that VM01 and VM02 are NOT running on the same host or you will be sat like I was wondering why I didnt see any traffic. The reason for this is we want to see the VXLAN traffic going between hosts over the VXLAN transport network.

When the packet capture starts you should see the VXLAN UDP traffic with a source IP of one ESXi host and a destination of the other ESXi host, with a destination UDP port of 8472. Looking at the packet capture we can only see that it is a VXLAN packet, and that it has a payload. The important thing here is that the traffic is unicast what we havent done is captured the traffic that transported the ARP request using multicast, we will do this later in this blog.

 

packet capture none decoded.png

What we can now do is use the VXLAN decoder in Wireshark to look agt the VXLAN header and the contents of the packet. To do this select a packet and right click then select "decode as" and choose a transport protocol of VXLAN.

 

packet capture set decode to vxlan.png

 

Now we should see the contents of the VXLAN payload i.e. the original packet which in this case is the ICMP echo-request and echo-reply. You can also view the VXLAN header showing the VNI of 5000 which is the VXLAN Network Identifier which was assigned to our VXLAN.

 

packet capture decoded.png

 

As we missed the original ARP request sent by VM01 for VM02's MAC address from VM01 we will run a continous ping to an IP address not in use 172.16.1.4. this will keep VM01 sending an ARP request out as a broadcast. VXLAN will then encapsulate this packet into a multicast packet onto the multicast group 224.1.1.50 and this will be sent to all hosts that would have joined this group through IGMP. In our case we are not using PIM or IGMP snooping. As you can see in the none-decoded traffic flow below the source remains the hosts IP address but the destination is now the multicast group for this and possibly other VXLAN's.

 

packet capture ARP none decoded.png

 

In part 2 of this blog post we will decode the VXLAN multicast traffic, and show what information you can view on the ESXi hosts via an SSH session.

 

Kevin Barrass

Now that we have captured an ARP request that has been encapsulated in multicast by VXLAN we will now use the same option in Wireshark to decode the VXLAN packet. This shows the same VXLAN header and the encapsulated ARP request. The host that received this encapsulated ARP request would add the MAC address of the requesting VM in this case VM01 to its VXLAN mapping.

 

packet capture ARP decoded.png

 

If we re-run the constant ping from VM01 to VM02 we can log into each ESXi host and view its VXLAN mapping of VM MAC addresses. In this case we enable SSH on each host, log into the host and run the command:

 

esxcli network vswitch dvs vmware vxlan network mapping list --vds-name=dvSwitch --vxlan-id=5000

 

Where dvSwitch is our vNetwork Distributed switch name and 5000 is the VNI of our VXLAN. This will then show the mappings for our two VM's VM01 and VM02. The inner MAC is the MAC address of the VM and both the outer MAC and outer IP are that of the recipient ESXi hosts vmk1 VMKernel interface. If we were doing VXLAN over a routed network the out IP would still be that of the recipient ESXi host but the outer MAC would be that of the next hop PIM router. I will cover this in a later blog post on VXLAN over a pure layer 3 network using PIM Sparse Mode.

 

VXLAN mappings for VNI 5000

 

show vxlan mapping.png

 

As you can see the destination for VM01 on host ESXi1 matches the MAC and IP address of that hosts VMKernel interface vmk1.

vxlan vmk1 interface.png

Another thing you can do is to view from the ESXi cli the VNI to multicast group mapping using the command:

 

esxcli network vswitch dvs vmware vxlan network list --vds-name=dvSwitch --vxlan-id=5000

 

There are more combinations of "esxcli network vswitch dvs vmware vxlan" command than Ive played with or could cover in this blog post.

show vxlans and mcast groups via ssh.png

Thats the end of this blog post, as this is my 1st lab blog post I really appreciate any constructive feedback as I hope to add more blogs posts as I lab things up at home depending on free home time

 

VXLAN Lab setup as per Figure 1.0 below:

Computer with 16GB RAM, quad core i5 CPU and Solid state disk.

VMware Workstation verson 8

One VM for vCenter, one VM for shared storage and two ESXi VMs with virtualised VT-x enabled. vShield Manager as a VM on the virtual ESXi hosts.

 

Figure1.0

vxlan blog 1.gif

 

I hope to create a blog post for each of the below in the coming months:

 

          VXLAN over Layer 3 using PIM-SM

          vCNS Edge SSL VPN.

          vCNS Edge firewall and CLI/debugging

          and hopefully more.....

 

Thanks for reading.

Kevin Barrass

kjbarrass Novice

Welcome to my Blog

Posted by kjbarrass Nov 21, 2012

Only just set up this blog but hope over the coming weeks and months to upload how to build VXLAN/vCNS and vCloud labs on VMware Workstation. I will also put up examples of how to do packet captures to get a better understanding of how VXLAN works.

 

Kev Barrass