Adding Static route on ESXi 5.5!

[saki893]

Hi,

I am trying to add a static route in the esxi using the command "esxcli network ip route ipv4 add -g x.x.x.x -n x.x.x.x/xx". When I run this command a static route is created and is applied on vmkernel port group vmk0 by default. I want the static route applied to vmk3 instead of vmk0. Please advice me on how to do this.

Thanks

Sai

[VirtuallyMikeB]

Hello,

Are vmk0 and vmk3 on the same subnet? Is the gateway you're adding on the same subnet as vmk3?  Can you share the output of esxcfg-route -l, please, after you run the command?

Thanks,

Mike

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown

[saki893]

Mike,

vmk0 and vmk3 are on different subnets. vmk0 is the management kernel port using 10.20.27.253. vmk3's default gateway is 10.28.187.253. I can't edit the vmk3 ip settings to use the 10.28.187.253 as default gateway. It inherits the 10.20.27.253 from the management ipv4 settings. So I want to configure a static route between 10.28.184.0 network and 10.20.27.253. The below is the output you asked for after running the command.

esxcli network ip route ipv4 list

Network       Netmask        Gateway       Interface  Source

------------  -------------  ------------  ---------  ------

default       0.0.0.0        10.20.27.253  vmk0       MANUAL

10.28.184.0   255.255.252.0  10.20.27.253  vmk0       MANUAL

Thanks

Sai

[VirtuallyMikeB]

Hello Sai,

So what is the full command you're using?  If I'm understanding correctly, you want vmk3 to use it's own gateway for all traffic.  That's fine,- you'll simply use one static route for every subnet you want to originate at vmk3.  It looks like you're using the following command.  If so, it's not what you want.

esxcli network ip route ipv4 add -g 10.28.187.253 -n 10.28.184.0/22

The command above says tells ESXi, "if you want to reach the 10.28.184.0/22 network, use the gateway 10.28.187.253."  And vmk3 is already on that network.

You want to replace the subnet portion of that command with the destination network you want to reach.  You'll want something like

esxcli network ip route ipv4 add -g 10.28.187.253 -n 192.168.10.0/24

Where 192.168.10.0/24 is the subnet you want to reach via vmk3.  This can be considered a bad thing if you have many subnets to reach from vmk3, because that's many static routes on each host.  But it should work.

Can you share the full command you were using?

Thanks,

Mike

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown

[saki893]

Hi Mike,

Thanks for your support! The command I am using is as below:

esxcli network ip route ipv4 add -g 10.20.27.253 -n 10.28.184.0/22 where 10.28.184.0/22 is the network to reach via vmk3 and 10.20.27.253 is the main default gateway which is available in host's configuration , dns & routing section to be specific. That means the VM's deployed on the vmk3 port group should get the dhcp ipv4 addresses from 10.28.184.0/22 which is not happening instead some 169.x.x.x addresses are assigned.

thanks

sai

[VirtuallyMikeB]

So the command you're using is executing exactly as it should.  The output of esxcli network ip route ipv4 list shows exactly what you typed in.  This is not what you want, though.  Here's what you're telling it to do, shown with my high tech mspaint picture:

With the command you're running, you're telling ESXi that if it wants to reach the 10.28.184.0/22 network, it needs to go through the 10.20.27.253 gateway.  ESXi will not do this because vmk3 is already on that subnet and ESXi will choose a vmkernel port already on that subnet before using the routing table.

So let's get to your problem.  You have DHCP client VMs deployed in the 10.28.184.0/22 network that are getting APIPA addresses.  In which network are your DHCP servers?  I don't think this issue can be resolved by changing your ESXi vmkernel routing table.  Virtual machine traffic is not dependent on ESXi host routing.  If you're running Cisco switches with DHCP servers and DHCP clients on different subnets, you need "ip helper" statements on your routers or layer 3 switches to forward DHCP requests to the proper subnet.

Am I on the right track here?

Thanks,

Mike

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown

[saki893]

Mark,

What you mentioned regarding the DHCP servers make sense to me. I have to check with my peers on that. My objective here is to create a vmkernel port group vmk3 with ip address 10.28.184.10 in the 10.28.184.0/22 network having default gateway 10.28.187.253.During the creation of vmkernel port group I can assign the ipv4 address and subnet mask as desired, but can't set the default gateway to 10.28.187.253 as vmkernel port group by default inherits the default gateway 10.20.27.253 which I cant edit as the ESXi management interface will be down. So there is no way for the 10.28.184.0/22 traffic to go out through the 10.28.187.253 gateway. So I thought of adding the static routes might be helpful.

~Sai

[VirtuallyMikeB]

What is the purpose of creating vmk3? What will be its function?

If you want vmk3 to communicate outside its subnet, to a subnet *not* already configured on the ESXi host, keep using static routes.  The command will be as above:

esxcli network ip route ipv4 add -g 10.28.187.253 -n 192.168.10.0/24

Where 192.168.10.0/24 is the subnet you want to reach via vmk3.  This can be considered a bad thing if you have many subnets to reach from vmk3, because that's many static routes on each host.  But it should work.

For instance, let's consider you're using vmk3 for vSphere Replication traffic between Site A and Site B.  For security, you'd like to keep replication traffic segregated.  So at Site A, you have a vmkernel port created for vSphere Replication, call it vmk3, and vmk3 resides in subnet 10.28.184.0/22.  At Site B, your vSphere Replication appliance lives in subnet 192.168.10.0/24.

You want to force traffic from subnet 10.28.184.0/22 (and therefore vmk3) at Site A to subnet 192.168.10.0/24 at Site B, you must configure a static route.  The configuration would look like this:

esxcli network ip route ipv4 add -g 10.28.187.253 -n 192.168.10.0/24

You never have to configure a "default gateway" for vmk3 - rather you configure a static route.  When the ESXi host needs to send traffic to the 192.168.10.0/24 network, it will look to see if it has a vmkernel port already in that subnet.  It will see that it does not, so it will look at its routing table and find an entry that says to send traffic to the 10.28.187.253 address to reach that subnet.  So again it searches for a vmkernel port from which send the traffic and it finds that vmk3 is, indeed, already on that subnet.  ESXi will then forward traffic from vmk3, through the 10.28.187.253 gateway which eventually reaches the destination subnet.

Hope this helps,

Mike

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown

[saki893]

Mike,

Thanks for the detailed explanation. Here is the scenario. vmk0 is using 10.20.25.0/22 with default gateway 10.20.27.253 which is the esxi management kernel port group. Assume there is a scsi target at 10.28.184.10 in the 10.28.184.0/22 network with default gateway 10.28.187.253. For software isci I have to create a vmkernel port group. So assume I am creating the vmkernel port group vmk3 with ip address 10.28.184.8, 255.255.252.0 and default gateway 10.20.27.253 (not 10.28.187.253). In this case how will the esxi host reach the scsi server? As per your explanation above esxi looks for the vmkernel port group with ip address in the scsi target network. But my question is vmk3 is configured with irrelevant default gateway. How it can reach the 10.28.184.10 via 10.20.27.253 ? Please help me in understanding this.

Thanks

Sai

[VirtuallyMikeB · Accepted solution]

Hi Sai,

That's easy 🙂

vmk3: 10.28.184.8

iSCSI target: 10.28.184.10

vmk3 and your iSCSI target IP address are in the same subnet so there's no need for any gateway, default or not, or routing.  IP addresses within the same subnet can talk all day long without leaving the subnet or requiring communication via a gateway.

Yes, all vmkernel ports assume the default gateway of the management vmkernel port, but vmkernel ports other than management generally don't need to talk outside their own subnet in the first place.

Here are some examples of use cases for vmkernel ports and how they generally communicate:

NFS - should be talking only within its own subnet

iSCSI - should be talking only within its own subnet

vMotion - should be talking only within its own subnet

Fault Tolerance - should be talking only within its own subnet

vSphere Replication - can talk within its own subnet, perhaps a stretched VLAN if across sites, else a static route can be configured

Did I miss any vmkernel port types?  I think that about sums it up for the most part.

- Mike

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown

[saki893]

I cannot ping the vmk3 ip address from my desktop, so wondering if default gateway has something to do with that. Actually using this in NSX setup where this vmk3 will be used as a VTEP ip address.

[VirtuallyMikeB]

Ah, excellent.  So - without speaking to NSX or VXLANs or VTEPs - you'll only be able to ping vmk3 if your desktop is in the same subnet.  What subnet is your desktop in?  If it's in a different subnet, we're back to static routes.  You'll want to add a static route to your desktop's subnet using the gateway of the subnet vmk3 is in.

Now, that's just one part of this.  There are many other things that could be wrong in this setup, from the physical layer right on up.  I can't speak well enough to NXS or VXLAN networking to troubleshoot it, so I can't help too much there.

All the best,

Mike

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown

[cweinhold]

Until ESXi 5.5 came out, Mike's advice below is completely true. I.e., make sure there is no routing needed for any VMK's except the one you use for management. This is done by making sure NFS, iSCSI, vMotion, FT vmknics are don't require any routing (e.g., all NFS IP's are on the same subnet). And even with ESXi 5.5, this is the sensible way to go. But this is far from perfect. Take this ESXi vmknic layout:

vmk0   10.0.0.x  /24    default gateway = 10.0.0.1

vmk1    10.0.0.y  /24

vmk2     192.168.0.z  /24

With a traditional approach:

• vmk0 and vmk1 are on the same subnet. All vmk1 traffic to/from 10.0.0.xxx peers would egress and ingress on vmk1. But any vmk1 traffic that is routed would egress vmk0 and ingress vmk1.  This asymmetry is not a big deal if vmk0 and vmk1 are using the same vswitch and uplinks. But sometimes we bind vmk's to different uplinks to steer traffic. E.g., vmk1 might be vmotion on its own dedicated NICs.
• vmk2 is on a different subnet from vmk0. All vmk2 traffic to 192.168.0.xxx peers would go in/out vmk2. But, like above, any vmk2 routed traffic would egress vmk0 and ingress vmk2. That's not what people expect to happen! In this design, you probably don't want routing on vmk2 and would prefer that any accidental vmk2 routing configuration simply be dropped rather than routed out vmk0. Or, if you do want routing on vmk2, you would need to have added static routes.

This can be cleaned up with the "Custom TCP/IP stack" feature that is new to ESXi 5.5. It is equivalent to "VRF" in the networking world. SSH into the host and:

     esxcli network ip netstack add -N="stack_name"

Then within the vsphere web client, you can Manage | Networking | TCP/IP Configuration | <your-custom-stack>, and configure a unique set of DNS and routing information. Using the example above,

• vmk1 can now be given it's own default gateway of 10.0.0.1. Any routed vmk1 traffic will now egress vmk1 instead of vmk0.  This fixes the traffic steering problem.
• vmk2 can now be completely isolated if you don't want routing. E.g., it won't use vmk0's default route like it does today.
• vmk2 can also be fully routed. E.g., you can add default gateway 192.168.0.1, and be sure that the route will only be used by vmk2 traffic.

I encourage everyone deploying ESXi 5.5 to create custom TCP/IP stacks for their vmknic. Leave management, vmotion, FT on the default stack since ESXi 5.5 doesn't support them anywhere else. But use them for iSCSI, NFS, VXLAN/STT tunnels with NSX, and Nexus 1000v L3 control .

-Craig

[iforbes]

@cweinhold: Your solution is exactly what I was looking for. I'm dealing with routed NFS in my scenario. I've created an NFS vmkernel (vmk1) at 10.10.20.10. The ESXi management vmkernel (vmk0) is 10.10.10.10. The NFS target is 192.168.123.10. The ESXi default gateway is 10.10.10.1. So, as it stands now any NFS traffic directed to 192.168.123.10 gets routed via the default gateway. ESXi sees multiple vmkernels and doesn't know what to use, so it will chose the first vmkernel (vmk0). In the past the NFS target would have been on the same subnet as my NFS vmkernel, but not the case here.

With the custom TCP/IP stack featue (I'm running ESXi 6) I can give the NFS vmkernel is own gateway (10.10.20.1). I take it that I'll have to create a static route for the NFS traffic to use the NFS vmkernel gateway though?

Ian

[iforbes]

Hmmm. Looks like the stack feature doesn't actually work. Have you gotten it to work?

TCP/IP Stack Configuration issue in vSphere 6 | VIRTUALIZATION BLOG

Ian