override default gateway for vmkernel adapter (vsphere 6.5)

[mishaelpl]

hey,

I am using vsphere 6.5u3 enterprise plus.

I need to add additional vmkernel adapter to 100+ hosts for vsphere replication traffic. The additional vmkernel adapter will be in a dedicated L3 subnet.

I want to use powercli/api to achieve this. So far I was able to create vmkernel and enable vsphere replication traffic using PS, but i am struggling to override default gateway on vmk.

I've researched this and it seems that nobody provided the solution to this yet. All solutions were based on changing default gateway for default tcp/ip stack, or creating new tcp/ip stack.

I came across a potential method, but i don't know how to proceed further. Any help would be appreciated!

$esx = get-vmhost $hostname

$vmk = $esx.ExtensionData.Config.Network.Vnic | where device -eq "vmk2"

$vmk.Spec.IpRouteSpec.IpRouteConfig.DefaultGateway  # this seems to be the setting to modify, but how?

$gw = New-Object vmware.vim.HostVirtualNicIpRouteSpec

$gw.IpRouteConfig.DefaultGateway = "X.X.X.X" # error: The property 'DefaultGateway' cannot be found on this object. Verify that the property exists and can be set.

?

reference:

https://vdc-repo.vmware.com/vmwb-repository/dcr-public/6b586ed2-655c-49d9-9029-bc416323cb22/fa0b429a...

[scott28tt]

Moderator: Moved to PowerCLI

[LucD]

Do you have some more info on that potential method?
Afaik, creating a new TCP/IP stack is the only way to have a different gateway than the default one.

[mishaelpl]

that's pretty much all I figured out so far 😕

when I override default gateway on vmk using vsphere client and then execute this:

$esx = get-vmhost $hostname

$vmk = $esx.ExtensionData.Config.Network.Vnic | where device -eq "vmk2"

$vmk.Spec.IpRouteSpec.IpRouteConfig.DefaultGateway

it picks up the value Iset, so it must be that setting.

I got it from here:

https://vdc-repo.vmware.com/vmwb-repository/dcr-public/6b586ed2-655c-49d9-9029-bc416323cb22/fa0b429a...

The description says:

"The ip route configuration used by the vmkernel adapter. This attribute allows the vmkernel adapter to specify its own default gateway.

Since vSphere API 6.5"

The override vm kernel adapter setting in vsphere client was also introduced in vi 6.5.

Just don't know how to apply configuration changes...

[LucD · Accepted solution]

Ok, gotcha.

That should work with the esxcli command as well.

Something like this

$esxName = 'MyEsx'

$vmkName = 'vmk1'

$esx = Get-VMHost -Name $esxName

$esxcli = Get-EsxCli -VMHost $esx -V2

$if = $esxcli.network.ip.interface.ipv4.get.Invoke(@{interfacename=$vmkName})

$iArg = @{

    netmask = $if[0].IPv4Netmask

    type    = $if[0].AddressType.ToLower()

    ipv4    = $if[0].IPv4Address

    interfacename = $if[0].Name

    gateway = '192.168.10.222'

}

$esxcli.network.ip.interface.ipv4.set.Invoke($iArg)

$esxcli.network.ip.interface.ipv4.get.Invoke(@{interfacename=$vmkName})

[mishaelpl]

as a workaround, I was able to do this using ansible and vmware_vmkernel module.

vmware_vmkernel – Manages a VMware VMkernel Adapter of an ESXi host — Ansible Documentation

it accepts parameters to override default gateway and setup all required services for vsphere replication.

Example:

-  name: Add Management vmkernel port to Distributed Switch

   vmware_vmkernel:

      hostname: '{{ vcenter_hostname }}'

      username: '{{ vcenter_username }}'

      password: '{{ vcenter_password }}'

      esxi_hostname: '{{ esxi_hostname }}'

      dvswitch_name: dvSwitch1

      portgroup_name: dvPG_0001

      network:

         type: 'static'

         ip_address: 192.168.127.10

         subnet_mask: 255.255.255.0

         default_gateway: 192.168.127.1

      state: present

      enable_replication: True

      delegate_to: localhost

It would be awesome to know if this can be done via API/Powercli too though...

[mishaelpl]

just saw your msg, will check it out too. Definitely prefer powercli over ansible 😉

[mishaelpl]

you are the magician indeed! It works like a charm. Thank you so much.

[alphenit]

@LucD Is overriding the default gateway for a vmkernel not a valid way to change the gw for the services that the vmkernel in question delivers?
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.networking.doc/GUID-44CAFD7C-6352-4...

[LucD]

Each service that runs on a TCPIP stack uses the gateway for that stack.
The vMotion service runs/should run on the vMotion TCPIP stack, which is another stack, which can have a different default gateway.

[alphenit]

@LucDI would agree with you. However I do find the text in the VMWare Docs that I pasted a bit contradicting:
###################
You might need to override the default gateway for a VMkernel adapter to provide a different gateway for vSphere vMotion.

Each TCP/IP stack on a host can have only one default gateway. This default gateway is part of the routing table and all services that operate on the TCP/IP stack use it.

For example, the VMkernel adapters vmk0 and vmk1 can be configured on a host.

• vmk0 is used for management traffic on the 10.162.10.0/24 subnet, with default gateway 10.162.10.1
• vmk1 is used for vMotion traffic on the 172.16.1.0/24 subnet

If you set 172.16.1.1 as the default gateway for vmk1, vMotion uses vmk1 as its egress interface with the gateway 172.16.1.1. The 172.16.1.1 gateway is a part of the vmk1 configuration and is not in the routing table. Only the services that specify vmk1 as an egress interface use this gateway. This provides additional Layer 3 connectivity options for services that need multiple gateways.
###################
So each TCP/IP stack can only have one gateway (10.162.10.1) but if you override the gateway for vmk1 (that uses the same stack), the service that uses vmk1 goes out gateway of 172.16.1.1  ?

[LucD]

This is then a static route over vmk1, as the text confirms ("... is not in the routing table").
Which for me is not a "default gateway" in the real sense of that expression.

Since there is a TCPIP stack for vMotion, I would wonder why you would like to use this for a vMotion vmk on the Default TCPIP stack.
See also Place vMotion Traffic on the vMotion TCP/IP Stack of an ESXi Host.

In the VMware Virtual SAN Layer 2 and Layer 3 Network Topologies document (p6), that same concept of static routes is explained for VSAN (in vSphere 6, where the was no VSAN TCPIP stack yet).

[alphenit]

@LucD thanks for explaining this further.
No I would not use this for vMotion (would use the vMotion tcp/ip stack for sure) but it would apply for other services such as NFS.

The wording you used is much clearer than what is mentioned in the VMWare docs. 
Thanks again @LucD  !