How migrate one vmk0 from VSS to VDS in one swoop

[dwchan]

I am having some issues when trying to use powerCLI to migrate the vmk0 from VSS to VDS within my nested ESX environment.  My current code looks something like his as I am trying to migrate the vmkt0 from the default portgroup on VSS over to a portgroup on VDS

        foreach ($vmhost in $vmhosts) {

            $vmhostname = $vmhost.name

            # Migrating VMkernel port (vmk0) on vSwitch to VDS

            My-Logger "Migrate $vmhostname VMkernel interfaces to VDS $VDSName..."

            $vmk = Get-VMHostNetworkAdapter -Name vmk0 -VMHost $vmhost

            $vdPortgroup = Get-VDPortGroup -VDSwitch (Get-VDSwitch -Name $VDSName) -Name $VLANMGTPortgroup

            Set-VMHostNetworkAdapter -PortGroup $vdPortgroup -VirtualNic $vmk -confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            }

No matter what I do, it will fail with the following error

Set-VMHostNetworkAdapter : 10/31/2020 2:23:59 AM Set-VMHostNetworkAdapter An error occurred while communicating with the

remote host. Network configuration change disconnected the host 'esx101.tataoui.com' from vCenter server and has been rolled

back.

At D:\VMware\vsphere-6.7-vghetto-standard-lab-deployment4.ps1:990 char:13

+             Set-VMHostNetworkAdapter -PortGroup $vdPortgroup -Virtual ...

+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Set-VMHostNetworkAdapter], HostCommunication

    + FullyQualifiedErrorId : Client20_VirtualNetworkServiceImpl_AddVMHostNetworkAdapter_VIError,VMware.VimAutomation.ViCore.

   Cmdlets.Commands.Host.SetVMHostNetworkAdapter

The log on VCSA say it is a throwable.proxy cause. 

What is so puzzling about this is that the same code work when I run this on the physical ESX host, but yet, it will not work within my Nested ESX host?  Any suggestion or feedback would be helpful

[Lalegre]

Hey dwchan​

If this is a Nested ESXi and you did not enabled MAC Learning on the VDS then probably the issue you are facing is that you do not have Promiscuous Mode enabled in the PortGroup where the Nested ESXi has their interfaces connected.

If this still do not resolve your issue, could you please give a brief description about the Portgroups, VLAN configuration, Security Policies and vNICs configured on your Nested ESXi?

[a_p_]

Discussion moved from VMware vSphere™ to VMware PowerCLI

[LucD]

It looks you are moving the VMKernel (vmk0) on which the Management is running.

The result is that you loose connection for the ESXi node, and the system is smart enough to roll back your change.

Is there another VMKernel on which you map Management temporarily?

[dwchan]

Given the question/problem I am having are between vSphere network and PowerCLI, not sure which forum it belong.  However, given I can't even do this through the GUI without an error, I may want to keep this on the vSphere forum first if that is ok

[a_p_]

No problem at all, I just moved it back to VMware vSphere™!

From your question it looked like it's a PowerCLI specific question.

André

[dwchan]

Will try to provide as some info on here, without getting too messy.   I have a single physical ESX 6.7U3 host,(2 more with a total of 3 when I get all the bugs out)  which by default started out with the default vSwitch0 (mgmt and VM portgroup, using vmnic0, with 4 more free).  I have a script (trim down version of William Lam work), where it will perform the following task.  Keep in mind not all settings are necessary best practice yet, but I just want to update the code and get the framework/plumbing in place.

Before I kick off the script, MacLearn feature, new to ESX 6.7 was not enabled

1. Deploy VCSA (the physical one)

2. Cluster DC/cluster - Add ESX host (physical host) to the cluster

3. Create VDS, enable Network IO control, and create three vds PortGroups

     Management Network - no VLAN tagging, security (Promiscuous, MAC address changes, Forget transmits  -  all default setting of Reject)

     Trunk Network - VLAN trunk (0-4094) , security (Promiscuous, MAC address changes, Forget transmits  -  all default setting of Reject)

     VM Network - no VLAN tagging, security (Promiscuous, MAC address changes, Forget transmits  -  all default setting of Reject)

     * May use VM Network maybe use at a later time

4. Add ESX host (physical host) to VDS

5. Add vmnic1 as dvsUpLink2 to VDS

6. Migrate VMkernel vmk0 on vSwitch to vds

7. Migrate any Virtual Machine from vSwitch (using the VM Network portgroup) to vds - this only applies here as I deploy the VCSA on the first host

     - if VM exist, change the VM network adapter to the right portgroup

8. Remove legacy VSS - vSwitch0

9. Reclaim vmnic0 on host and add to vds - by default, vmnic0 added in as dvsUpLink1

10. Enable vMotion on ESXi host

11. Configure vds uplink policy - dvsUpLink1 active while dvsUpLink2 as standby for all three portgroups

Here is the code I use for my physical environment without issue. I will provide my nested code in my next reply

    if($DeployVDS -eq 1) {

        # Create VDS

        My-Logger "Creating VDS $VDSName ..."

        $vds = New-VDSwitch -Server $vc -Name $VDSName -Location (Get-Datacenter -Name $NewVCDatacenterName) -LinkDiscoveryProtocol LLDP -LinkDiscoveryProtocolOperation Listen -MaxPorts 128 -Version 6.6.0 -MTU 9000 -NumUplinkPorts 2

        My-Logger "Enable Network IO Control on VDS $VDSName ..."

        (Get-VDSwitch $VDSName | Get-View).EnableNetworkResourceManagement($true)

        # Create DVPortgroup

        My-Logger "Creating new DVPortgroup $VLAMGMTPortgroup ..."

        New-VDPortgroup -Server $vc -Name $VLAMGMTPortgroup -Vds $vds -NumPorts 24 -PortBinding Ephemeral

        My-Logger "Creating new DVPortgroup $VLANVMPortgroup ..."

        New-VDPortgroup -Server $vc -Name $VLANVMPortgroup -Vds $vds -NumPorts 24 -PortBinding Static

        My-Logger "Creating new DVPortgroup $VLANTrunkPortgroup ..."

        New-VDPortgroup -Server $vc -Name $VLANTrunkPortgroup -Vds $vds -NumPorts 24 -VlanTrunkRange 0-4094 -PortBinding Static

        # Add ESXi host to VDS

        $vmhosts = Get-Cluster -Server $vc -Name $NewVCVSANClusterName | Get-VMHost

        foreach ($vmhost in $vmhosts) {

            $vmhostname = $vmhost.name

            # Add ESXi host to VDS

            My-Logger "Adding $vmhostname to VDS $VDSName..."

            Add-VDSwitchVMHost -Server $vc -VDSwitch $vds -VMHost $vmhost | Out-File -Append -LiteralPath $verboseLogFile

            # Adding Physical NIC (vmnic1) to VDS

            My-Logger "Adding vmnic1 to VDS $VDSName..."

            # https://vbombarded.wordpress.com/2015/01/29/migrate-esxi-host-physical-adapters-to-specific-dvuplink...

            $uplinks = $vmhost | Get-VDSwitch | Get-VDPort -Uplink | where {$_.ProxyHost -like $vmhost.Name}

            $config = New-Object VMware.Vim.HostNetworkConfig

            $config.proxySwitch = New-Object VMware.Vim.HostProxySwitchConfig[] (1)

            $config.proxySwitch[0] = New-Object VMware.Vim.HostProxySwitchConfig

            $config.proxySwitch[0].changeOperation = "edit"

            $config.proxySwitch[0].uuid = $vds.Key

            $config.proxySwitch[0].spec = New-Object VMware.Vim.HostProxySwitchSpec

            $config.proxySwitch[0].spec.backing = New-Object VMware.Vim.DistributedVirtualSwitchHostMemberPnicBacking

            $config.proxySwitch[0].spec.backing.pnicSpec = New-Object VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec[] (2)

            $config.proxySwitch[0].spec.backing.pnicSpec[0] = New-Object VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec

            $config.proxySwitch[0].spec.backing.pnicSpec[0].pnicDevice = "vmnic1"

            $config.proxySwitch[0].spec.backing.pnicSpec[0].uplinkPortKey = ($uplinks | where {$_.Name -eq "dvUplink2"}).key

            $_this = Get-View (Get-View $vmhost).ConfigManager.NetworkSystem

            $_this.UpdateNetworkConfig($config, "modify")

            # Migrating VMkernel port (vmk0) on vSwitch to VDS

            My-Logger "Migrate $vmhostname VMkernel interfaces to VDS $VDSName..."

            $vmk = Get-VMHostNetworkAdapter -Name vmk0 -VMHost $vmhost

            Set-VMHostNetworkAdapter -PortGroup $VLAMGMTPortgroup -VirtualNic $vmk -confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            # Migrating Virtual Machines from vSwitch to VDS if Virtual Machines exist on Host

            My-Logger "Migrate Virtual Machine from VSS to VDS $VDSName if VM exist..."

            My-Logger "Check to see if VM exist on host $vmhostname..."

            $VMonHost = (Get-VM).count

            if($VMonHost -gt 0) {

                My-Logger "$VMonHost Virtual Machine on host $vmhostname will be migrated to DVS..."

                $vdPortgroup = Get-VDPortGroup -VDSwitch (Get-VDSwitch -Name $VDSName) -Name $VLANVMPortgroup

                Get-VM -Location $vmhostname | Get-NetworkAdapter | where { $_.NetworkName -eq $VMNetwork } | Set-NetworkAdapter -Portgroup $vdPortgroup -confirm:$false #did not work as it detect 2 entries

            }

            My-Logger "Removing legacy Standard Switch - vSwitch0 on $vmhostnam..."

            Remove-VirtualSwitch -VirtualSwitch vSwitch0 -Confirm:$false

            # Reclaim vmnic0 on host and add to VDS

            My-Logger "Reclaim vmnic0 on host and add to VDS $VDSName..."

            $pNIC_vSS = $vmhost | Get-VMHostNetworkAdapter -Physical -Name vmnic0

            Add-VDSwitchPhysicalNetworkAdapter -Server $vc -DistributedSwitch $vds -VMHostPhysicalNic $pNIC_vSS -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

        }

    }

[dwchan]

Thank you, it may switch back over to PowerCLI, but for now, trying to understand what I am missing because it is in a nested environment.  My gut is telling me it is some a configuration / setting issue versus coding.  Just can't put my finger on it

[dwchan]

In respond to your comment "If this is a Nested ESXi and you did not enabled MAC Learning on the VDS then probably the issue you are facing is that you do not have Promiscuous Mode enabled in the PortGroup where the Nested ESXi has their interfaces connected."

My nested ESXI hosts are using a vds portgroup call "Trunk Network" on my physical ESX host / VCSA.  It is set as VLAN Trunk but with default reject setting for all of its security settings and MacLearn (new feature on 6.7) has not been enabled.  Are you implying that

1. I have to enable promiscuous mode on the Portgroup that the nested ESX VMs are using?  So in my case, "Trunk Network" portgroup on my physical ESX host.

or

2. Do I just need to enable the MacLearn filter on that vds portgroup on the physical ESX level?  William Lam did a PowerCLI for that I can use.  I know right now is set to default which is not enable

or

3. I need to enable the MacLearn filter AND enable promiscuous mode?

[dwchan]

As a continuation to my earlier reply, here are some information to my nested ESX environment

I deployed my nested ESX hosts (3 of them) and a separate VCSA to manage the nested ESX on the same box as my physical ESX host.  That was a mouthful

So on the physical ESX host (ESX01), I have a total of 5 VM

     VCSA100 - physical VCSA for my physical ESX

     ESX101 - nested ESX host #1 w/ 4 vmnic

     ESX102 - nested ESX host #2 w/ 4 vmnic

     ESX103 - nested ESX host #3 w/ 4 vmnic

     VCSA10 - physical VCSA for my nested ESX

I deployed the three nested ESX VM without issue.  I did modify it a little from William Lam original script to match my use case as I am using both HDD and SDD for my vsan and I will be using 4 vmnic versus his with just 2

Here is a quick snippet to the code

        $NestedESXiHostnameToIPs.GetEnumerator() | Sort-Object -Property Value | Foreach-Object {

            $VMName = $_.Key

            $VMIPAddress = $_.Value

            $ovfconfig = Get-OvfConfiguration $NestedESXiApplianceOVA

            $networkMapLabel = ($ovfconfig.ToHashTable().keys | where {$_ -Match "NetworkMapping"}).replace("NetworkMapping.","").replace("-","_").replace(" ","_")

            $ovfconfig.NetworkMapping.$networkMapLabel.value = $VMNetwork

            $ovfconfig.common.guestinfo.hostname.value = $VMName

            $ovfconfig.common.guestinfo.ipaddress.value = $VMIPAddress

            $ovfconfig.common.guestinfo.netmask.value = $VMNetmask

            $ovfconfig.common.guestinfo.gateway.value = $VMGateway

            $ovfconfig.common.guestinfo.dns.value = $VMDNS

            $ovfconfig.common.guestinfo.domain.value = $VMDomain

            $ovfconfig.common.guestinfo.ntp.value = $VMNTP

            $ovfconfig.common.guestinfo.syslog.value = $VMSyslog

            $ovfconfig.common.guestinfo.password.value = $VMPassword

            if($VMSSH -eq "true") {

                $VMSSHVar = $true

            } else {

                $VMSSHVar = $false

            }

            $ovfconfig.common.guestinfo.ssh.value = $VMSSHVar

            My-Logger "Deploying Nested ESXi VM $VMName ..."

            if($DeploymentTarget -eq "VMC") {

                $vm = Import-VApp -Source $NestedESXiApplianceOVA -OvfConfiguration $ovfconfig -Name $VMName -Location $resourcePool -VMHost $vmhost -Datastore $datastore -DiskStorageFormat thin -InventoryLocation $folder

            } else {

                $vm = Import-VApp -Source $NestedESXiApplianceOVA -OvfConfiguration $ovfconfig -Name $VMName -Location $cluster -VMHost $vmhost -Datastore $datastore -DiskStorageFormat thin

            }

            My-Logger "Updating vCPU Count to $NestedESXivCPU & vMEM to $NestedESXivMEM GB ..."

            Set-VM -Server $viConnection -VM $vm -NumCpu $NestedESXivCPU -MemoryGB $NestedESXivMEM -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            My-Logger "Updating vSAN Caching VMDK size to $NestedESXiCachingvDisk GB ..."

            Get-HardDisk -Server $viConnection -VM $vm -Name "Hard disk 2" | Remove-HardDisk -DeletePermanently -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            New-HardDisk -Server $viConnection -VM $vm -Datastore "SSD_VSAN" -CapacityGB $NestedESXiCachingvDisk -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            My-Logger "Updating vSAN Capacity VMDK size to $NestedESXiCapacityvDisk GB ..."

            Get-HardDisk -Server $viConnection -VM $vm -Name "Hard disk 2" | Remove-HardDisk -DeletePermanently -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            New-HardDisk -Server $viConnection -VM $vm -Datastore "HDD_VSAN" -CapacityGB $NestedESXiCapacityvDisk -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            My-Logger "Updating with 2 additional NIC  (vmnic2 and vmnic3) ..."

            New-NetworkAdapter -Server $viConnection -VM $vm -Portgroup $VMNetwork -StartConnected -Type Vmxnet3 -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            New-NetworkAdapter -Server $viConnection -VM $vm -Portgroup $VMNetwork -StartConnected -Type Vmxnet3 -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile

            $orignalExtraConfig = $vm.ExtensionData.Config.ExtraConfig

            $a = New-Object VMware.Vim.OptionValue

            $a.key = "ethernet2.filter4.name"

            $a.value = "dvfilter-maclearn"

            $b = New-Object VMware.Vim.OptionValue

            $b.key = "ethernet2.filter4.onFailure"

            $b.value = "failOpen"

            $c = New-Object VMware.Vim.OptionValue

            $c.key = "ethernet3.filter4.name"

            $c.value = "dvfilter-maclearn"

            $d = New-Object VMware.Vim.OptionValue

            $d.key = "ethernet3.filter4.onFailure"

            $d.value = "failOpen"

            $orignalExtraConfig+=$a

            $orignalExtraConfig+=$b

            $orignalExtraConfig+=$c

            $orignalExtraConfig+=$d

            $spec = New-Object VMware.Vim.VirtualMachineConfigSpec

            $spec.ExtraConfig = $orignalExtraConfig

            My-Logger "Adding guestinfo customization properties to $vmname ..."

            $task = $vm.ExtensionData.ReconfigVM_Task($spec)

            $task1 = Get-Task -Id ("Task-$($task.value)")

            $task1 | Wait-Task | Out-Null

            My-Logger "Powering On $vmname ..."

            $vm | Start-Vm -RunAsync | Out-Null

        }

I notice William Lam code and within his OVA that he adds this to the VM

"ethernet1.filter4.name"  = "dvfilter-maclearn"

So somewhat follow suit and added those same parameters to my ethernet2 and 3 (vmnic2 and vmnic3).  I am not sure if it is needed as I am trying to get a better understanding of this maclearn filter.  given I am using vmnic2 or vmnic3 in any of my code, so I do not foresee this being my problem, yet

Afterward, I have a separate script, but basically the same MO to deployment and confirm my vds

1. Deploy VCSA (another physical one for my nested ESX hosts)

2. Create DC/cluster - Add nested ESX host (ESX101, 102, 103) to the cluster

3. Create VDS, enable Network IO control, and create three vds PortGroups

     Management Network - no VLAN tagging, security (Promiscuous, MAC address changes, Forget transmits  -  all default setting of Reject)

     Trunk Network - VLAN trunk (0-4094) , security (Promiscuous, MAC address changes, Forget transmits  -  all default setting of Reject)

     VM Network - no VLAN tagging, security (Promiscuous, MAC address changes, Forget transmits  -  all default setting of Reject)

     * May use VM Network maybe use at a later time

4. Add nested ESX host to VDS

5. Add vmnic1 as dvsUpLink2 to VDS

6. Migrate VMkernel vmk0 on vSwitch to vds

This is where everything start to fall apart

Here are some additional info

1. I try both changing the Management Network on my nested vds VLAN from no vlan to vlan 10 and to Trunk, didn't make a difference in term of PowerCLI

2. I am trying to repeat this, but for what ever reason, I can migrate the vmk0 (on vss0) over to my vds "Trunk Management" portgroup with the GUI if I move both the NIC and Network at the same time. And I think I can do the same with the dvs portgroup "Management Network" if I change it at the GUI from no vlan to trunk after the failure and move both network and nic at the same time.

In short, I am lost and looking for ideas and hopefully root cause with my problem

[TacoSauce]

Hi dwchan,

We experience the same in our nested test environment. Did you succeeded to find a solution?

[scott28tt]

@dwchan 

Moderator: Moved to Nested Virtualization Discussions