VMs on Different ESXi Hosts Can't Communicate With Each Other

[Viljanen]

Hello,

I need little a bit help. We have six ESX Hosts 6.0 and there are about 250 vms running.These hosts are connected with two (Dell M100E) 10G netcard to the two Nexus, one for production, vmotion and manage and second one for SCSI. Sometimes when we migrate one vm to another host, it loose connection to another vm's (it´s vary). Is it possible that we have somehow wrong configuration in our Nexus or what you can suggest to do next. This problem has started 2 weeks aqo.

Thanks in advanced for any help...

NEXUS 1

ip domain-lookup
logging event link-status default
service unsupported-transceiver
class-map type qos class-fcoe
class-map type queuing class-fcoe
  match qos-group 1
class-map type queuing class-all-flood
  match qos-group 2
class-map type queuing class-ip-multicast
  match qos-group 2
class-map type network-qos class-fcoe
  match qos-group 1
class-map type network-qos class-all-flood
  match qos-group 2
class-map type network-qos class-ip-multicast
  match qos-group 2
policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216
    multicast-optimize
system qos
  service-policy type network-qos jumbo
policy-map type control-plane copp-system-policy-customized
  class copp-system-class-default
    police cir 2048 kbps bc 6400000 bytes
slot 1

vrf context management
spanning-tree mode mst
spanning-tree mst configuration
  name kerava
  revision 1
  instance 1 vlan 1-1005
vpc domain 10
  role priority 8192
  peer-keepalive destination 10.10.100.231 source 10.10.100.230
  peer-gateway
  auto-recovery
  ip arp synchronize
port-profile default max-ports 512

interface Vlan1

interface Vlan200
  no shutdown
  no ip redirects
  ip address 192.168.1.230/24

interface port-channel10
  description Trunk-C6513
  shutdown
  switchport mode trunk
  switchport trunk allowed vlan 1-240
  spanning-tree bpdufilter enable
  vpc 10

interface port-channel110
  description VCP_PEER
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface Ethernet1/1
  description **MASTER1**
  switchport mode trunk
  spanning-tree bpdufilter enable
  speed 1000

interface Ethernet1/2
  description Trunk-C6513
  switchport mode trunk
  spanning-tree bpdufilter enable

interface Ethernet1/3
  description PKESXHOST9_A1
  switchport mode trunk
  spanning-tree port type edge trunk

interface Ethernet1/4
  description PKESXHOST10_A1
  switchport mode trunk
  spanning-tree port type edge trunk

interface Ethernet1/5
  description PKBACKUP02, iscsi
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/6
  description PKBACKUP02, tuotanto
  switchport access vlan 240
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/7
  description PKESXHOST11_A1
  switchport mode trunk
  spanning-tree port type edge trunk

interface Ethernet1/8
  description PKESXHOST12_A1
  switchport mode trunk
  spanning-tree port type edge trunk

interface Ethernet1/9
  description PKFACTAAP01_A1, tuotanto
  switchport access vlan 240
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/10
  description PKFACTAAP01_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/11
  description PKFACTADB01_A1, tuotanto
  switchport access vlan 240
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/12
  description PKFACTADB01_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/13
  description DR4000_A1, tuotanto
  switchport access vlan 240
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/14
  description COMPELLENT
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/15
  description COMPELLENT
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/16
  description VCP_PEER
  switchport mode trunk
  channel-group 110

interface Ethernet1/17
  description EQL1
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/18
  description EQL2
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/19
  description EQL3
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/20

interface Ethernet1/21
  description PKESXHOST12_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/22

interface Ethernet1/23
  description PKESXHOST10_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/24
  description PKESXHOST9_B1_iSCSI, po24
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/25
  description PKESXHOST11_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/26
  description PKESXHOST5_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/27
  description PKBACKUP01_B1_Tuotanto
  switchport access vlan 240
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/28
  description PKESXHOST8_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/29
  description PKESXHOST7_B1_iSCSI
  switchport access vlan 900
  spanning-tree port type edge
  flowcontrol receive on
  flowcontrol send on

interface Ethernet1/30
  description PKESXHOST8_A1
  switchport mode trunk
  spanning-tree port type edge trunk

interface Ethernet1/31
  description PKESXHOST7_A1
  switchport mode trunk
  spanning-tree port type edge trunk

interface Ethernet1/32
  description VCP_PEER
  switchport mode trunk
  channel-group 110

interface mgmt0
  ip address 10.10.100.230/24
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.1.bin
boot system bootflash:/n5000-uk9.5.2.1.N1.1.bin
ip route 0.0.0.0/0 192.168.1.1

[bayupw]

Could you share your vSwitch/Distributed switch setup such as how many vSwitch, how many uplinks, what teaming policy are you using for the PortGroup ?

[Viljanen]

We use standard vSwitchs for production, vmotion and management where is the problem. iSCSI is okay (DvSwitch).If you need more information let me know.You shoud also know that Dell's 1000e Blade's netcard 10G is sharing among these three by netcard software (firm)..

[Viljanen]

[bayupw]

Sorry for late reply.

When you mentioned VMs can't communicate after vMotion.

Is it permanent or it will work after a few minutes?

Could you check if both vmnic6 & vmnic7 are being used by the VMs?

You can verify with esxtop, see this blog post Which NIC is my VM using? Load Balancing Visibility with vSphere - VMware vSphere Blog

Try to isolate the issue, is it only happening on specific ESXi hosts or maybe on specific vmnic (e.g. only when VM is using vmnic7).

Let us know how it goes