Lost connection to all Server VMs on a ESXi host

[nevilleonline]

Hi All,

We recently migrated our VMS from a ESXi Cluster 6.0 to EsXi 6.5. Everything went smoothly for 55 days till one host suddenly went mad and we lost access to all VMs on that host. Rebooted the host after migrating VMs from that host to fix the issue. The next morning same thing happens on another host. Finally rebooted all host so that we can breath. The issue did not return on that day. Found production and DMZ network in Active/Standby. Whereas its a norm to make this active/active. Could this be a problem? We are using Fujitsu RX 2500 M4 servers with fujitsu custom image ESXi 6.5 U2. Problem on ESXi host 06 occurred between 3 pm to 4 pm on 19th Sep 2018 and we first noticed it early morning at  6 am 20th Sept 2018 on ESxi host 05.

Below is the vmkernal.log on both the servers during time of disaster

On Host05

2018-09-20T01:50:39.837Z cpu2:5560221)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T01:53:25.422Z cpu31:2042457)WARNING: NetPort: 1934: failed to disable port 0x200002b on vSwitch0: Busy

2018-09-20T01:54:20.843Z cpu0:5560221)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T01:56:28.426Z cpu2:70620)WARNING: NetPort: 1934: failed to disable port 0x200000a on vSwitch0: Busy

2018-09-20T01:57:20.858Z cpu5:5560223)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T01:58:06.836Z cpu10:5560220)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T02:00:56.862Z cpu4:3128662)WARNING: NetPort: 1934: failed to disable port 0x200003a on vSwitch0: Busy

2018-09-20T02:02:18.379Z cpu14:70620)WARNING: NetPort: 1934: failed to disable port 0x200000a on vSwitch0: Busy

2018-09-20T02:03:27.814Z cpu1:5560220)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T02:04:22.813Z cpu6:5560222)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T02:05:25.772Z cpu3:5559297)WARNING: NetPort: 1934: failed to disable port 0x2000040 on vSwitch0: Busy

2018-09-20T02:06:13.813Z cpu8:5560222)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T02:07:54.818Z cpu14:5560222)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T02:09:28.392Z cpu25:70620)WARNING: NetPort: 1934: failed to disable port 0x200000a on vSwitch0: Busy

2018-09-20T02:10:00.417Z cpu17:2042457)WARNING: NetPort: 1934: failed to disable port 0x200002b on vSwitch0: Busy

2018-09-20T02:11:35.810Z cpu10:5560222)WARNING: NetPort: 1934: failed to disable port 0x2000042 on vSwitch0: Busy

2018-09-20T02:13:03.372Z cpu18:70620)WARNING: NetPort: 1934: failed to disable port 0x200000a on vSwitch0: Busy

2018-09-20T02:22:41.362Z cpu16:5558469)WARNING: NetPort: 1934: failed to disable port 0x200003e on vSwitch0: Busy

2018-09-20T02:26:50.051Z cpu1:2758152)WARNING: NetPort: 1934: failed to disable port 0x2000038 on vSwitch0: Busy

2018-09-20T04:14:18.453Z cpu21:5558769)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

On Host06

2018-09-19T14:15:20.852Z cpu21:2850426)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:15:50.862Z cpu24:2850427)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:16:06.814Z cpu26:2850427)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:16:26.895Z cpu24:2850422)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:16:46.895Z cpu26:2850425)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:17:01.871Z cpu18:2850425)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:17:21.863Z cpu30:2850427)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:17:41.893Z cpu23:2850425)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:17:56.888Z cpu30:2850425)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:18:16.888Z cpu24:2850419)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:18:36.906Z cpu28:2850419)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:18:42.495Z cpu21:2850419)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:19:08.883Z cpu30:2850419)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:19:23.890Z cpu29:2850423)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:19:39.887Z cpu22:2850423)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:20:00.914Z cpu20:2850421)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:20:06.041Z cpu31:2850421)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:20:26.833Z cpu20:2850422)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:20:46.856Z cpu22:2850419)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:21:06.884Z cpu27:2850419)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:21:26.896Z cpu23:2850423)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:21:46.903Z cpu20:2850421)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:22:06.906Z cpu28:2850422)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:22:32.880Z cpu22:2850422)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:22:53.875Z cpu18:2850422)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

2018-09-19T14:23:18.873Z cpu16:2850427)WARNING: NetPort: 1934: failed to disable port 0x200003f on vSwitch0: Busy

Could this be a network driver issue? Any help would be great. I am also attaching my vmkernel log from both the host.

Regards,

Neville

[SureshKumarMuth]

Similar issue reported earlier, can you check if the following post is applicable for you and try the workaround step

Vsphere ESXi 6.5 external network connectivity lost on DL380Gen8