- VMware Technology Network
- :
- Cloud & SDDC
- :
- vSAN
- :
- VMware vSAN Discussions
- :
- [BUG] a tale of vSAN health check : vSAN: Basic (u...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
[BUG] a tale of vSAN health check : vSAN: Basic (unicast) connectivity check go wrong
Hi,
I would like to share my story, which is kind a failed logic in the vSAN health logic.
A customer contacted due to his 2 nodes vSAN started to report connectivity problems with the witness hosted on a external site. After the usual checks, everything looked operational
vmkping didn't report any lost packets until I started to use the new diag aka "esxcli network diag ping"
esxcli network diag ping -H X.X.1.46 -I vmk0
Trace:
Received Bytes: 64
Host: X.X.1.46
ICMP Seq: 0
TTL: 59
Round-trip Time: 16540 us
Dup: false
Detail:
Received Bytes: 155
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 191
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 289
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 282
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 261
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 283
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 155
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 191
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 228
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 284
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 207
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 228
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 284
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 180
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 268
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 268
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 253
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 64
Host: X.X.1.46
ICMP Seq: 1
TTL: 59
Round-trip Time: 17301 us
Dup: false
Detail:
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 240
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 240
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 240
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 134
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 556
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 240
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 370
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 370
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 370
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 370
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 344
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 215
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 284
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 199
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 199
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 283
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 284
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 212
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 284
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 250
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 212
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 283
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 274
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 284
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 419
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 198
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 281
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 281
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 281
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 281
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 281
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 187
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 213
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 195
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 159
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 346
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 351
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 199
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 197
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 210
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 342
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 347
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 344
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 345
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 342
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 345
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 223
Host: X.X.40.107
ICMP Seq: -1
TTL: 128
Round-trip Time: -1 us
Dup: false
Detail: Destination port unreachable
Received Bytes: 64
Host: X.X.1.46
ICMP Seq: 2
TTL: 59
Round-trip Time: 15400 us
Dup: false
Detail:
Summary:
Host Addr: X.X.1.46
Transmitted: 3
Received: 3
Duplicated: 0
Packet Lost: 0
Round-trip Min: 15400 us
Round-trip Avg: 16413 us
Round-trip Max: 17301 us
X.X.1.46 is my Witness tagged vmk but why the X.X.40.107, a local machine on the network was responding to this test.
Further investigation with tcpdump-uw icmp showed us both ESX host were constantly pinging this X.X.40.107 address!
We decided to focus on all traffic to this destination adress and we found both esx were trying to forward syslog traffic to this IP
The customer told us the syslog the address changed but the configuration change was not reflected on each ESX.
Once the syslog configuration was adapted, esxcli network diag ping and vSAN health check came back to a green status
Oli