This is going to be long, so suggestions if any, what I can try further to fix this fault.
This is a homelab, so it's not production, and does not use certified hardware for ESXi or vSAN.
I've got a three node ESXi - Build VMware ESXi, 7.0.3, 21930508, 128GB RAM - ASRock Rack - Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz.
ESXi is installed to an NVM disk Samsung NVMe 970 1TB using a PCIe card.
2 x D1541D4U-2O8R
1 x D1541D4U-2T8R
When this was first built, using much earlier versions of ESXi and vSAN, this worked flawlessly and stable (no changes were made) with no issues, and then one day, virtual machines were disappearing in front of my eyes and status changed to Inaccessible. This was the first time in using a VMware product I've ever lost virtual machines, so machines were restored to NFS datastores and vSAN was abandoned.
The three nodes are almost identical, two nodes use 2xSamsung NVMe 970 1TB as Cache Tier, and 8x256GB Samsung 870 SSD. (via SAS 3008 controller), the third node uses 2xSamsung NVMe 970 1TB as Cache Tier, and 2x2TB Enterprise Intel SSDs.
Networking, two nodes use SFP+ Fibre/DAC and third node uses SFP+ RJ45 converter, as the motherboard is different model, just networking change.
After trial and error, the issue looked like it's the D1541D4U-2T8R node, what is odd is that using the Proactive Network Test which when first built the networking results passed, I noticed that throughput was very poor, I use a Mirotik CRS305-1G-4S+IN just for vSAN networking, which is jumbo frame enabled VSS.
So I replaced the SFP+RJ45 to a Mirotik module, rather than Ubiquiti, the same issue, changed the cable to CAT8, same issue. So decided to make some changes and add a new network interface Intel DA-520, on the HCL for 7.0 and 8.0.
Performance is now > 9,600 - so I thought this would fix the issue.
No, same issue.
Create a new virtual machine on this host, which works fine, install and OS, Power off the virtual machine, and it takes a while, and then the VM disappears as inaccessible.
I've also used the Virtual Machine create proactive test, and oddly this passes, but the node with the issue takes 5 minutes to delete the VM, and then it becomes inaccessible.
So today I removed all DGs, and re-created using NVMe (Cache and Capacity), SSD connected to SAS (Cache and Capacity), SSD connected to SATA
None of these changes make any different.
My only options left are to all hosts have been updated to latest ESXi build has not made any difference.
0.Update vCenter Server to latest build
1. Trash vSAN on all nodes - rebuild.
2. Trash this host - rebuild.
It is possible that I have a faulty host, although all VMs function correctly using NFS, iSCSI, and Local NVMe, SSD datastores.
It's only of a VM touches the faulty node, that issues arise.
I could just reduce to two nodes, and use this host as a witness, just is a waste of local flash storage.
Comments any ideas, I know it's VMware Explore week, ideas and comments.
Thanks
To clarify, EVERY vm that is stored on this host disappears?
What are the results of:
esxcli vsan health
esxcli vsan cluster get
@einstein-a-go-g, I would advise you reproduce the issue (or if already have VMs inaccessible then current state is fine), share the details of any red health checks in Cluster > Monitor > vSAN > Skyline Health, attach a copy of vmkernel.log from the suspected problematic node and share the output of the following command run from any node:
# cmmds-tool find -f python | grep CONFIG_STATUS -B 4 -A 6 | grep 'uuid\|content' | grep -o 'state\\\":\ [0-9]*' | sort | uniq -c
the VM remains ON and working, until the VM is Powered OFF, there is a delay, and the Inventory changes to Inaccessible. for all VMs hosted on this host which may have an issue.
Today, I have a Red Alert which is unusual with Performance.
[root@esxi010:~] esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2023-08-29T09:23:14Z
Local Node UUID: 61e8797b-f7de-b629-b9d7-d05099c30c65
Local Node Type: NORMAL
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 61e8797b-f7de-b629-b9d7-d05099c30c65
Sub-Cluster Backup UUID: 61ad5536-ee89-59b7-274d-d05099c1a420
Sub-Cluster UUID: 52a55e28-13ae-1d14-e1db-77d918e8e5d7
Sub-Cluster Membership Entry Revision: 2
Sub-Cluster Member Count: 3
Sub-Cluster Member UUIDs: 61e8797b-f7de-b629-b9d7-d05099c30c65, 61ad5536-ee89-59b7-274d-d05099c1a420, 6196d936-96b8-4744-3a16-00113238c190
Sub-Cluster Member HostNames: esxi010.cyrus-consultants.co.uk, esxi008.cyrus-consultants.co.uk, esxi006.cyrus-consultants.co.uk
Sub-Cluster Membership UUID: 4dddea64-652c-4471-e11d-a0369f1f3848
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: 6766d3e8-d88c-4c8f-b65a-1219f7d38814 37 2023-08-29T09:17:04.195
Mode: REGULAR
[root@esxi010:~] esxcli vsan health cluster list
Health Test Name Status
-------------------------------------------------- ------
Overall health red (vSAN performance service issue)
Performance service red
Stats DB object green
Stats primary election red
Network green
Hosts with connectivity issues green
vSAN cluster partition green
All hosts have a vSAN vmknic configured green
vSAN: Basic (unicast) connectivity check green
vSAN: MTU check (ping with large packet size) green
vMotion: Basic (unicast) connectivity check green
vMotion: MTU check (ping with large packet size) green
Network latency check green
Physical disk green
Operation health green
Disk capacity green
Congestion green
Component limit health green
Component metadata health green
Memory pools (heaps) green
Memory pools (slabs) green
Data green
vSAN object health green
vSAN object format health green
Cluster green
Advanced vSAN configuration in sync green
vSAN daemon liveness green
vSAN Disk Balance green
Resync operations throttling green
Software version compatibility green
Disk format version green
Capacity utilization green
Storage space green
Read cache reservations green
Component green
What if the most consumed host fails green
[root@esxi010:~] /etc/init.d/vsanmgmtd restart
Terminating watchdog process with PID 2098640
I've tried restarting vsanmgmtd on all hosts, and also enable/disable performance for cluster, and error is still present.
on power off, error also emailed this is usually for the weird host with issues, HA has kicked in here and tried to power up on another host.
This email is to notify you that an alarm has been triggered in your vCenter:
Alarm alarm.HostErrorAlarm on Host esxi008.cyrus-consultants.co.uk
because Issue detected on esxi008.cyrus-consultants.co.uk in Cyrus HQ: BC: 3177: File New Virtual Machine.nvram closed with dirty buffers. Possible data loss. (2023-08-29T09:35:53.874Z cpu15:3559419).
3 state\": 39
139 state\": 47
9 state\": 7
@einstein-a-go-g, Most of the data here is in an unhealthy and async-stale state (state 39 and state 47) - this could be due to network communication issues between the nodes or something else e.g. a disk issue - I would lean towards the former due to 'RPC to DOM op readPolicy returned: No connection' being logged.
Can you run the following from problem node and non-problem node(change the vmk2 to whatever vmk has vSAN-traffic enabled on it, determine this via 'esxcli vsan network list', change the MTU (8972 below) to '1472' if using 1500 MTU on that vmk, leave as-is if using 9000 MTU) and share the output?:
# for i in `localcli vsan cluster unicastagent list | grep true | awk '{ print $4}'`; do echo "pinging $i 100 times"; echo; vmkping -I vmk2 $i -s 8972 -d -c 100 -i 0.1; echo; echo "**************************"; done
Can you confirm the following are set to 0 on all nodes:
# vsish -e get /config/VSAN/intOpts/DOMPauseAllCCPs
# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates
What does Skyline Health in vCenter UI say? Asking as 'esxcli vsan health cluster list' might be misleading if some nodes are not reporting to those queries.
The 2 red alerts relating to Perf service and Stats primary could also be due to communication issues between the nodes.
Thanks for your reply, family has taken me away at present, but I will be back to try and fix this issue, because I would like to understand why it broke so badly.
edit. Managed to get some time!
[root@esxi006:~] for i in `localcli vsan cluster unicastagent list | grep true | awk '{ print $4}'`; do echo "pinging $i 100 times"; echo; vmkping -
I vmk1 $i -s 8972 -d -c 100 -i 0.1; echo; echo "**************************"; done
pinging 10.10.1.2 100 times
PING 10.10.1.2 (10.10.1.2): 8972 data bytes
8980 bytes from 10.10.1.2: icmp_seq=0 ttl=64 time=0.238 ms
8980 bytes from 10.10.1.2: icmp_seq=1 ttl=64 time=0.211 ms
8980 bytes from 10.10.1.2: icmp_seq=2 ttl=64 time=0.272 ms
8980 bytes from 10.10.1.2: icmp_seq=3 ttl=64 time=0.303 ms
8980 bytes from 10.10.1.2: icmp_seq=4 ttl=64 time=0.171 ms
8980 bytes from 10.10.1.2: icmp_seq=5 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.2: icmp_seq=6 ttl=64 time=0.218 ms
8980 bytes from 10.10.1.2: icmp_seq=7 ttl=64 time=0.189 ms
8980 bytes from 10.10.1.2: icmp_seq=8 ttl=64 time=0.302 ms
8980 bytes from 10.10.1.2: icmp_seq=9 ttl=64 time=0.254 ms
8980 bytes from 10.10.1.2: icmp_seq=10 ttl=64 time=0.269 ms
8980 bytes from 10.10.1.2: icmp_seq=11 ttl=64 time=0.297 ms
8980 bytes from 10.10.1.2: icmp_seq=12 ttl=64 time=0.310 ms
8980 bytes from 10.10.1.2: icmp_seq=13 ttl=64 time=0.223 ms
8980 bytes from 10.10.1.2: icmp_seq=14 ttl=64 time=0.221 ms
8980 bytes from 10.10.1.2: icmp_seq=15 ttl=64 time=0.329 ms
8980 bytes from 10.10.1.2: icmp_seq=16 ttl=64 time=0.172 ms
8980 bytes from 10.10.1.2: icmp_seq=17 ttl=64 time=0.198 ms
8980 bytes from 10.10.1.2: icmp_seq=18 ttl=64 time=0.310 ms
8980 bytes from 10.10.1.2: icmp_seq=19 ttl=64 time=0.171 ms
8980 bytes from 10.10.1.2: icmp_seq=20 ttl=64 time=0.185 ms
8980 bytes from 10.10.1.2: icmp_seq=21 ttl=64 time=0.243 ms
8980 bytes from 10.10.1.2: icmp_seq=22 ttl=64 time=0.331 ms
8980 bytes from 10.10.1.2: icmp_seq=23 ttl=64 time=0.276 ms
8980 bytes from 10.10.1.2: icmp_seq=24 ttl=64 time=0.238 ms
8980 bytes from 10.10.1.2: icmp_seq=25 ttl=64 time=0.177 ms
8980 bytes from 10.10.1.2: icmp_seq=26 ttl=64 time=0.190 ms
8980 bytes from 10.10.1.2: icmp_seq=27 ttl=64 time=0.248 ms
8980 bytes from 10.10.1.2: icmp_seq=28 ttl=64 time=0.319 ms
8980 bytes from 10.10.1.2: icmp_seq=29 ttl=64 time=0.249 ms
8980 bytes from 10.10.1.2: icmp_seq=30 ttl=64 time=0.254 ms
8980 bytes from 10.10.1.2: icmp_seq=31 ttl=64 time=0.184 ms
8980 bytes from 10.10.1.2: icmp_seq=32 ttl=64 time=0.321 ms
8980 bytes from 10.10.1.2: icmp_seq=33 ttl=64 time=0.228 ms
8980 bytes from 10.10.1.2: icmp_seq=34 ttl=64 time=0.245 ms
8980 bytes from 10.10.1.2: icmp_seq=35 ttl=64 time=0.261 ms
8980 bytes from 10.10.1.2: icmp_seq=36 ttl=64 time=0.235 ms
8980 bytes from 10.10.1.2: icmp_seq=37 ttl=64 time=0.259 ms
8980 bytes from 10.10.1.2: icmp_seq=38 ttl=64 time=0.311 ms
8980 bytes from 10.10.1.2: icmp_seq=39 ttl=64 time=0.202 ms
8980 bytes from 10.10.1.2: icmp_seq=40 ttl=64 time=0.247 ms
8980 bytes from 10.10.1.2: icmp_seq=41 ttl=64 time=0.243 ms
8980 bytes from 10.10.1.2: icmp_seq=42 ttl=64 time=0.299 ms
8980 bytes from 10.10.1.2: icmp_seq=43 ttl=64 time=0.270 ms
8980 bytes from 10.10.1.2: icmp_seq=44 ttl=64 time=0.208 ms
8980 bytes from 10.10.1.2: icmp_seq=45 ttl=64 time=0.206 ms
8980 bytes from 10.10.1.2: icmp_seq=46 ttl=64 time=0.311 ms
8980 bytes from 10.10.1.2: icmp_seq=47 ttl=64 time=0.330 ms
8980 bytes from 10.10.1.2: icmp_seq=48 ttl=64 time=0.164 ms
8980 bytes from 10.10.1.2: icmp_seq=49 ttl=64 time=0.180 ms
8980 bytes from 10.10.1.2: icmp_seq=50 ttl=64 time=0.196 ms
8980 bytes from 10.10.1.2: icmp_seq=51 ttl=64 time=0.280 ms
8980 bytes from 10.10.1.2: icmp_seq=52 ttl=64 time=0.278 ms
8980 bytes from 10.10.1.2: icmp_seq=53 ttl=64 time=0.243 ms
8980 bytes from 10.10.1.2: icmp_seq=54 ttl=64 time=0.228 ms
8980 bytes from 10.10.1.2: icmp_seq=55 ttl=64 time=0.176 ms
8980 bytes from 10.10.1.2: icmp_seq=56 ttl=64 time=0.219 ms
8980 bytes from 10.10.1.2: icmp_seq=57 ttl=64 time=0.194 ms
8980 bytes from 10.10.1.2: icmp_seq=58 ttl=64 time=0.283 ms
8980 bytes from 10.10.1.2: icmp_seq=59 ttl=64 time=0.210 ms
8980 bytes from 10.10.1.2: icmp_seq=60 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.2: icmp_seq=61 ttl=64 time=0.196 ms
8980 bytes from 10.10.1.2: icmp_seq=62 ttl=64 time=0.249 ms
8980 bytes from 10.10.1.2: icmp_seq=63 ttl=64 time=0.159 ms
8980 bytes from 10.10.1.2: icmp_seq=64 ttl=64 time=0.336 ms
8980 bytes from 10.10.1.2: icmp_seq=65 ttl=64 time=0.216 ms
8980 bytes from 10.10.1.2: icmp_seq=66 ttl=64 time=0.186 ms
8980 bytes from 10.10.1.2: icmp_seq=67 ttl=64 time=0.297 ms
8980 bytes from 10.10.1.2: icmp_seq=68 ttl=64 time=0.180 ms
8980 bytes from 10.10.1.2: icmp_seq=69 ttl=64 time=0.232 ms
8980 bytes from 10.10.1.2: icmp_seq=70 ttl=64 time=0.256 ms
8980 bytes from 10.10.1.2: icmp_seq=71 ttl=64 time=0.271 ms
8980 bytes from 10.10.1.2: icmp_seq=72 ttl=64 time=0.276 ms
8980 bytes from 10.10.1.2: icmp_seq=73 ttl=64 time=0.239 ms
8980 bytes from 10.10.1.2: icmp_seq=74 ttl=64 time=0.241 ms
8980 bytes from 10.10.1.2: icmp_seq=75 ttl=64 time=0.227 ms
8980 bytes from 10.10.1.2: icmp_seq=76 ttl=64 time=0.292 ms
8980 bytes from 10.10.1.2: icmp_seq=77 ttl=64 time=0.201 ms
8980 bytes from 10.10.1.2: icmp_seq=78 ttl=64 time=0.186 ms
8980 bytes from 10.10.1.2: icmp_seq=79 ttl=64 time=0.195 ms
8980 bytes from 10.10.1.2: icmp_seq=80 ttl=64 time=0.266 ms
8980 bytes from 10.10.1.2: icmp_seq=81 ttl=64 time=0.237 ms
8980 bytes from 10.10.1.2: icmp_seq=82 ttl=64 time=0.271 ms
8980 bytes from 10.10.1.2: icmp_seq=83 ttl=64 time=0.222 ms
8980 bytes from 10.10.1.2: icmp_seq=84 ttl=64 time=0.350 ms
8980 bytes from 10.10.1.2: icmp_seq=85 ttl=64 time=0.222 ms
8980 bytes from 10.10.1.2: icmp_seq=86 ttl=64 time=0.312 ms
8980 bytes from 10.10.1.2: icmp_seq=87 ttl=64 time=0.243 ms
8980 bytes from 10.10.1.2: icmp_seq=88 ttl=64 time=0.215 ms
8980 bytes from 10.10.1.2: icmp_seq=89 ttl=64 time=0.219 ms
8980 bytes from 10.10.1.2: icmp_seq=90 ttl=64 time=0.334 ms
8980 bytes from 10.10.1.2: icmp_seq=91 ttl=64 time=0.290 ms
8980 bytes from 10.10.1.2: icmp_seq=92 ttl=64 time=0.254 ms
8980 bytes from 10.10.1.2: icmp_seq=93 ttl=64 time=0.245 ms
8980 bytes from 10.10.1.2: icmp_seq=94 ttl=64 time=0.249 ms
8980 bytes from 10.10.1.2: icmp_seq=95 ttl=64 time=0.223 ms
8980 bytes from 10.10.1.2: icmp_seq=96 ttl=64 time=0.220 ms
8980 bytes from 10.10.1.2: icmp_seq=97 ttl=64 time=0.292 ms
8980 bytes from 10.10.1.2: icmp_seq=98 ttl=64 time=0.205 ms
8980 bytes from 10.10.1.2: icmp_seq=99 ttl=64 time=0.189 ms
--- 10.10.1.2 ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 0.159/0.244/0.350 ms
**************************
pinging 10.10.1.3 100 times
PING 10.10.1.3 (10.10.1.3): 8972 data bytes
8980 bytes from 10.10.1.3: icmp_seq=0 ttl=64 time=0.313 ms
8980 bytes from 10.10.1.3: icmp_seq=1 ttl=64 time=0.293 ms
8980 bytes from 10.10.1.3: icmp_seq=2 ttl=64 time=0.308 ms
8980 bytes from 10.10.1.3: icmp_seq=3 ttl=64 time=0.331 ms
8980 bytes from 10.10.1.3: icmp_seq=4 ttl=64 time=0.289 ms
8980 bytes from 10.10.1.3: icmp_seq=5 ttl=64 time=0.324 ms
8980 bytes from 10.10.1.3: icmp_seq=6 ttl=64 time=0.387 ms
8980 bytes from 10.10.1.3: icmp_seq=7 ttl=64 time=0.352 ms
8980 bytes from 10.10.1.3: icmp_seq=8 ttl=64 time=0.375 ms
8980 bytes from 10.10.1.3: icmp_seq=9 ttl=64 time=0.310 ms
8980 bytes from 10.10.1.3: icmp_seq=10 ttl=64 time=0.233 ms
8980 bytes from 10.10.1.3: icmp_seq=11 ttl=64 time=0.287 ms
8980 bytes from 10.10.1.3: icmp_seq=12 ttl=64 time=0.212 ms
8980 bytes from 10.10.1.3: icmp_seq=13 ttl=64 time=0.269 ms
8980 bytes from 10.10.1.3: icmp_seq=14 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.3: icmp_seq=15 ttl=64 time=0.253 ms
8980 bytes from 10.10.1.3: icmp_seq=16 ttl=64 time=0.315 ms
8980 bytes from 10.10.1.3: icmp_seq=17 ttl=64 time=0.267 ms
8980 bytes from 10.10.1.3: icmp_seq=18 ttl=64 time=0.280 ms
8980 bytes from 10.10.1.3: icmp_seq=19 ttl=64 time=0.258 ms
8980 bytes from 10.10.1.3: icmp_seq=20 ttl=64 time=0.258 ms
8980 bytes from 10.10.1.3: icmp_seq=21 ttl=64 time=0.274 ms
8980 bytes from 10.10.1.3: icmp_seq=22 ttl=64 time=0.342 ms
8980 bytes from 10.10.1.3: icmp_seq=23 ttl=64 time=0.228 ms
8980 bytes from 10.10.1.3: icmp_seq=24 ttl=64 time=0.464 ms
8980 bytes from 10.10.1.3: icmp_seq=25 ttl=64 time=0.312 ms
8980 bytes from 10.10.1.3: icmp_seq=26 ttl=64 time=0.239 ms
8980 bytes from 10.10.1.3: icmp_seq=27 ttl=64 time=0.243 ms
8980 bytes from 10.10.1.3: icmp_seq=28 ttl=64 time=0.274 ms
8980 bytes from 10.10.1.3: icmp_seq=29 ttl=64 time=0.254 ms
8980 bytes from 10.10.1.3: icmp_seq=30 ttl=64 time=0.230 ms
8980 bytes from 10.10.1.3: icmp_seq=31 ttl=64 time=0.213 ms
8980 bytes from 10.10.1.3: icmp_seq=32 ttl=64 time=0.338 ms
8980 bytes from 10.10.1.3: icmp_seq=33 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.3: icmp_seq=34 ttl=64 time=0.220 ms
8980 bytes from 10.10.1.3: icmp_seq=35 ttl=64 time=0.278 ms
8980 bytes from 10.10.1.3: icmp_seq=36 ttl=64 time=0.346 ms
8980 bytes from 10.10.1.3: icmp_seq=37 ttl=64 time=0.221 ms
8980 bytes from 10.10.1.3: icmp_seq=38 ttl=64 time=0.205 ms
8980 bytes from 10.10.1.3: icmp_seq=39 ttl=64 time=0.276 ms
8980 bytes from 10.10.1.3: icmp_seq=40 ttl=64 time=0.244 ms
8980 bytes from 10.10.1.3: icmp_seq=41 ttl=64 time=0.209 ms
8980 bytes from 10.10.1.3: icmp_seq=42 ttl=64 time=0.280 ms
8980 bytes from 10.10.1.3: icmp_seq=43 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.3: icmp_seq=44 ttl=64 time=0.325 ms
8980 bytes from 10.10.1.3: icmp_seq=45 ttl=64 time=0.252 ms
8980 bytes from 10.10.1.3: icmp_seq=46 ttl=64 time=0.234 ms
8980 bytes from 10.10.1.3: icmp_seq=47 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.3: icmp_seq=48 ttl=64 time=0.252 ms
8980 bytes from 10.10.1.3: icmp_seq=49 ttl=64 time=0.266 ms
8980 bytes from 10.10.1.3: icmp_seq=50 ttl=64 time=0.363 ms
8980 bytes from 10.10.1.3: icmp_seq=51 ttl=64 time=0.367 ms
8980 bytes from 10.10.1.3: icmp_seq=52 ttl=64 time=0.305 ms
8980 bytes from 10.10.1.3: icmp_seq=53 ttl=64 time=0.324 ms
8980 bytes from 10.10.1.3: icmp_seq=54 ttl=64 time=0.299 ms
8980 bytes from 10.10.1.3: icmp_seq=55 ttl=64 time=0.235 ms
8980 bytes from 10.10.1.3: icmp_seq=56 ttl=64 time=0.314 ms
8980 bytes from 10.10.1.3: icmp_seq=57 ttl=64 time=0.223 ms
8980 bytes from 10.10.1.3: icmp_seq=58 ttl=64 time=0.272 ms
8980 bytes from 10.10.1.3: icmp_seq=59 ttl=64 time=0.378 ms
8980 bytes from 10.10.1.3: icmp_seq=60 ttl=64 time=0.315 ms
8980 bytes from 10.10.1.3: icmp_seq=61 ttl=64 time=0.240 ms
8980 bytes from 10.10.1.3: icmp_seq=62 ttl=64 time=0.293 ms
8980 bytes from 10.10.1.3: icmp_seq=63 ttl=64 time=0.301 ms
8980 bytes from 10.10.1.3: icmp_seq=64 ttl=64 time=0.285 ms
8980 bytes from 10.10.1.3: icmp_seq=65 ttl=64 time=0.206 ms
8980 bytes from 10.10.1.3: icmp_seq=66 ttl=64 time=0.217 ms
8980 bytes from 10.10.1.3: icmp_seq=67 ttl=64 time=0.284 ms
8980 bytes from 10.10.1.3: icmp_seq=68 ttl=64 time=0.271 ms
8980 bytes from 10.10.1.3: icmp_seq=69 ttl=64 time=0.287 ms
8980 bytes from 10.10.1.3: icmp_seq=70 ttl=64 time=0.269 ms
8980 bytes from 10.10.1.3: icmp_seq=71 ttl=64 time=0.223 ms
8980 bytes from 10.10.1.3: icmp_seq=72 ttl=64 time=0.277 ms
8980 bytes from 10.10.1.3: icmp_seq=73 ttl=64 time=0.258 ms
8980 bytes from 10.10.1.3: icmp_seq=74 ttl=64 time=0.208 ms
8980 bytes from 10.10.1.3: icmp_seq=75 ttl=64 time=0.260 ms
8980 bytes from 10.10.1.3: icmp_seq=76 ttl=64 time=0.271 ms
8980 bytes from 10.10.1.3: icmp_seq=77 ttl=64 time=0.316 ms
8980 bytes from 10.10.1.3: icmp_seq=78 ttl=64 time=0.256 ms
8980 bytes from 10.10.1.3: icmp_seq=79 ttl=64 time=0.205 ms
8980 bytes from 10.10.1.3: icmp_seq=80 ttl=64 time=0.191 ms
8980 bytes from 10.10.1.3: icmp_seq=81 ttl=64 time=0.265 ms
8980 bytes from 10.10.1.3: icmp_seq=82 ttl=64 time=0.269 ms
8980 bytes from 10.10.1.3: icmp_seq=83 ttl=64 time=0.230 ms
8980 bytes from 10.10.1.3: icmp_seq=84 ttl=64 time=0.252 ms
8980 bytes from 10.10.1.3: icmp_seq=85 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.3: icmp_seq=86 ttl=64 time=0.252 ms
8980 bytes from 10.10.1.3: icmp_seq=87 ttl=64 time=0.246 ms
8980 bytes from 10.10.1.3: icmp_seq=88 ttl=64 time=0.396 ms
8980 bytes from 10.10.1.3: icmp_seq=89 ttl=64 time=0.299 ms
8980 bytes from 10.10.1.3: icmp_seq=90 ttl=64 time=0.303 ms
8980 bytes from 10.10.1.3: icmp_seq=91 ttl=64 time=0.231 ms
8980 bytes from 10.10.1.3: icmp_seq=92 ttl=64 time=0.249 ms
8980 bytes from 10.10.1.3: icmp_seq=93 ttl=64 time=0.231 ms
8980 bytes from 10.10.1.3: icmp_seq=94 ttl=64 time=0.343 ms
8980 bytes from 10.10.1.3: icmp_seq=95 ttl=64 time=0.262 ms
8980 bytes from 10.10.1.3: icmp_seq=96 ttl=64 time=0.381 ms
8980 bytes from 10.10.1.3: icmp_seq=97 ttl=64 time=0.219 ms
8980 bytes from 10.10.1.3: icmp_seq=98 ttl=64 time=0.272 ms
8980 bytes from 10.10.1.3: icmp_seq=99 ttl=64 time=0.304 ms
--- 10.10.1.3 ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 0.191/0.278/0.464 ms
**************************
[root@esxi006:~]
[root@esxi008:~] for i in `localcli vsan cluster unicastagent list | grep true | awk '{ print $4}'`; do echo "pinging $i 100 times"; echo; vmk
ping -I vmk1 $i -s 8972 -d -c 100 -i 0.1; echo; echo "**************************"; done
pinging 10.10.1.1 100 times
PING 10.10.1.1 (10.10.1.1): 8972 data bytes
8980 bytes from 10.10.1.1: icmp_seq=0 ttl=64 time=0.212 ms
8980 bytes from 10.10.1.1: icmp_seq=1 ttl=64 time=0.163 ms
8980 bytes from 10.10.1.1: icmp_seq=2 ttl=64 time=0.173 ms
8980 bytes from 10.10.1.1: icmp_seq=3 ttl=64 time=0.286 ms
8980 bytes from 10.10.1.1: icmp_seq=4 ttl=64 time=0.402 ms
8980 bytes from 10.10.1.1: icmp_seq=5 ttl=64 time=0.178 ms
8980 bytes from 10.10.1.1: icmp_seq=6 ttl=64 time=0.235 ms
8980 bytes from 10.10.1.1: icmp_seq=7 ttl=64 time=0.243 ms
8980 bytes from 10.10.1.1: icmp_seq=8 ttl=64 time=0.301 ms
8980 bytes from 10.10.1.1: icmp_seq=9 ttl=64 time=0.264 ms
8980 bytes from 10.10.1.1: icmp_seq=10 ttl=64 time=0.228 ms
8980 bytes from 10.10.1.1: icmp_seq=11 ttl=64 time=0.235 ms
8980 bytes from 10.10.1.1: icmp_seq=12 ttl=64 time=0.237 ms
8980 bytes from 10.10.1.1: icmp_seq=13 ttl=64 time=0.328 ms
8980 bytes from 10.10.1.1: icmp_seq=14 ttl=64 time=0.211 ms
8980 bytes from 10.10.1.1: icmp_seq=15 ttl=64 time=0.261 ms
8980 bytes from 10.10.1.1: icmp_seq=16 ttl=64 time=0.308 ms
8980 bytes from 10.10.1.1: icmp_seq=17 ttl=64 time=0.225 ms
8980 bytes from 10.10.1.1: icmp_seq=18 ttl=64 time=0.257 ms
8980 bytes from 10.10.1.1: icmp_seq=19 ttl=64 time=0.250 ms
8980 bytes from 10.10.1.1: icmp_seq=20 ttl=64 time=0.291 ms
8980 bytes from 10.10.1.1: icmp_seq=21 ttl=64 time=0.250 ms
8980 bytes from 10.10.1.1: icmp_seq=22 ttl=64 time=0.209 ms
8980 bytes from 10.10.1.1: icmp_seq=23 ttl=64 time=0.266 ms
8980 bytes from 10.10.1.1: icmp_seq=24 ttl=64 time=0.211 ms
8980 bytes from 10.10.1.1: icmp_seq=25 ttl=64 time=0.224 ms
8980 bytes from 10.10.1.1: icmp_seq=26 ttl=64 time=0.239 ms
8980 bytes from 10.10.1.1: icmp_seq=27 ttl=64 time=0.217 ms
8980 bytes from 10.10.1.1: icmp_seq=28 ttl=64 time=0.305 ms
8980 bytes from 10.10.1.1: icmp_seq=29 ttl=64 time=0.236 ms
8980 bytes from 10.10.1.1: icmp_seq=30 ttl=64 time=0.273 ms
8980 bytes from 10.10.1.1: icmp_seq=31 ttl=64 time=0.215 ms
8980 bytes from 10.10.1.1: icmp_seq=32 ttl=64 time=0.315 ms
8980 bytes from 10.10.1.1: icmp_seq=33 ttl=64 time=0.230 ms
8980 bytes from 10.10.1.1: icmp_seq=34 ttl=64 time=0.265 ms
8980 bytes from 10.10.1.1: icmp_seq=35 ttl=64 time=0.260 ms
8980 bytes from 10.10.1.1: icmp_seq=36 ttl=64 time=0.317 ms
8980 bytes from 10.10.1.1: icmp_seq=37 ttl=64 time=0.191 ms
8980 bytes from 10.10.1.1: icmp_seq=38 ttl=64 time=0.200 ms
8980 bytes from 10.10.1.1: icmp_seq=39 ttl=64 time=0.300 ms
8980 bytes from 10.10.1.1: icmp_seq=40 ttl=64 time=0.197 ms
8980 bytes from 10.10.1.1: icmp_seq=41 ttl=64 time=0.197 ms
8980 bytes from 10.10.1.1: icmp_seq=42 ttl=64 time=0.215 ms
8980 bytes from 10.10.1.1: icmp_seq=43 ttl=64 time=0.331 ms
8980 bytes from 10.10.1.1: icmp_seq=44 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.1: icmp_seq=45 ttl=64 time=0.202 ms
8980 bytes from 10.10.1.1: icmp_seq=46 ttl=64 time=0.264 ms
8980 bytes from 10.10.1.1: icmp_seq=47 ttl=64 time=0.233 ms
8980 bytes from 10.10.1.1: icmp_seq=48 ttl=64 time=0.329 ms
8980 bytes from 10.10.1.1: icmp_seq=49 ttl=64 time=0.297 ms
8980 bytes from 10.10.1.1: icmp_seq=50 ttl=64 time=0.180 ms
8980 bytes from 10.10.1.1: icmp_seq=51 ttl=64 time=0.207 ms
8980 bytes from 10.10.1.1: icmp_seq=52 ttl=64 time=0.341 ms
8980 bytes from 10.10.1.1: icmp_seq=53 ttl=64 time=0.264 ms
8980 bytes from 10.10.1.1: icmp_seq=54 ttl=64 time=0.279 ms
8980 bytes from 10.10.1.1: icmp_seq=55 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.1: icmp_seq=56 ttl=64 time=0.266 ms
8980 bytes from 10.10.1.1: icmp_seq=57 ttl=64 time=0.271 ms
8980 bytes from 10.10.1.1: icmp_seq=58 ttl=64 time=0.249 ms
8980 bytes from 10.10.1.1: icmp_seq=59 ttl=64 time=0.273 ms
8980 bytes from 10.10.1.1: icmp_seq=60 ttl=64 time=0.275 ms
8980 bytes from 10.10.1.1: icmp_seq=61 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.1: icmp_seq=62 ttl=64 time=0.279 ms
8980 bytes from 10.10.1.1: icmp_seq=63 ttl=64 time=0.331 ms
8980 bytes from 10.10.1.1: icmp_seq=64 ttl=64 time=0.170 ms
8980 bytes from 10.10.1.1: icmp_seq=65 ttl=64 time=0.205 ms
8980 bytes from 10.10.1.1: icmp_seq=66 ttl=64 time=0.325 ms
8980 bytes from 10.10.1.1: icmp_seq=67 ttl=64 time=0.290 ms
8980 bytes from 10.10.1.1: icmp_seq=68 ttl=64 time=0.256 ms
8980 bytes from 10.10.1.1: icmp_seq=69 ttl=64 time=0.261 ms
8980 bytes from 10.10.1.1: icmp_seq=70 ttl=64 time=0.306 ms
8980 bytes from 10.10.1.1: icmp_seq=71 ttl=64 time=0.224 ms
8980 bytes from 10.10.1.1: icmp_seq=72 ttl=64 time=0.203 ms
8980 bytes from 10.10.1.1: icmp_seq=73 ttl=64 time=0.256 ms
8980 bytes from 10.10.1.1: icmp_seq=74 ttl=64 time=0.244 ms
8980 bytes from 10.10.1.1: icmp_seq=75 ttl=64 time=0.372 ms
8980 bytes from 10.10.1.1: icmp_seq=76 ttl=64 time=0.267 ms
8980 bytes from 10.10.1.1: icmp_seq=77 ttl=64 time=0.220 ms
8980 bytes from 10.10.1.1: icmp_seq=78 ttl=64 time=0.237 ms
8980 bytes from 10.10.1.1: icmp_seq=79 ttl=64 time=0.247 ms
8980 bytes from 10.10.1.1: icmp_seq=80 ttl=64 time=0.233 ms
8980 bytes from 10.10.1.1: icmp_seq=81 ttl=64 time=0.245 ms
8980 bytes from 10.10.1.1: icmp_seq=82 ttl=64 time=0.192 ms
8980 bytes from 10.10.1.1: icmp_seq=83 ttl=64 time=0.279 ms
8980 bytes from 10.10.1.1: icmp_seq=84 ttl=64 time=0.208 ms
8980 bytes from 10.10.1.1: icmp_seq=85 ttl=64 time=0.299 ms
8980 bytes from 10.10.1.1: icmp_seq=86 ttl=64 time=0.185 ms
8980 bytes from 10.10.1.1: icmp_seq=87 ttl=64 time=0.227 ms
8980 bytes from 10.10.1.1: icmp_seq=88 ttl=64 time=0.294 ms
8980 bytes from 10.10.1.1: icmp_seq=89 ttl=64 time=0.260 ms
8980 bytes from 10.10.1.1: icmp_seq=90 ttl=64 time=0.268 ms
8980 bytes from 10.10.1.1: icmp_seq=91 ttl=64 time=0.166 ms
8980 bytes from 10.10.1.1: icmp_seq=92 ttl=64 time=0.285 ms
8980 bytes from 10.10.1.1: icmp_seq=93 ttl=64 time=0.285 ms
8980 bytes from 10.10.1.1: icmp_seq=94 ttl=64 time=0.209 ms
8980 bytes from 10.10.1.1: icmp_seq=95 ttl=64 time=0.203 ms
8980 bytes from 10.10.1.1: icmp_seq=96 ttl=64 time=0.183 ms
8980 bytes from 10.10.1.1: icmp_seq=97 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.1: icmp_seq=98 ttl=64 time=0.173 ms
8980 bytes from 10.10.1.1: icmp_seq=99 ttl=64 time=0.274 ms
--- 10.10.1.1 ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 0.163/0.251/0.402 ms
**************************
pinging 10.10.1.3 100 times
PING 10.10.1.3 (10.10.1.3): 8972 data bytes
8980 bytes from 10.10.1.3: icmp_seq=0 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.3: icmp_seq=1 ttl=64 time=0.365 ms
8980 bytes from 10.10.1.3: icmp_seq=2 ttl=64 time=0.232 ms
8980 bytes from 10.10.1.3: icmp_seq=3 ttl=64 time=0.377 ms
8980 bytes from 10.10.1.3: icmp_seq=4 ttl=64 time=0.217 ms
8980 bytes from 10.10.1.3: icmp_seq=5 ttl=64 time=0.289 ms
8980 bytes from 10.10.1.3: icmp_seq=6 ttl=64 time=0.316 ms
8980 bytes from 10.10.1.3: icmp_seq=7 ttl=64 time=0.194 ms
8980 bytes from 10.10.1.3: icmp_seq=8 ttl=64 time=0.314 ms
8980 bytes from 10.10.1.3: icmp_seq=9 ttl=64 time=0.240 ms
8980 bytes from 10.10.1.3: icmp_seq=10 ttl=64 time=0.286 ms
8980 bytes from 10.10.1.3: icmp_seq=11 ttl=64 time=0.286 ms
8980 bytes from 10.10.1.3: icmp_seq=12 ttl=64 time=0.344 ms
8980 bytes from 10.10.1.3: icmp_seq=13 ttl=64 time=0.313 ms
8980 bytes from 10.10.1.3: icmp_seq=14 ttl=64 time=0.293 ms
8980 bytes from 10.10.1.3: icmp_seq=15 ttl=64 time=0.345 ms
8980 bytes from 10.10.1.3: icmp_seq=16 ttl=64 time=0.361 ms
8980 bytes from 10.10.1.3: icmp_seq=17 ttl=64 time=0.371 ms
8980 bytes from 10.10.1.3: icmp_seq=18 ttl=64 time=0.356 ms
8980 bytes from 10.10.1.3: icmp_seq=19 ttl=64 time=0.379 ms
8980 bytes from 10.10.1.3: icmp_seq=20 ttl=64 time=0.326 ms
8980 bytes from 10.10.1.3: icmp_seq=21 ttl=64 time=0.334 ms
8980 bytes from 10.10.1.3: icmp_seq=22 ttl=64 time=0.356 ms
8980 bytes from 10.10.1.3: icmp_seq=23 ttl=64 time=0.294 ms
8980 bytes from 10.10.1.3: icmp_seq=24 ttl=64 time=0.234 ms
8980 bytes from 10.10.1.3: icmp_seq=25 ttl=64 time=0.302 ms
8980 bytes from 10.10.1.3: icmp_seq=26 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.3: icmp_seq=27 ttl=64 time=0.300 ms
8980 bytes from 10.10.1.3: icmp_seq=28 ttl=64 time=0.284 ms
8980 bytes from 10.10.1.3: icmp_seq=29 ttl=64 time=0.278 ms
8980 bytes from 10.10.1.3: icmp_seq=30 ttl=64 time=0.283 ms
8980 bytes from 10.10.1.3: icmp_seq=31 ttl=64 time=0.259 ms
8980 bytes from 10.10.1.3: icmp_seq=32 ttl=64 time=0.267 ms
8980 bytes from 10.10.1.3: icmp_seq=33 ttl=64 time=0.350 ms
8980 bytes from 10.10.1.3: icmp_seq=34 ttl=64 time=0.269 ms
8980 bytes from 10.10.1.3: icmp_seq=35 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.3: icmp_seq=36 ttl=64 time=0.247 ms
8980 bytes from 10.10.1.3: icmp_seq=37 ttl=64 time=0.241 ms
8980 bytes from 10.10.1.3: icmp_seq=38 ttl=64 time=0.343 ms
8980 bytes from 10.10.1.3: icmp_seq=39 ttl=64 time=0.340 ms
8980 bytes from 10.10.1.3: icmp_seq=40 ttl=64 time=0.229 ms
8980 bytes from 10.10.1.3: icmp_seq=41 ttl=64 time=0.189 ms
8980 bytes from 10.10.1.3: icmp_seq=42 ttl=64 time=0.267 ms
8980 bytes from 10.10.1.3: icmp_seq=43 ttl=64 time=0.331 ms
8980 bytes from 10.10.1.3: icmp_seq=44 ttl=64 time=0.335 ms
8980 bytes from 10.10.1.3: icmp_seq=45 ttl=64 time=0.219 ms
8980 bytes from 10.10.1.3: icmp_seq=46 ttl=64 time=0.216 ms
8980 bytes from 10.10.1.3: icmp_seq=47 ttl=64 time=0.236 ms
8980 bytes from 10.10.1.3: icmp_seq=48 ttl=64 time=0.275 ms
8980 bytes from 10.10.1.3: icmp_seq=49 ttl=64 time=0.263 ms
8980 bytes from 10.10.1.3: icmp_seq=50 ttl=64 time=0.349 ms
8980 bytes from 10.10.1.3: icmp_seq=51 ttl=64 time=0.259 ms
8980 bytes from 10.10.1.3: icmp_seq=52 ttl=64 time=0.219 ms
8980 bytes from 10.10.1.3: icmp_seq=53 ttl=64 time=0.265 ms
8980 bytes from 10.10.1.3: icmp_seq=54 ttl=64 time=0.274 ms
8980 bytes from 10.10.1.3: icmp_seq=55 ttl=64 time=0.380 ms
8980 bytes from 10.10.1.3: icmp_seq=56 ttl=64 time=0.233 ms
8980 bytes from 10.10.1.3: icmp_seq=57 ttl=64 time=0.223 ms
8980 bytes from 10.10.1.3: icmp_seq=58 ttl=64 time=0.253 ms
8980 bytes from 10.10.1.3: icmp_seq=59 ttl=64 time=0.227 ms
8980 bytes from 10.10.1.3: icmp_seq=60 ttl=64 time=0.230 ms
8980 bytes from 10.10.1.3: icmp_seq=61 ttl=64 time=0.327 ms
8980 bytes from 10.10.1.3: icmp_seq=62 ttl=64 time=0.261 ms
8980 bytes from 10.10.1.3: icmp_seq=63 ttl=64 time=0.318 ms
8980 bytes from 10.10.1.3: icmp_seq=64 ttl=64 time=0.253 ms
8980 bytes from 10.10.1.3: icmp_seq=65 ttl=64 time=0.335 ms
8980 bytes from 10.10.1.3: icmp_seq=66 ttl=64 time=0.211 ms
8980 bytes from 10.10.1.3: icmp_seq=67 ttl=64 time=0.298 ms
8980 bytes from 10.10.1.3: icmp_seq=68 ttl=64 time=0.346 ms
8980 bytes from 10.10.1.3: icmp_seq=69 ttl=64 time=0.306 ms
8980 bytes from 10.10.1.3: icmp_seq=70 ttl=64 time=0.266 ms
8980 bytes from 10.10.1.3: icmp_seq=71 ttl=64 time=0.355 ms
8980 bytes from 10.10.1.3: icmp_seq=72 ttl=64 time=0.228 ms
8980 bytes from 10.10.1.3: icmp_seq=73 ttl=64 time=0.313 ms
8980 bytes from 10.10.1.3: icmp_seq=74 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.3: icmp_seq=75 ttl=64 time=0.320 ms
8980 bytes from 10.10.1.3: icmp_seq=76 ttl=64 time=0.333 ms
8980 bytes from 10.10.1.3: icmp_seq=77 ttl=64 time=0.333 ms
8980 bytes from 10.10.1.3: icmp_seq=78 ttl=64 time=0.325 ms
8980 bytes from 10.10.1.3: icmp_seq=79 ttl=64 time=0.340 ms
8980 bytes from 10.10.1.3: icmp_seq=80 ttl=64 time=0.314 ms
8980 bytes from 10.10.1.3: icmp_seq=81 ttl=64 time=0.298 ms
8980 bytes from 10.10.1.3: icmp_seq=82 ttl=64 time=0.245 ms
8980 bytes from 10.10.1.3: icmp_seq=83 ttl=64 time=0.323 ms
8980 bytes from 10.10.1.3: icmp_seq=84 ttl=64 time=0.331 ms
8980 bytes from 10.10.1.3: icmp_seq=85 ttl=64 time=0.302 ms
8980 bytes from 10.10.1.3: icmp_seq=86 ttl=64 time=0.310 ms
8980 bytes from 10.10.1.3: icmp_seq=87 ttl=64 time=0.292 ms
8980 bytes from 10.10.1.3: icmp_seq=88 ttl=64 time=0.343 ms
8980 bytes from 10.10.1.3: icmp_seq=89 ttl=64 time=0.340 ms
8980 bytes from 10.10.1.3: icmp_seq=90 ttl=64 time=0.280 ms
8980 bytes from 10.10.1.3: icmp_seq=91 ttl=64 time=0.296 ms
8980 bytes from 10.10.1.3: icmp_seq=92 ttl=64 time=0.459 ms
8980 bytes from 10.10.1.3: icmp_seq=93 ttl=64 time=0.278 ms
8980 bytes from 10.10.1.3: icmp_seq=94 ttl=64 time=0.277 ms
8980 bytes from 10.10.1.3: icmp_seq=95 ttl=64 time=0.312 ms
8980 bytes from 10.10.1.3: icmp_seq=96 ttl=64 time=0.306 ms
8980 bytes from 10.10.1.3: icmp_seq=97 ttl=64 time=0.274 ms
8980 bytes from 10.10.1.3: icmp_seq=98 ttl=64 time=0.206 ms
8980 bytes from 10.10.1.3: icmp_seq=99 ttl=64 time=0.251 ms
--- 10.10.1.3 ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 0.189/0.292/0.459 ms
**************************
[root@esxi008:~]
[root@esxi010:~] for i in `localcli vsan cluster unicastagent list | grep true | awk '{ print $4}'`; do echo "pinging $i 100 times"; echo;
vmkping -I vmk1 $i -s 8972 -d -c 100 -i 0.1; echo; echo "**************************"; done
pinging 10.10.1.1 100 times
PING 10.10.1.1 (10.10.1.1): 8972 data bytes
8980 bytes from 10.10.1.1: icmp_seq=0 ttl=64 time=0.254 ms
8980 bytes from 10.10.1.1: icmp_seq=1 ttl=64 time=0.215 ms
8980 bytes from 10.10.1.1: icmp_seq=2 ttl=64 time=0.285 ms
8980 bytes from 10.10.1.1: icmp_seq=3 ttl=64 time=0.203 ms
8980 bytes from 10.10.1.1: icmp_seq=4 ttl=64 time=0.210 ms
8980 bytes from 10.10.1.1: icmp_seq=5 ttl=64 time=0.326 ms
8980 bytes from 10.10.1.1: icmp_seq=6 ttl=64 time=0.245 ms
8980 bytes from 10.10.1.1: icmp_seq=7 ttl=64 time=0.214 ms
8980 bytes from 10.10.1.1: icmp_seq=8 ttl=64 time=0.225 ms
8980 bytes from 10.10.1.1: icmp_seq=9 ttl=64 time=0.277 ms
8980 bytes from 10.10.1.1: icmp_seq=10 ttl=64 time=0.246 ms
8980 bytes from 10.10.1.1: icmp_seq=11 ttl=64 time=0.276 ms
8980 bytes from 10.10.1.1: icmp_seq=12 ttl=64 time=0.268 ms
8980 bytes from 10.10.1.1: icmp_seq=13 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.1: icmp_seq=14 ttl=64 time=0.328 ms
8980 bytes from 10.10.1.1: icmp_seq=15 ttl=64 time=0.288 ms
8980 bytes from 10.10.1.1: icmp_seq=16 ttl=64 time=0.361 ms
8980 bytes from 10.10.1.1: icmp_seq=17 ttl=64 time=0.383 ms
8980 bytes from 10.10.1.1: icmp_seq=18 ttl=64 time=0.367 ms
8980 bytes from 10.10.1.1: icmp_seq=19 ttl=64 time=0.290 ms
8980 bytes from 10.10.1.1: icmp_seq=20 ttl=64 time=0.335 ms
8980 bytes from 10.10.1.1: icmp_seq=21 ttl=64 time=0.283 ms
8980 bytes from 10.10.1.1: icmp_seq=22 ttl=64 time=0.369 ms
8980 bytes from 10.10.1.1: icmp_seq=23 ttl=64 time=0.235 ms
8980 bytes from 10.10.1.1: icmp_seq=24 ttl=64 time=0.244 ms
8980 bytes from 10.10.1.1: icmp_seq=25 ttl=64 time=0.286 ms
8980 bytes from 10.10.1.1: icmp_seq=26 ttl=64 time=0.275 ms
8980 bytes from 10.10.1.1: icmp_seq=27 ttl=64 time=0.287 ms
8980 bytes from 10.10.1.1: icmp_seq=28 ttl=64 time=0.286 ms
8980 bytes from 10.10.1.1: icmp_seq=29 ttl=64 time=0.284 ms
8980 bytes from 10.10.1.1: icmp_seq=30 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.1: icmp_seq=31 ttl=64 time=0.234 ms
8980 bytes from 10.10.1.1: icmp_seq=32 ttl=64 time=0.290 ms
8980 bytes from 10.10.1.1: icmp_seq=33 ttl=64 time=0.312 ms
8980 bytes from 10.10.1.1: icmp_seq=34 ttl=64 time=0.382 ms
8980 bytes from 10.10.1.1: icmp_seq=35 ttl=64 time=0.223 ms
8980 bytes from 10.10.1.1: icmp_seq=36 ttl=64 time=0.286 ms
8980 bytes from 10.10.1.1: icmp_seq=37 ttl=64 time=0.368 ms
8980 bytes from 10.10.1.1: icmp_seq=38 ttl=64 time=0.328 ms
8980 bytes from 10.10.1.1: icmp_seq=39 ttl=64 time=0.310 ms
8980 bytes from 10.10.1.1: icmp_seq=40 ttl=64 time=0.290 ms
8980 bytes from 10.10.1.1: icmp_seq=41 ttl=64 time=0.356 ms
8980 bytes from 10.10.1.1: icmp_seq=42 ttl=64 time=0.215 ms
8980 bytes from 10.10.1.1: icmp_seq=43 ttl=64 time=0.282 ms
8980 bytes from 10.10.1.1: icmp_seq=44 ttl=64 time=0.319 ms
8980 bytes from 10.10.1.1: icmp_seq=45 ttl=64 time=0.302 ms
8980 bytes from 10.10.1.1: icmp_seq=46 ttl=64 time=0.268 ms
8980 bytes from 10.10.1.1: icmp_seq=47 ttl=64 time=0.307 ms
8980 bytes from 10.10.1.1: icmp_seq=48 ttl=64 time=0.297 ms
8980 bytes from 10.10.1.1: icmp_seq=49 ttl=64 time=0.289 ms
8980 bytes from 10.10.1.1: icmp_seq=50 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.1: icmp_seq=51 ttl=64 time=0.246 ms
8980 bytes from 10.10.1.1: icmp_seq=52 ttl=64 time=0.253 ms
8980 bytes from 10.10.1.1: icmp_seq=53 ttl=64 time=0.292 ms
8980 bytes from 10.10.1.1: icmp_seq=54 ttl=64 time=0.368 ms
8980 bytes from 10.10.1.1: icmp_seq=55 ttl=64 time=0.314 ms
8980 bytes from 10.10.1.1: icmp_seq=56 ttl=64 time=0.269 ms
8980 bytes from 10.10.1.1: icmp_seq=57 ttl=64 time=0.356 ms
8980 bytes from 10.10.1.1: icmp_seq=58 ttl=64 time=0.382 ms
8980 bytes from 10.10.1.1: icmp_seq=59 ttl=64 time=0.213 ms
8980 bytes from 10.10.1.1: icmp_seq=60 ttl=64 time=0.226 ms
8980 bytes from 10.10.1.1: icmp_seq=61 ttl=64 time=0.290 ms
8980 bytes from 10.10.1.1: icmp_seq=62 ttl=64 time=0.312 ms
8980 bytes from 10.10.1.1: icmp_seq=63 ttl=64 time=0.218 ms
8980 bytes from 10.10.1.1: icmp_seq=64 ttl=64 time=0.298 ms
8980 bytes from 10.10.1.1: icmp_seq=65 ttl=64 time=0.301 ms
8980 bytes from 10.10.1.1: icmp_seq=66 ttl=64 time=0.223 ms
8980 bytes from 10.10.1.1: icmp_seq=67 ttl=64 time=0.204 ms
8980 bytes from 10.10.1.1: icmp_seq=68 ttl=64 time=0.321 ms
8980 bytes from 10.10.1.1: icmp_seq=69 ttl=64 time=0.260 ms
8980 bytes from 10.10.1.1: icmp_seq=70 ttl=64 time=0.229 ms
8980 bytes from 10.10.1.1: icmp_seq=71 ttl=64 time=0.227 ms
8980 bytes from 10.10.1.1: icmp_seq=72 ttl=64 time=0.263 ms
8980 bytes from 10.10.1.1: icmp_seq=73 ttl=64 time=0.224 ms
8980 bytes from 10.10.1.1: icmp_seq=74 ttl=64 time=0.258 ms
8980 bytes from 10.10.1.1: icmp_seq=75 ttl=64 time=0.303 ms
8980 bytes from 10.10.1.1: icmp_seq=76 ttl=64 time=0.265 ms
8980 bytes from 10.10.1.1: icmp_seq=77 ttl=64 time=0.281 ms
8980 bytes from 10.10.1.1: icmp_seq=78 ttl=64 time=0.318 ms
8980 bytes from 10.10.1.1: icmp_seq=79 ttl=64 time=0.282 ms
8980 bytes from 10.10.1.1: icmp_seq=80 ttl=64 time=0.225 ms
8980 bytes from 10.10.1.1: icmp_seq=81 ttl=64 time=0.334 ms
8980 bytes from 10.10.1.1: icmp_seq=82 ttl=64 time=0.276 ms
8980 bytes from 10.10.1.1: icmp_seq=83 ttl=64 time=0.246 ms
8980 bytes from 10.10.1.1: icmp_seq=84 ttl=64 time=0.238 ms
8980 bytes from 10.10.1.1: icmp_seq=85 ttl=64 time=0.291 ms
8980 bytes from 10.10.1.1: icmp_seq=86 ttl=64 time=0.252 ms
8980 bytes from 10.10.1.1: icmp_seq=87 ttl=64 time=0.317 ms
8980 bytes from 10.10.1.1: icmp_seq=88 ttl=64 time=0.275 ms
8980 bytes from 10.10.1.1: icmp_seq=89 ttl=64 time=0.278 ms
8980 bytes from 10.10.1.1: icmp_seq=90 ttl=64 time=0.296 ms
8980 bytes from 10.10.1.1: icmp_seq=91 ttl=64 time=0.285 ms
8980 bytes from 10.10.1.1: icmp_seq=92 ttl=64 time=0.314 ms
8980 bytes from 10.10.1.1: icmp_seq=93 ttl=64 time=0.324 ms
8980 bytes from 10.10.1.1: icmp_seq=94 ttl=64 time=0.315 ms
8980 bytes from 10.10.1.1: icmp_seq=95 ttl=64 time=0.359 ms
8980 bytes from 10.10.1.1: icmp_seq=96 ttl=64 time=0.307 ms
8980 bytes from 10.10.1.1: icmp_seq=97 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.1: icmp_seq=98 ttl=64 time=0.211 ms
8980 bytes from 10.10.1.1: icmp_seq=99 ttl=64 time=0.288 ms
--- 10.10.1.1 ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 0.203/0.282/0.383 ms
**************************
pinging 10.10.1.2 100 times
PING 10.10.1.2 (10.10.1.2): 8972 data bytes
8980 bytes from 10.10.1.2: icmp_seq=0 ttl=64 time=0.228 ms
8980 bytes from 10.10.1.2: icmp_seq=1 ttl=64 time=0.252 ms
8980 bytes from 10.10.1.2: icmp_seq=2 ttl=64 time=0.352 ms
8980 bytes from 10.10.1.2: icmp_seq=3 ttl=64 time=0.248 ms
8980 bytes from 10.10.1.2: icmp_seq=4 ttl=64 time=0.314 ms
8980 bytes from 10.10.1.2: icmp_seq=5 ttl=64 time=0.291 ms
8980 bytes from 10.10.1.2: icmp_seq=6 ttl=64 time=0.275 ms
8980 bytes from 10.10.1.2: icmp_seq=7 ttl=64 time=0.365 ms
8980 bytes from 10.10.1.2: icmp_seq=8 ttl=64 time=0.250 ms
8980 bytes from 10.10.1.2: icmp_seq=9 ttl=64 time=0.375 ms
8980 bytes from 10.10.1.2: icmp_seq=10 ttl=64 time=0.261 ms
8980 bytes from 10.10.1.2: icmp_seq=11 ttl=64 time=0.280 ms
8980 bytes from 10.10.1.2: icmp_seq=12 ttl=64 time=0.379 ms
8980 bytes from 10.10.1.2: icmp_seq=13 ttl=64 time=0.287 ms
8980 bytes from 10.10.1.2: icmp_seq=14 ttl=64 time=0.307 ms
8980 bytes from 10.10.1.2: icmp_seq=15 ttl=64 time=0.262 ms
8980 bytes from 10.10.1.2: icmp_seq=16 ttl=64 time=0.347 ms
8980 bytes from 10.10.1.2: icmp_seq=17 ttl=64 time=0.345 ms
8980 bytes from 10.10.1.2: icmp_seq=18 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.2: icmp_seq=19 ttl=64 time=0.284 ms
8980 bytes from 10.10.1.2: icmp_seq=20 ttl=64 time=0.307 ms
8980 bytes from 10.10.1.2: icmp_seq=21 ttl=64 time=0.329 ms
8980 bytes from 10.10.1.2: icmp_seq=22 ttl=64 time=0.316 ms
8980 bytes from 10.10.1.2: icmp_seq=23 ttl=64 time=0.322 ms
8980 bytes from 10.10.1.2: icmp_seq=24 ttl=64 time=0.324 ms
8980 bytes from 10.10.1.2: icmp_seq=25 ttl=64 time=0.209 ms
8980 bytes from 10.10.1.2: icmp_seq=26 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.2: icmp_seq=27 ttl=64 time=0.266 ms
8980 bytes from 10.10.1.2: icmp_seq=28 ttl=64 time=0.346 ms
8980 bytes from 10.10.1.2: icmp_seq=29 ttl=64 time=0.271 ms
8980 bytes from 10.10.1.2: icmp_seq=30 ttl=64 time=0.358 ms
8980 bytes from 10.10.1.2: icmp_seq=31 ttl=64 time=0.255 ms
8980 bytes from 10.10.1.2: icmp_seq=32 ttl=64 time=0.234 ms
8980 bytes from 10.10.1.2: icmp_seq=33 ttl=64 time=0.272 ms
8980 bytes from 10.10.1.2: icmp_seq=34 ttl=64 time=0.334 ms
8980 bytes from 10.10.1.2: icmp_seq=35 ttl=64 time=0.352 ms
8980 bytes from 10.10.1.2: icmp_seq=36 ttl=64 time=0.266 ms
8980 bytes from 10.10.1.2: icmp_seq=37 ttl=64 time=0.232 ms
8980 bytes from 10.10.1.2: icmp_seq=38 ttl=64 time=0.397 ms
8980 bytes from 10.10.1.2: icmp_seq=39 ttl=64 time=0.355 ms
8980 bytes from 10.10.1.2: icmp_seq=40 ttl=64 time=0.259 ms
8980 bytes from 10.10.1.2: icmp_seq=41 ttl=64 time=0.299 ms
8980 bytes from 10.10.1.2: icmp_seq=42 ttl=64 time=0.357 ms
8980 bytes from 10.10.1.2: icmp_seq=43 ttl=64 time=0.224 ms
8980 bytes from 10.10.1.2: icmp_seq=44 ttl=64 time=0.262 ms
8980 bytes from 10.10.1.2: icmp_seq=45 ttl=64 time=0.327 ms
8980 bytes from 10.10.1.2: icmp_seq=46 ttl=64 time=0.350 ms
8980 bytes from 10.10.1.2: icmp_seq=47 ttl=64 time=0.216 ms
8980 bytes from 10.10.1.2: icmp_seq=48 ttl=64 time=0.270 ms
8980 bytes from 10.10.1.2: icmp_seq=49 ttl=64 time=0.350 ms
8980 bytes from 10.10.1.2: icmp_seq=50 ttl=64 time=0.208 ms
8980 bytes from 10.10.1.2: icmp_seq=51 ttl=64 time=0.435 ms
8980 bytes from 10.10.1.2: icmp_seq=52 ttl=64 time=0.284 ms
8980 bytes from 10.10.1.2: icmp_seq=53 ttl=64 time=0.314 ms
8980 bytes from 10.10.1.2: icmp_seq=54 ttl=64 time=0.325 ms
8980 bytes from 10.10.1.2: icmp_seq=55 ttl=64 time=0.298 ms
8980 bytes from 10.10.1.2: icmp_seq=56 ttl=64 time=0.257 ms
8980 bytes from 10.10.1.2: icmp_seq=57 ttl=64 time=0.224 ms
8980 bytes from 10.10.1.2: icmp_seq=58 ttl=64 time=0.251 ms
8980 bytes from 10.10.1.2: icmp_seq=59 ttl=64 time=0.276 ms
8980 bytes from 10.10.1.2: icmp_seq=60 ttl=64 time=0.267 ms
8980 bytes from 10.10.1.2: icmp_seq=61 ttl=64 time=0.337 ms
8980 bytes from 10.10.1.2: icmp_seq=62 ttl=64 time=0.267 ms
8980 bytes from 10.10.1.2: icmp_seq=63 ttl=64 time=0.278 ms
8980 bytes from 10.10.1.2: icmp_seq=64 ttl=64 time=0.242 ms
8980 bytes from 10.10.1.2: icmp_seq=65 ttl=64 time=0.283 ms
8980 bytes from 10.10.1.2: icmp_seq=66 ttl=64 time=0.212 ms
8980 bytes from 10.10.1.2: icmp_seq=67 ttl=64 time=0.283 ms
8980 bytes from 10.10.1.2: icmp_seq=68 ttl=64 time=0.240 ms
8980 bytes from 10.10.1.2: icmp_seq=69 ttl=64 time=0.273 ms
8980 bytes from 10.10.1.2: icmp_seq=70 ttl=64 time=0.263 ms
8980 bytes from 10.10.1.2: icmp_seq=71 ttl=64 time=0.272 ms
8980 bytes from 10.10.1.2: icmp_seq=72 ttl=64 time=0.213 ms
8980 bytes from 10.10.1.2: icmp_seq=73 ttl=64 time=0.220 ms
8980 bytes from 10.10.1.2: icmp_seq=74 ttl=64 time=0.228 ms
8980 bytes from 10.10.1.2: icmp_seq=75 ttl=64 time=0.255 ms
8980 bytes from 10.10.1.2: icmp_seq=76 ttl=64 time=0.238 ms
8980 bytes from 10.10.1.2: icmp_seq=77 ttl=64 time=0.176 ms
8980 bytes from 10.10.1.2: icmp_seq=78 ttl=64 time=0.261 ms
8980 bytes from 10.10.1.2: icmp_seq=79 ttl=64 time=0.286 ms
8980 bytes from 10.10.1.2: icmp_seq=80 ttl=64 time=0.243 ms
8980 bytes from 10.10.1.2: icmp_seq=81 ttl=64 time=0.244 ms
8980 bytes from 10.10.1.2: icmp_seq=82 ttl=64 time=0.301 ms
8980 bytes from 10.10.1.2: icmp_seq=83 ttl=64 time=0.207 ms
8980 bytes from 10.10.1.2: icmp_seq=84 ttl=64 time=0.304 ms
8980 bytes from 10.10.1.2: icmp_seq=85 ttl=64 time=0.315 ms
8980 bytes from 10.10.1.2: icmp_seq=86 ttl=64 time=0.317 ms
8980 bytes from 10.10.1.2: icmp_seq=87 ttl=64 time=0.306 ms
8980 bytes from 10.10.1.2: icmp_seq=88 ttl=64 time=0.222 ms
8980 bytes from 10.10.1.2: icmp_seq=89 ttl=64 time=0.254 ms
8980 bytes from 10.10.1.2: icmp_seq=90 ttl=64 time=0.283 ms
8980 bytes from 10.10.1.2: icmp_seq=91 ttl=64 time=0.199 ms
8980 bytes from 10.10.1.2: icmp_seq=92 ttl=64 time=0.250 ms
8980 bytes from 10.10.1.2: icmp_seq=93 ttl=64 time=0.319 ms
8980 bytes from 10.10.1.2: icmp_seq=94 ttl=64 time=0.248 ms
8980 bytes from 10.10.1.2: icmp_seq=95 ttl=64 time=0.391 ms
8980 bytes from 10.10.1.2: icmp_seq=96 ttl=64 time=0.317 ms
8980 bytes from 10.10.1.2: icmp_seq=97 ttl=64 time=0.297 ms
8980 bytes from 10.10.1.2: icmp_seq=98 ttl=64 time=0.280 ms
8980 bytes from 10.10.1.2: icmp_seq=99 ttl=64 time=0.266 ms
--- 10.10.1.2 ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 0.176/0.284/0.435 ms
**************************
[root@esxi010:~]
esxi006 and esxi008 both report current value of 0 to
vsish -e get /config/VSAN/intOpts/DOMPauseAllCCPs
but esxi010 reports a 1
[root@esxi010:~] vsish -e get /config/VSAN/intOpts/DOMPauseAllCCPs
Vmkernel Config Option {
Default value:0
Min value:0
Max value:1
Current value:1
hidden config option:1
Description:Setting this option will block all CCP in owner setup, resulting in objects coming to APD state. Used for cluster reboot/shutdown
Host specific config option:0
Option update requires reboot:0
Option update requires maintenance mode:0
}
esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates
It is zero on all hosts
[root@esxi010:~] esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates
Value of IgnoreClusterMemberListUpdates is 0
Currently a new alert has appeared
Stats primary election
@einstein-a-go-g DOMPauseAllCCPs should be set to 0 on all nodes, set this on any that are set to 1:
# vsish -e set /config/VSAN/intOpts/DOMPauseAllCCPs 0
After doing that, my bet is everything will be fine here.
The Stats primary election state is likely due to the vSAN performance stats objects being one of the impacted objects, this should also clear once above is done.
I was going to set it to 1, but I thought I would wait for a response, because I know what it's like helping people via forums.
The Red Alert has disappeared in Skyline Health, and Syncing Objects has started.
So, the cause, this host did crash a while back, and this was set to 1 and caused all these issues ?
The Proactive Tests works correctly now, and does no take 5 minutes for the "faulty" host to respond.
@einstein-a-go-g This parameter gets set to 1 (enabled) as part of vSAN cluster shutdown feature initiated from vCenter - for this to be enabled on nodes basically means this must have been run at some point and not undone/undone correctly/fully,
All your objects are healthy and cluster functioning normally now, yes?
Yes, initial tests now looking good. Still checking.
The only issue I remember is that esxi010 crashed with a PSOD, and after that virtual machines were becoming inaccessible on all hosts and disappearing after they were powered down, and I lost all virtual machines on vSAN, requiring restore from backups.
I then noticed it was the power off which caused the machines to become inaccessible, they were still up, so I quickly migrated the remainder of VMs on vSAN to NFS datastore to save them - that worked.
But any other new VMs created on vSAN datastore on any host was unreliable, and originally all was green according to Skyline Health.
Lessons learned here.
