Perhaps you did, but even so I wanna ask you.
to specify the case --> Did you try to move virtual machines onto different ESXi hosts? I wanna sure it is happening on same host.
Compatibility --> On the other hand, please check NIC firmware and driver version, maybe NIC could not handle the traffic accurately.
Compatibiliy --> please check snart array controller card driver and firmware version? Are they certified for VMware VSAN?
Health --> Did you check VMware VSAN monitoring tab "health" on VSAN? İs it green?
Network --> What is your Network design for VSAN? Share network on VSAN or dedicated VDS with dedicated uplink. If you use shared network for VSAN/Data/Vmotion, did you enabled NIOC? You should prioritize VSAN and Data traffic on VDS by increasing their share value.
Thanks for the reply, first of all, this phenomenon can occur on any host. The network is configured for each host 2x10GB network, the network delay is normal, and the daily traffic is very small. Vsan health is all green, and disk controllers and drivers are all certified.
Did you check physical network site? Switch ports, discard packets, CRC..
I faced this issue, the problem was pyhsical switch side. It couldnt handle mircoburst traffic.
As far as I understand you use one VDS with 2x10 uplink per server. Did you prioritize VSAN traffic via NIOC on VDS? If there is a congestion on server side, it would evaluate VSAN traffic better.
Sounds similar to this post I've just commented on here.
What switches are you running? Check your switches for output discards.
Do you see network errors in observer? We have duplicate data, acks, retransmits and out of order frames.
Thanks for the reply, I think it is not a network problem, the first choice, continuous monitoring of network delay has been ongoing, no abnormalities have been found. At the same time, continuous copying and pull operation testing of large files between hosts and virtual machines does not cause the above write latency problem.
Try this: take a VM, apply a Storage Policy to it which has no redundancy (FTT=0), then look at the physical placement of the components (on which host) and vMotion the VM to that host and make sure it stays there. Now, you are forcing data-locality.
Then, check the latency.
If the problem is still there, you can rule out the network as the network is not used when the VM sits on the same host that has the only data (as there is no mirror copy somewhere else it needs to read/write from).
If the latency-problem is gone, you DO have a network issue. To verify, give the VM a policy with FTT=1 and let it finish re-creating the object mirrors. After that, the problem should be back, indicating that there is a delay being introduced when hosts communicate with each other over the network.
Saying "I do not think we have a network issue" is a bit dangerous as many vSAN problems come from the network. I see it all the time. I see people route the traffic between vSAN nodes over many hops and do all kind of things that introduce inter-node latency.
Another thing to try, force the vSAN vmkernel-port to flow over another NIC. This usually means another switch as well (hopefully). See what happens then.
Also, as you have a write-latency issue and not a read-latency issue (at least you don't speak about it), is the cluster stretched? Do you have Site-read-locality activated?
Temporarily we solved the issue by migrating VSAN backend traffic onto existing Nexus 5672 UP switches. It is running now very stable now.
Data/vMotion/management traffic still running on Nexus 5548 switches.
Final decision with our network team, we will replace Cisco 5548UP switches in near time with recent Cisco Nexus series switches. Because old Nexus are not aware microburst traffic and doesnt handle high packet transmits.
Thanks for your suggestion, this delay occurs on a certain number of virtual ones, and the delay problem mainly occurs in mysql write and query. After we have recently deployed the mysql database to another virtual machine, the problem is solved. We tried the Windows and centos7.6 versions of the operating system no longer have a delay. The virtual machine we had problems with was the centos7.3 version. Is this related to the operating system?