We purchased new Dell R760 with 7 onboard NVMe SSD drives that are setup in a RAID 5 on a PERC H965i.
The server are setup with the custom Dell ISO of ESXi 8.0.1 build-21813344.
I was setting up a new windows server to act as a proxy for veeam and notice the server just hung up for around 15 mins (which might not be related to the issue i am posting about). So I went out to see if there was something going on using esxtop. Everything was fine except for the storage area. There was barely any activity happening with the windows server as far as disk activity went but the latency numbers are a real head scratcher.
This is one sampling and this just randomly happens with 1 VM setup.
CMD/s: 626.54
READS/s: 622.37
WRITES/s: 4.16
MBREAD/s: 2.57
DAVG/cmd: 136384.03
KAVG/cmd: -136383.89 (Yes that is negative)
GAVG/cmd: .014
QAVG/cmd: 142262.59
I downloaded iometer to do a load test of the storage and the numbers showed what i would expect.
CMD/s: 225445.98
READS/s: 113026.56
WRITES/s: 112419.42
MBREAD/s: 110.38
DAVG/cmd: 0.06
KAVG/cmd: 0.00
GAVG/cmd: 0.06
QAVG/cmd: 0.00
I have opened a ticket with vmware but wanted to ask the community if anyone has seen anything like this.
Also I was on the phone with dell pro support for about 3 hours and they wanted me to call vmware since they could not find anything.
All drivers and firmware on the storage are up to date.
I have not put this server into production yet until an answer can be found out. My fear is I will move over servers and there will be an issue.