Does ESXi maintain write buffer/cache?

[kirand]

Hello,

I am observing high random write IOPS in ESXi6.5 with NVMe devices. When I say high IOPS, it is around 150% of the actual spec(HGST NVMe drive SPEC 4K random write=140K, test result IOPS=220K). My expectation was to see less than 100% of the spec value but the result is strange.

I have checked with another Huawei device to confirm and the behavior is same.(result is more than 200% of spec value)

VMware published a paper on ESXi 6.5 VM encryption where we can see baseline performance of Samsung NVMe PM1725 Random Write IOPS as 350K (Spec is 120K). Don't know how that is possible to achieve 3 times more than the vendor spec value.

This is possible only when ESXi cache the IO somewhere in the stack. Does ESXi buffer/cache the write IO? Any information on this behavior would be appreciated.

VM Encryption paper: http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vm-encryption-vsphere65-p...

Page number 10 shows 350K random write baseline performance with Samsung PM1725

Samsung PM1725 Data Sheet:: http://www.samsung.com/semiconductor/global/file/insight/2015/11/pm1725-ProdOverview-2015-0.pdf

This shows the actual spec value of 120K random write IOPS

(test env: ESXi 6.5, 1x Ubuntu VM;8vCPU; 32GB RAM, 3 pvscsi controllers, 3 VMDK 100GB each, 4K 100%random 0% read)

[Linjo]

How many hosts? How many disk-groups? FTT?

How do you do the test? IO-meter?

// Linjo

[kirand]

I am going to use the same NVMe drive as cache drive in my VSAN cluster which I am currently building, I just wanted to note down the baseline performance before I look at VSAN performance. The results are from a single standalone ESXi 6.5 host and single VM with a datastore created from the NVMe SSD.

Test Setup:

Server: Dell PowerEdge R730

CPU: 2x Intel® Xeon® E5-2650 v4 @2.2GHz (12Core)

RAM: 128GB

VM Config: 8vCPU, 32GB RAM, 3 PVSCSI Controllers, Ubuntu 16.04 LTS

VMDK: 3VMDK of each 100GB; one disk to each pvscsi controller

BM Tool: FIO; 4K 100% Random 0% Read;

teady state: Ran the workload after few hours of filling the drive( to avoid the zero fill and to precondition the drive)

Other performance numbers (4K Random Read, 128K Sequential Read, 128K Sequential Write) are as expected and only Random Writes behavior is abnormal.

Does the size of the disk (100GB VMDK vs 1600GB actual disk size) has anything to do with the Random Write performance? I know, the size matters in case of HDD, but not sure if randomness has any impact on the flash drive size.