Buffered Storage I/O in ESX 4?

[CrendKing]

Hello everyone. I have a question: does ESX 4 always do buffered storage I/O? I found a knowledge base article at Storage IO crash consistency with VMware products saying "VMware ESX acknowledges a write or read to a guest operating system only after that write or read is acknowledged by the hardware controller to ESX. Applications running inside virtual machines on ESX are afforded the same crash consistency guarantees as applications running on physical machines or physical disk controllers." I suppose ESX should do unbuffered IO. However, I opened a RDM disk with and without O_SYNC open flag, I noticed that the throughput with O_SYNC is much lower than without O_SYNC. It seems ESX is still doing buffered IO. So is it because the knowledge base article out-of-date, or my set-up was incorrect? Thanks!

[mcowger]

Neither.

ESX always does unbuffered IO, but your GUEST OS or array may do its own buffering. By opening your file with O_SYNC you bypassed the guest OS's caching or your array's cache, resulting in lower performance.

Basically, ESX does none of its own caching, but your Guest or array could do its own, which ESX never prevents or disables.

--Matt

VCP, vExpert, Unix Geek

[CrendKing]

Thanks for reply! I can understand that guest OS have cache or buffer for file system. But for block devices, do OS also make buffer for them?

Also, if I do some throughput benchmark on the RDM disks both in a guest OS and in the ESX OS, I get much much better numbers in the former environment, even both with O_SYNC flag. This is fuddling, because the I/O path from the guest OS should be longer than from ESX, unless ESX treats the devices differently. The devices I do benchmark in the ESX are listed in esxcfg-scsidevs -l.

Thanks!

[mcowger · Accepted solution]

Most modern OS's have buffers for block devices as well as filesystems. At a minimum I know that XP, Linux > 2.4 and Solaris 8+ all do.

It doesn't surprise me that the service console gets poorer performance - its is intentionally limited in what it can do by the ESX kernel, both from a disk IO and networking perspective.

--Matt

VCP, vExpert, Unix Geek

[CrendKing]

Thanks!