It would be nice to have VM appliances available for many of the workloads described in the test. This would reduce the possible variation in the results due to configuration differences within the VMs (patch levels, versions, etc). I realize this might not be possible with the Windows-based VMs, but how about the RHEL VMs? It is probably too much to ask that the vendors in question provide temporary benchmark licenses for such pre-configured appliances in order to provide standardized building blocks for the metrics.
The investment in time to create these VMs is pretty significant and seems to require a bit of cross-platform expertise. Not that this is a bad thing, but it may limit the number of folks able to run the tests.
That being said, I can tell that a lot of time and energy has gone into creating this benchmark -- and document.
My two cents -- thrown in to seed the pot.
Doug
Doug Baer, Solution Architect, Advanced Services, Broadcom | VCDX #019, vExpert 2012-23