Intel P3700 for VSAN

[lbcis]

I was in the middle of designing a VMware VDI cluster and will likely use VSAN.

I just saw Intel releasing their new P3700 PCI-E card last week. I know the S3700 is on the HCL --- any reason to think the P3700 would not be (now if not eventually?) Its too new for me to find anything concrete.

I was hoping to drop some in a group of HP DL380p servers and call it a day. Seems like a great card for this type of solution.

Thoughts?

[depping]

lbcis wrote:

I was in the middle of designing a VMware VDI cluster and will likely use VSAN.

I just saw Intel releasing their new P3700 PCI-E card last week. I know the S3700 is on the HCL --- any reason to think the P3700 would not be (now if not eventually?) Its too new for me to find anything concrete.

I was hoping to drop some in a group of HP DL380p servers and call it a day. Seems like a great card for this type of solution.

Thoughts?

I knew the P3700 was about to be released, and I have been told that Intel is looking to qualify this for VSAN. However I cannot guarantee ANYTHING, so this is still a risk.

[JohnNicholson]

One of the Intel Flash guys made a cryptic comment to me a few months back about not bothering with the 910 and waiting for something better for VSAN if I wanted to use a PCI-Express flash device.  I suspect this thing should be tested shortly, but as Duncan warns its best to wait for official support.

I have been using the S3700 based with great performance for VDI.  (Nothing like doing recomposes of pools without users noticing an impact).

[depping]

I have asked the folks responsible for the HCL what the status is.... more soon!

[lbcis]

I appreciate the updates! I'm have a quote for a few of these and would like to order asap. HCL seems like a matter of when not if.

[depping]

lbcis wrote:

I appreciate the updates! I'm have a quote for a few of these and would like to order asap. HCL seems like a matter of when not if.

I would hold off for now. I have been told that NVMe hasn't gone through certification/testing even. Not sure when this will happen, waiting for a reply on that.

[arielsanchezmo1]

I think everyone in the process of buying new servers is seriously considering NVMe dirves like the Intel P3700's (for vsan, flash read cache, or solutions like Pernix). Please prioritize this support - the servers and drives are already coming out, and there is an open source driver out http://nvmevmklinuxdriver.sourceforge.net/.

Windows 2012 R2 and Redhat already officially support it.

I would at least tell people how much time the process is expected to take - I am delaying purchases for my 3-year hardware lifecycle renewal because I want NVMe.

[dosle]

I agree with arielsanchezmor. I'm checking the HCL every morning looking for updates to Storage and I/O Controllers.

[davidmirror]

I confirm that the driver that VMware released for NVMe (VMware vSphere 5: Private Cloud Computing, Server and Data Center Virtualization) do the trick for Intel P3700 but is only usable for a regular VMFS datastore, because VSAN (at least in our environment) does not recognize the device for the SSD role in the diskgroup.

[CHogan]

Indeed - as I mentioned in a previous post, you need to check the VSAN VCG for supportability - VMware Compatibility Guide: vsan

Just because a device appears on the vSphere HCL doesn't mean that VSAN automatically supports it.

HTH

Cormac

[Khoen]

Are you sure of that ? Intel is communicating about "Experimental" VSAN test configurations based on P3700 : http://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/intel-vmware-scalable-en...

[crosdorff]

Cormac says it's not on the HCL yet and thus not supported.

He's not saying it's not working.

With an available drivers there is big change it is working, but no support as long it's not on the HCL.

[Khoen]

I know that

I was answering to David, who said that the P3700 is only usable as VMFS datastore

[davidmirror]

Hi

Well yes, i'm sure and none of the reference architectures in the doc actually use the P3700 for VSAN but the S3700.

We were able to create a VMFS-3 datastore using the P3700 as extent and then upgrade it to VMFS-5. Clearly one of the reason it's still not supported yet but we are highly interested in use it for VSAN as soon as it is in the vSphere+VSAN HCL.

Thank you

[Khoen]

Hi,

Excuse me but the "High Performance Experimental" use the P3700 as the SSD cache ...

[davidmirror]

Right, my fault.

Still, at least in our environment which is no exactly the same as the Intel's experimental, it is not recognized by VSAN.

Thanks.

[markartinf]

Can anyone provide guidance on how to compile the drivers found here? SourceForge.net: nvme vmklinux driver - Project Web Hosting - Open Source Software …

I happen to be sitting on a handful of P3700 400GB cards and fancy loading vSAN in the lab.

[JCunningham53]

Intel P3700/3600 VMware drivers now available :

Intel® Download Center

No direct mention of VSan.......

Jeff

[arielsanchezmo1]

great link! nice to see 5.5 and 6.0 supported. I wonder if this is the same driver 6.0 GA is shipping with...

i'm buying some of these and once i have them, i'll paste what the 6.0 GA CD gives them in "excli storage core adapter list" and for driver version and PCI device IDs (like in VMware KB: Determining Network/Storage firmware and driver version in ESXi/ESX 4.x, ESXi 5.x and ESX...). I know that the cards differ if bought through Dell or Intel directly, so curious to see what everyone is having best results with.

Or, if someone already has these, can you put it here? For example, for the Micron p420m (bought through Dell)

vmhba1    mtip32xx_native  link-n/a    pscsi.vmhba1                              (0:65:0.0) Micron RealSSD P420m PCIe Controller

~ # vmkload_mod -s mtip32xx_native

vmkload_mod module information

input file: /usr/lib/vmware/vmkmod/mtip32xx_native

Version: 3.9.4-1OEM.550.0.0.1331820

License: BSD

Required name-spaces:

  com.vmware.vmkapi#v2_2_0_0

Parameters:

  atapassthru: int

    This parameter is DEPRECATED !!

  mtipPcieMaxPayload: int

    MaxPayload 0=128 1=256 2=512

~ # vmkchdev -l | grep vmhba1

0000:41:00.0 1344:5161 1344:2200 vmkernel vmhba1

which in the HCL would take you to

Brand Name	Model	Device Type	Supported Releases
DELL	RealSSD P420m PCIe Flash Drive 700GB	SCSI	ESXi 6.0 ESXi 5.5 U2 ESXi 5.5 U1 ESXi 5.5

[elerium]

I'm running a new VSAN 6 cluster using the Intel P3700 1.6TB. It's still not on HCL but I had no problems getting it detected and working on ESXi 6.0.0. Performance is better than I expected, am using these for Hybrid deployment and am seeing 10.5k IOPS on each host. For the driver I am just using the inbox NVME driver (nvme (1.0e.0.35-1vmw.600.0.0.2494585) that ships with ESXi 6.0.0. I did flash the SSDs to latest firmware (8DV10131) using Intel DCT. Haven't tried using the Intel driver since I'm happy with existing performance/stability. Will be nice to get "official" HCL support for this card from VMWare but it's working fine for me.

4x Dell r730xd ESXi 6.0.0 each with:

384GB RAM, H730 mini on RAID0, Disk Group:  1x Intel DC P3700 1.6TB + 7x WD RE4 4TB SAS, 10 Gb NICs

Switch is Cisco 6500.

esxcli storage core adapter list:

vmhba1    nvme         link-n/a    pscsi.vmhba1                        (0000:81:00.0) Intel Corporation <class> Non-Volatile memory controller

vmkchdev -l | grep vmhba1:

0000:81:00.0 8086:0953 8086:3702 vmkernel vmhba1