Problem with Promise vTrak e610f (NMP errors)

[MichaelLove]

I've been running an ESXi 4 server connected to a Promise vTrak E610f SANS connected via fiber channel and was running fine

After the upgrade to ESXi 5 I've been having problems with my logs filling up with this:

2011-10-11T22:37:29.013Z cpu1:3624)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x12 (0x4124003d4840) to dev "eui.2262000155e8917c" on path "vmhba3:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.Act:EVAL
2011-10-11T22:37:29.013Z cpu1:3624)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "eui.2262000155e8917c" state in doubt; requested fast path state update...
2011-10-11T22:37:29.013Z cpu1:3624)ScsiDeviceIO: 2305: Cmd(0x4124003d4840) 0x12, CmdSN 0x3fa4 to dev "eui.2262000155e8917c" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2011-10-11T22:37:30.021Z cpu1:2049)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "eui.2262000155e8917c" state in doubt; requested fast path state update...
2011-10-11T22:37:30.021Z cpu1:2049)ScsiDeviceIO: 2316: Cmd(0x4124003d4840) 0x12, CmdSN 0x3fa4 to dev "eui.2262000155e8917c" failed H:0x7 D:0x2 P:0x0 Possible sense data: 0x5 0x24 0x0.
2011-10-11T22:37:30.222Z cpu1:2049)ScsiDeviceIO: 2305: Cmd(0x4124003d4840) 0x12, CmdSN 0x3fa4 to dev "eui.2262000155e8917c" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2011-10-11T22:37:30.626Z cpu1:3624)ScsiDeviceIO: 2316: Cmd(0x4124003d4840) 0x12, CmdSN 0x3fa4 to dev "eui.2262000155e8917c" failed H:0x7 D:0x2 P:0x0 Possible sense data: 0x5 0x24 0x0.
2011-10-11T22:37:30.827Z cpu1:2049)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "eui.2262000155e8917c" state in doubt; requested fast path state update...

I've been experimenting with it the last few days and this is the only storage server I'm having a problem with (tried it with two Sun x4500 servers that are serving drives over FC with COMSTAR). However, no other computer is having a problem with the vTrak, only VMWare.

I only get the above errors when writing to the datastore, not reading.

If I pass the disk through to a virtual machine as a raw disk I can read and write to it without any errors and nothing in my vmkernel.log file.

I've tried combinations of different controllers and cards and pathing options. I've changed several settings on the vTrak with no effect. I've updated the firmware in my FC cards with no change.

I've read posts from others with similar error messages, but haven't been able to resolve it.

The vTrak is running the latest firmware available, too.

Any thoughts?

[VDBG]

I did the work for you and found out what the release notes were by calling up support. The Apple Beta 10.07 is apparently the same release notes as Channel firmware 3.34.

To make things easy. I've attached it for you.

Do you know if people have succesfully used firmware 3.34 with ESXi 5.x?

[dexterous]

3.34 is pretty old (more than two years now). I wouldn't run vSphere 5 with anything older than 3.36. The key here is to disable VAAI in ESXi 5 which can be done on the command line using these three commands (do them one at a time):

To Disable VAAI in ESXi 5.X

esxcli system settings advanced set --int-value 0 --option /DataMover/HardwareAcceleratedMove
esxcli system settings advanced set --int-value 0 --option /DataMover/HardwareAcceleratedInit
esxcli system settings advanced set --int-value 0 --option /VMFS3/HardwareAcceleratedLocking

[jwestlake]

Hi,

We're needing to do something similar here with our Apple VTrak E/J610s and VMware 5.1... the Promise support guy I got assigned to says there's no way to do this, upgrade or ESX 5.1.  I don't understand why, if it's the same physical hardware, it can't be flashed to the latest firmware... wouldn't that just overwrite the Apple-specific stuff and be done with it?

Is there no other way to get these boxen to talk to ESX 5.1?

Out of curiosity, what would happen if you did flash the latest non-Apple firmware onto the Apple VTrak?  Does that brick it?

[avguidry]

dexterous,

I've been looking for this answer for some time.  So one question, does only the subsystem chassis need to be replaced or the jbod chasis also?

thanks

[dexterous]

It doesn't brick it but it wont boot which will require you to go back to the old firmware. If you do what you're describing (flashing the wrong firmware), the controllers will go into maintenance mode if they detect a firmware / serial number mismatch.

[dexterous]

I dont know the answer to that but its entirely possible that it will work. I would suggest that you do the following:

Follow my instructions to flash the controllers in the E-Class to channel SKU firmware then get the unit totally up to date with the latest firmware.

Shut everything off, plug in the J-Class unit and see if everything boots. If it does boot, re-apply the latest firmware to the E-Class now that the J-Class has been plugged in. Note that failing to re-apply the firmware after the J-Class has been plugged in will result in the SAS expanders in the J-Class NOT being upgraded to the latest firmware.

[dexterous]

By the way, Promise hasn't released firmware for the Channel version of the Ex10 series for more than a year now. Its likely that these units are very very close to reaching end of life / end of support status with promise. Either way, trust me when I tell you that when it comes to VMware, there are MUCH better options on the market than any Promise RAID solution. The company I work for now sells JetStor brand storage made by Advanced Computer and Network Corp - http://www.acnc.com and I have found that even their cheapest stuff is at least 5x faster. My favorite JetStor chassis is the 724iFD which is similar to the Promise Ex30 series only MUCH better (it's got VAAI support!). Before you go spending $2500 to convert your very old Promise unit, please consider spending a few more bucks on a new JetStor unit, you will be much happier.

[ealaqqad]

Just in case people still hit this post, I have documented the full story of ATS-Only VMFS datastores at:

http://www.virtualizationteam.com/virtualization-vmware/vsphere-virtualization-vmware/ats-only-vmfs-...

Hope that help,

Eiad

[EDSI]

Dexterous,

We wanted to see if you would help me decide if we can get any configuration working on our system.

We have 1 Promise Vtrak E610F with the Apple Firmware, 3 Dell ESXi 5.1 servers that we are trying to get some extra storage on with this Promise.

[zebity2]

Hi Dexterous,

thank you for the very useful posting on Promise e-class and Mac OS firmware.

I have a Promise e-class VTRAK RAID box (16 disk with dual fibre channel controllers - E610fD), but it was dropped and now the lower 4 drive bays do not function (all ways get disk failures for disks in these slots, even through if you put disk in other slot they report as ok).

My question is about what type of mix and max can you to across Promise chassis and controllers. Purchasing a replacement e-class chassis is a very expensive option, so I would rather purchase a much cheaper p-class or j-class and then swap the good controllers from my existing E610fD into the replacement chassis.

If I then followed your firmware update procedure would I then be able to get my Mac Promise controller to work in Non Mac and Non e-class chassis or will controller firmware enforce on "real e-class" being able to be used with the controllers.

I actually have my E610fD connected to a Mac but as I am not using XSAN I do not believe there is any advantage in using the Mac firmware.

Thank you in advance for advise.

Cheers,

Zebity

(Promise E610f with Brocade 200e switch and Xserve Early 2009)