ESX 4.1 VAAI and IBM XIV 10.2.4 ... wont work??

[henber]

Hello,

We have had our XIV upgraded to the latest code 10.2.4 that will enable VAAI support. We have followed everything that the http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=102197... .... but still our hardware accelleration info shows either unknown or Not Supported. We are running the ESX 4.1 u1 so we have the latest VIB:s for VAAI support etc.

If have a look in an ESX on the available SATP plugins running

esxcli corestorage claimrule list -c VAAI

We get this output:

Rule Class  Rule   Class    Type    Plugin                 Matches

VAAI        65430  runtime  vendor  VMW_VAAIP_SYMM      vendor=EMC model=SYMMETRIX

VAAI        65430  file     vendor  VMW_VAAIP_SYMM             vendor=EMC model=SYMMETRIX

VAAI        65431  runtime  vendor  VMW_VAAIP_CX               vendor=DGC model=*

VAAI        65431  file     vendor  VMW_VAAIP_CX                      vendor=DGC model=*

VAAI        65432  runtime  vendor  VMW_VAAIP_EQL           vendor=EQLOGIC model=*

VAAI        65432  file     vendor  VMW_VAAIP_EQL                  vendor=EQLOGIC model=*

VAAI        65433  runtime  vendor  VMW_VAAIP_NETAPP  vendor=NETAPP model=*

VAAI        65433  file     vendor  VMW_VAAIP_NETAPP          vendor=NETAPP model=*

VAAI        65434  runtime  vendor  VMW_VAAIP_HDS           vendor=HITACHI model=*

VAAI        65434  file     vendor  VMW_VAAIP_HDS                  vendor=HITACHI model=*

VAAI        65435  runtime  vendor  VMW_VAAIP_LHN           vendor=LEFTHAND model=*

VAAI        65435  file     vendor  VMW_VAAIP_LHN                  vendor=LEFTHAND model=*

and in the info on a datastore it looks like this: (esxcfg-scsidevs -l)

eui.0017380005e200d2

   Device Type: Direct-Access

   Size: 966656 MB

   Display Name: IBM Fibre Channel Disk (eui.0017380005e200d2)

   Multipath Plugin: NMP

   Console Device: /dev/sds

   Devfs Path: /vmfs/devices/disks/eui.0017380005e200d2

  Vendor: IBM       Model: 2810XIV           Revis: 10.2

   SCSI Level: 5  Is Pseudo: false Status: on

   Is RDM Capable: true  Is Removable: false

   Is Local: false

   Other Names:

      vml.01000800003630303035453230304432323831305849

   VAAI Status: unknown

So he knows that it is a XIV and its on Revis:10.2 ?

Do we have to manually install the SATP plugin for vendor:IBM model:XIV  ??? I read an early guide for 3Par wich showed how to do that for their storage...

Does anyone have a good answer for this?

And as I said before.. we have followed the F.A.Q for this so the block size etc is the same for all the datastores and so on.

very greatful for answers.

BR

Henrik

[depping]

You will need to define a claim rule for this yourself. Check out page 112 from this document:

www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf

[dkvello]

It seems that the following claim-rules has been added by default in ESX/ESXi 4.1.U1

But, You can add them like this otherwise:

esxcli nmp satp addrule --satp="VMW_SATP_ALUA" --psp="VMW_PSP_RR" --claim-option="tpgs_on" --vendor="IBM" --model="2810XIV" --description="IBM 2810XIV arrays with ALUA support" 
esxcli nmp satp addrule --satp="VMW_SATP_DEFAULT_AA" --psp="VMW_PSP_RR" --claim-option="tpgs_off" --vendor="IBM" --model="2810XIV" --description="IBM 2810XIV arrays without ALUA support"
esxcli corestorage claimrule load
esxcli corestorage claimrule run

[henber]

Hello, and thank you for your answers.

DKvello I saw that you hade replyed the same on  IBM:s forum, the thread started by my colleague. We have looked for the claim rule after upgrading to U1 but it doesnt seem to work.

We will wait and see if they will release any minor update for ESX for this as AntonyV says:

"The VMWare-certified driver will be available by the end of February"

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=361270&tstart=0

If not we will try adding them manually.

Another question, If we create our own claimrules and then suppose that we get claimrules by a VUM update, can there be a missmatch between these rules then?

BR

Henrik

[dkvello · Accepted solution]

You're right. I did get the message from XIV support that the VAAI support had been held back for a couple of weeks due to an issue with it.

It would also mean that there will be a 10.2.4a or similar of the firmware. We had upgrades planned for 25.February but was asked to wait til 15.March for an updated firmware.

[henber]

Ah thank you for that answer.. really good to know that it have been hold back. We are in the process of upgrading the code on the XIV boxes we have. But if there will be a minor new release we should probably wait for that.

BR

Henrik

[henber]

Anthony Vandewerdt who is  a storage specialist at IBM has written the following post about this.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/anthonyv/entry/xiv_vaai_support_whats_the_...

F Y I

BR

Henrik

[rteglgaa]

Hi there,

Did you get it to work?

Since the VAAI driver for VMware by IBM was released last week, we've installed it on our ESXi servers (successfully), and the XIV is running 10.2.4a firmware.

However, it is still not working. All LUNs says "unsupported" until we try a VAAI operation - then they switch to "Not Supported".

From the ESXi console I can see that the driver and claim rules have been installed and is active, but there's no active devices with VAAI enabled

Anyone else experiencing this?

By the way, the ESXi server are running 4.1 U1 with enterprise plus licenses.

/Rasmus

[henber]

Hello,

yes we got it running yesterday. We are running ESX 4.1 U1 and installed the VAAI driver for ESX. We had already updated our XIV to 10.2.4.a .

We noticed though that IBM release a driver about a week ago, and then they updated it a couple of days ago. The latest driver is from 2011-apr-29 . And you will find it here.

http://www.ibm.com/support/fixcentral/swg/doIdentifyFixes?query.parent=ibm/Storage_Disk&query.produc... Storage System (2810, 2812)&query.release=All&query.platform=All&source=SPE

Here you have the install document on how to install it.

ftp://delivery04-mul.dhe.ibm.com/ecc/hsb/H38200136/IBM_Storage_DD_v1.1.0_for_VMware_VAAI_IG.pdf

After installing the driver on the ESX servers we Entered them in standby mode, and the online again. Then we did a svmotion between our datastores and then it changed to Supported.

I hope you will get it to work because we noticed quite some diffrence. We moved a VM (850 Gb) from one datastore to another. Pre VAAI it took about 2-4 hours.... With VAAI enabled it took about 45 minutes.

Maybe you got the driver they released about 1-1.5 week ago? try the latest one.

BR

Henrik

[rteglgaa]

Hi Henrik,

Thanks for your reply.

I've now tried to re-install the driver with the one from the IBM website - though it appears to be the same as we used before.

The LUNs are still reporting Not Supported after a test migration. They are of the same block size (which is a requirement from VMware for VAAI to work), but still - no luck

When checking if the driver has been loaded (according to the IBM driver install guide), I get the following:

~ # esxcfg-module -l | grep ibm_vaaip_module
ibm_vaaip_module         0    12
~ #

In the example given in the IBM driver install guide the "0" is an "8" - and should be the number of devices on the server. I think this is the problem... but why isn't the driver seeing any devices? :smileyconfused:

/Rasmus

[bjornzak]

Rasmus.

Did you solve this?

Having exactly the same probs as you.

But i use the latest driver 1.2.0 and 10.2.4a

/Bjorn

[dkvello]

Sorry about the delay...

What I had to do was to manually remove all claim-rules I had made pertaining to the "t10" and other rule-attempts to force the recognition so that there was no more XIV/Storwize/SVC related VAAI rules.

I un-installed the 1.1.x ibm_vaai driver.

Then I installed/updated the new 1.2.x ibm_vaai driver and after a reboot i ran the /usr/sbin/claim_ibm.sh

After this,  the XIV was recognized and all (currently three) primitives where active.

The impact on performance was tremendous 🙂

All the snapshot activity that went on during Veeam Backup (SAN based) was all but elliminated (the time it took).

+++

[rteglgaa]

Unfortunately I haven't solved this yet Awaiting a tech who will try and replicate our setup, to see if he can provoke the same issue.

/Rasmus

[dkvello]

hmm, I did have to fiddle a bit with this myself. All seemed fine except for the damned "0".

esxcfg-module -l | grep ibm_vaaip_module ibm_vaaip_module         0    12

All seemed correct, the driver was loaded, all firmware was in place, claim rules seemed ok...

Yet, no claimed devices no mater how many times i ran the claim script.

[rteglgaa]

Hmm, and since we didn't create any manual claim rules, the only thing that you've done different from us, is updating the driver? I will consider doing so. But the release notes doesn't mention any bug fixes.

/Rasmus

[dkvello]

There's allways bugfixes 🙂

Now, what exactly does your list of active claimrules tell You ?

[rteglgaa]

Just updated the driver to the newest version - still no luck

Results from 'esxcli corestorage claimrule list -c VAAI':

~ # esxcli corestorage claimrule list -c VAAI
Rule Class  Rule   Class    Type    Plugin            Matches                                
VAAI        5001   runtime  vendor  IBM_VAAIP_MODULE  vendor=IBM model=2810XIV               
VAAI        5001   file     vendor  IBM_VAAIP_MODULE  vendor=IBM model=2810XIV               
VAAI        5002   runtime  vendor  IBM_VAAIP_MODULE  vendor=IBM model=2145                  
VAAI        5002   file     vendor  IBM_VAAIP_MODULE  vendor=IBM model=2145                  
VAAI        65430  runtime  vendor  VMW_VAAIP_SYMM    vendor=EMC model=SYMMETRIX             
VAAI        65430  file     vendor  VMW_VAAIP_SYMM    vendor=EMC model=SYMMETRIX             
VAAI        65431  runtime  vendor  VMW_VAAIP_CX      vendor=DGC model=*                     
VAAI        65431  file     vendor  VMW_VAAIP_CX      vendor=DGC model=*                     
VAAI        65432  runtime  vendor  VMW_VAAIP_EQL     vendor=EQLOGIC model=*                 
VAAI        65432  file     vendor  VMW_VAAIP_EQL     vendor=EQLOGIC model=*                 
VAAI        65433  runtime  vendor  VMW_VAAIP_NETAPP  vendor=NETAPP model=*                  
VAAI        65433  file     vendor  VMW_VAAIP_NETAPP  vendor=NETAPP model=*                  
VAAI        65434  runtime  vendor  VMW_VAAIP_HDS     vendor=HITACHI model=*                 
VAAI        65434  file     vendor  VMW_VAAIP_HDS     vendor=HITACHI model=*                 
VAAI        65435  runtime  vendor  VMW_VAAIP_LHN     vendor=LEFTHAND model=*                
VAAI        65435  file     vendor  VMW_VAAIP_LHN     vendor=LEFTHAND model=*                
~ #

[bjornzak]

Having a case with IBM and its been escaleted.

I talked to them yesterday.

I removed the 1.2.0 driver.

Rebooted.

Removed the claimrules manually for XIV and SVC.

Rebooted.

Reinstalled driver

rebooted.

Run /usr/sbin/claim_ibm.sh

no differences.

So. I keep you updated if IBM returns to me.

[bjornzak]

Now its fixed.

But i think you will open a case with IBM.

the fix is

the parameter xcopy_allow was set to no, most likely because this system was updated from 10.2.4 to 10.2.4a. Parameter was changed to Yes with TA-Tool by support and now its working as it should

This should have been done by the IBMer installing the code

[henber]

A little update to follow up this thread,

As bjzak wrote the funktion may have to be enabled by IBM support in the XIV.

After that these steps should do the trick on the ESX.

Install the latest IBM vaaip module driver from IBM - Module 268846

Run the command /usr/sbin/claim_ibm.sh to add the claim rules to the ESX

Run esxcli corestorage claimrule list -c VAAI to see if the claim rules are loaded

You can also run esxcfg-module -l | grep ibm_vaaip_module to list this.

after that reboot and then initiate a storage migration or a clone or any other storage related copy task with a file larger than 4 Mb to change the status of the datastore from unknown/unsupported to Supported.

BR

Henrik