VMware extensions - .vib, .json, and the HP P2000

[#1]

Hey all,

The below is a paste from the document I placed on my website. It discusses .vib files, .json files, and presents a MASK_PATH solution for the HP P2000's management LUN.

For the downloads, and updated content, see here:

http://www.lolware.net/lolp2000.html

tl;dr

For those of you who don't know, the HP P2000 produces a "management LUN", operating on LUN 0, which it presents to all hosts within its zone.
I believe this to be the first community created and supported, VIB file for VMware ESXi, outside of firewall management components. It can be applied like a standard VMware patch.
Its purpose is to mask those devices off from VMware.

The HP P2000 - On VMware

The goal of this HP management LUN is that, on a Windows 2003/8 host, you can run HP's various management tools. The reality however is that the majority of users, running VMware, Xenserver or similar are never able to access those tools, and have every management option available to them on the web GUI, or occasionally the CLI.
It's been suggested in several online forums that a VMware best practise is to mask these paths off. I haven't been able to determine if there's a significant risk involved in not doing so - but I certainly find all the extra paths cosmetically annoying.

~ # esxcfg-scsidevs -l naa.600c0ff00013e0670000000000000000 Device Type: Enclosure Svc Dev Size: 0 MB Display Name: HP Fibre Channel Enclosure Svc Dev (naa.600c0ff00013e0670000000000000000) Multipath Plugin: NMP Console Device: /vmfs/devices/genscsi/naa.600c0ff00013e0670000000000000000 Devfs Path: /vmfs/devices/genscsi/naa.600c0ff00013e0670000000000000000 Vendor: HP Model: P2000G3 FC/iSCSI Revis: T230 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: true Is Removable: false Is Local: false Is SSD: false Other Names: vml.020d000000600c0ff00013e0670000000000000000503230303047 VAAI Status: unsupported

Officially..

The closest thing to an official, supported method of doing so is to break into the Technical Support mode (and watch your various alarms trip) and run these commands:

esxcli storage core claimrule add --rule 110 -t location -C 0 -L 0 -P MASK_PATH esxcli storage core claimrule load Rules would be verified in this way: esxcli storage core claimrule list You could revert to the default state like this: esxcli storage core claimrule remove --rule 110

VMware's official documentation on this can be found here:http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100944...

Unofficially

The difficulty I had was that, by the time there was a data center full of servers, managing this sort of process becomes a mess. The next step in automating anything VMware based is PowerCLI. For this purpose though, whilst it would do the job, I thought I could do better. The following article was very helpful. Many thanks Duncan:http://www.yellow-bricks.com/2011/11/29/how-to-create-your-own-vib-file/

Reverse Engineering an existing HP bundle

Following a similar process, instead of starting with a SCSI driver, I started with the HP P2000 VAAI bundle, and reverse engineering this package led to creating my own package.
The file you first download from HP is:

• hp_vaaip_p2000_210.zip
  Unzipping this reveals HP's release notes, and the file hp_vaaip_p2000_offline-bundle-210.zip (the actual patch). This is suitable for checking into VMware's Update Manager.
  
• Using a standard zip application from here, we see a series of metadata, and the vib file Hewlett-Packard_bootbank_vmware-esx-hp_vaaip_p2000_2.1.0-2.vib.
  
• Here we catch up to Duncan. The unix "ar" file, easiest opened with 7-Zip on Windows, contained the file vmware-esx-hp_v.
  
• Now we deviate from Duncan, the file is not a .tgz. It does start with gzip (again, 7-zip opens this) to get vmware-esx-hp_v.vtar. This is however, VMware's proprietry format: This can only be decompressed with VMware's tool:http://www.virtuallyghetto.com/2011/08/how-to-create-and-modify-vgz-vmtar.html
  By copying the .vtar to an ESXi host, using the commands above, I was able to extract vmware.esx-hp_v.tar. As you may expect, this was a tar file. Inside it we see:
  usr/libexec/jumpstart/plugins/hp_vaaip_p2000.json
  

You can download this original HP file here, and see an example of how HP implements custom rules. This in turn points you at the /usr/libexec/jumpstartplugins/*.json files in any ESXi server, which contain a wealth of customisation, VMware configuration. This leads to my file, lolp2000.json, which implements masks appropriate to achieve our goal. However, you can't just dump this on a filesystem. Reasons why are explained here:http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200738...

By generally following the reverse process of breaking down the .vib file, I rebuilt it from the lolp2000.json. There were a few places I deviated from Duncan's guide:

• I chose to stick with a .vtar, even though it's a bit more annoying
  
• Rather than removing the checksums, I generated correct checksums and updated the descriptor accordingly. Although it would be rather hard to corrupt a roughly 300 byte file, I always feel it's safer to use such mechanisms when available.
  
• When installing the .vib file, I found that esxcli would complain about the order of the files within its archive. Specifically, the format seemed to have to be: ar -r filename.vib descriptor.xml sig.pkcs7 test_v
  

Placing the test file earlier in the command, as per Duncan's example, only generated errors like "first file is not descriptor.xml".

But rly... .json

This isn't totally about .vib files, they are documented now. What do we put in one? Well there's a directory full of .json plugins that are basically execute on bootup. Quoting from HP's original:

{ "operations" : { "start" : [ /* Register plugin */ /* use -d <dependency_on_any_other_module_if_any> if necessary */ "storage core plugin registration add -m hp_vaaip_p2000 -N VAAI -P HP_VAAIP_P2000", /* Add the Filter Rule for FC */ "storage core claimrule add --claimrule-class Filter --autoassign --plugin VAAI_FILTER --type vendor --vendor HP --model \"P2000 G3 FC\"", /* Add the VAAI Rule for FC */ "storage core claimrule add --claimrule-class VAAI --autoassign --plugin HP_VAAIP_P2000 --type vendor --vendor HP --model \"P2000 G3 FC\"", ... and so on...

There's a series of operations other than "start" that appear in the default distribution, but this one's the most obvious - it refers to a series of esxcli commands that are executed when the system starts. But when?
Well, it ends this way:

"metadata" : { "version" : 1, "requires" : ["vmkeventd", "vmkernel vmkapi compatibility", "PSA filter pre-claim configuration"], "provides" : ["PSA Filter claim rules", "PSA VAAI claim rules", "storage i/o multipathing"] }

Anything subsequent that "requires" the "PSA Filter claim rules" will execute after this file. What made me stumble was there there is no "dependencies" field, as far as I could see. How do I make a claimrule execute before several other modules, which appear to require the LUNS be operational? The answer is, use an existing "provides".
There are several such modules already in use, which "require" the "path masks" component. So I just made lolp2000 require it:

"provides" : ["path masks"]

Use

I'd encourage the community to test this patch out. Although obviously, any VMware patch can have significant impact, the only file touched on your system is the new lolp2000.json file, and it's easy to audit its contents.
You will of course need to set your host acceptance level to "Community Supported":

esxcli software acceptance set --level CommunitySupported Then apply the patch like so: esxcli software vib install -v /tmp/lolp2000.vib

I'm presently looking into the "Community Packaging Tools" (great product for our needs) and hope to present a standard .zip bundle shortly.

tl;dr

For those of you who don't know, the HP P2000 produces a "management LUN", operating on LUN 0, which it presents to all hosts within its zone.
I believe this to be the first community created and supported, VIB file for VMware ESXi, outside of firewall management components. It can be applied like a standard VMware patch.
Its purpose is to mask those devices off from VMware.

The HP P2000 - On VMware

The goal of this HP management LUN is that, on a Windows 2003/8 host, you can run HP's various management tools. The reality however is that the majority of users, running VMware, Xenserver or similar are never able to access those tools, and have every management option available to them on the web GUI, or occasionally the CLI.
It's been suggested in several online forums that a VMware best practise is to mask these paths off. I haven't been able to determine if there's a significant risk involved in not doing so - but I certainly find all the extra paths cosmetically annoying.

~ # esxcfg-scsidevs -l naa.600c0ff00013e0670000000000000000 Device Type: Enclosure Svc Dev Size: 0 MB Display Name: HP Fibre Channel Enclosure Svc Dev (naa.600c0ff00013e0670000000000000000) Multipath Plugin: NMP Console Device: /vmfs/devices/genscsi/naa.600c0ff00013e0670000000000000000 Devfs Path: /vmfs/devices/genscsi/naa.600c0ff00013e0670000000000000000 Vendor: HP Model: P2000G3 FC/iSCSI Revis: T230 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: true Is Removable: false Is Local: false Is SSD: false Other Names: vml.020d000000600c0ff00013e0670000000000000000503230303047 VAAI Status: unsupported

Officially..

The closest thing to an official, supported method of doing so is to break into the Technical Support mode (and watch your various alarms trip) and run these commands:

esxcli storage core claimrule add --rule 110 -t location -C 0 -L 0 -P MASK_PATH esxcli storage core claimrule load Rules would be verified in this way: esxcli storage core claimrule list You could revert to the default state like this: esxcli storage core claimrule remove --rule 110

VMware's official documentation on this can be found here:http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100944...

Unofficially

The difficulty I had was that, by the time there was a data center full of servers, managing this sort of process becomes a mess. The next step in automating anything VMware based is PowerCLI. For this purpose though, whilst it would do the job, I thought I could do better. The following article was very helpful. Many thanks Duncan:http://www.yellow-bricks.com/2011/11/29/how-to-create-your-own-vib-file/

Reverse Engineering an existing HP bundle

Following a similar process, instead of starting with a SCSI driver, I started with the HP P2000 VAAI bundle, and reverse engineering this package led to creating my own package.
The file you first download from HP is:

• hp_vaaip_p2000_210.zip
  Unzipping this reveals HP's release notes, and the file hp_vaaip_p2000_offline-bundle-210.zip (the actual patch). This is suitable for checking into VMware's Update Manager.
  
• Using a standard zip application from here, we see a series of metadata, and the vib file Hewlett-Packard_bootbank_vmware-esx-hp_vaaip_p2000_2.1.0-2.vib.
  
• Here we catch up to Duncan. The unix "ar" file, easiest opened with 7-Zip on Windows, contained the file vmware-esx-hp_v.
  
• Now we deviate from Duncan, the file is not a .tgz. It does start with gzip (again, 7-zip opens this) to get vmware-esx-hp_v.vtar. This is however, VMware's proprietry format: This can only be decompressed with VMware's tool:http://www.virtuallyghetto.com/2011/08/how-to-create-and-modify-vgz-vmtar.html
  By copying the .vtar to an ESXi host, using the commands above, I was able to extract vmware.esx-hp_v.tar. As you may expect, this was a tar file. Inside it we see:
  usr/libexec/jumpstart/plugins/hp_vaaip_p2000.json
  

You can download this original HP file here, and see an example of how HP implements custom rules. This in turn points you at the /usr/libexec/jumpstartplugins/*.json files in any ESXi server, which contain a wealth of customisation, VMware configuration. This leads to my file, lolp2000.json, which implements masks appropriate to achieve our goal. However, you can't just dump this on a filesystem. Reasons why are explained here:http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200738...

By generally following the reverse process of breaking down the .vib file, I rebuilt it from the lolp2000.json. There were a few places I deviated from Duncan's guide:

• I chose to stick with a .vtar, even though it's a bit more annoying
  
• Rather than removing the checksums, I generated correct checksums and updated the descriptor accordingly. Although it would be rather hard to corrupt a roughly 300 byte file, I always feel it's safer to use such mechanisms when available.
  
• When installing the .vib file, I found that esxcli would complain about the order of the files within its archive. Specifically, the format seemed to have to be: ar -r filename.vib descriptor.xml sig.pkcs7 test_v
  

Placing the test file earlier in the command, as per Duncan's example, only generated errors like "first file is not descriptor.xml".

But rly... .json

This isn't totally about .vib files, they are documented now. What do we put in one? Well there's a directory full of .json plugins that are basically execute on bootup. Quoting from HP's original:

{ "operations" : { "start" : [ /* Register plugin */ /* use -d <dependency_on_any_other_module_if_any> if necessary */ "storage core plugin registration add -m hp_vaaip_p2000 -N VAAI -P HP_VAAIP_P2000", /* Add the Filter Rule for FC */ "storage core claimrule add --claimrule-class Filter --autoassign --plugin VAAI_FILTER --type vendor --vendor HP --model \"P2000 G3 FC\"", /* Add the VAAI Rule for FC */ "storage core claimrule add --claimrule-class VAAI --autoassign --plugin HP_VAAIP_P2000 --type vendor --vendor HP --model \"P2000 G3 FC\"", ... and so on...

There's a series of operations other than "start" that appear in the default distribution, but this one's the most obvious - it refers to a series of esxcli commands that are executed when the system starts. But when?
Well, it ends this way:

"metadata" : { "version" : 1, "requires" : ["vmkeventd", "vmkernel vmkapi compatibility", "PSA filter pre-claim configuration"], "provides" : ["PSA Filter claim rules", "PSA VAAI claim rules", "storage i/o multipathing"] }

Anything subsequent that "requires" the "PSA Filter claim rules" will execute after this file. What made me stumble was there there is no "dependencies" field, as far as I could see. How do I make a claimrule execute before several other modules, which appear to require the LUNS be operational? The answer is, use an existing "provides".
There are several such modules already in use, which "require" the "path masks" component. So I just made lolp2000 require it:

"provides" : ["path masks"]

Use

I'd encourage the community to test this patch out. Although obviously, any VMware patch can have significant impact, the only file touched on your system is the new lolp2000.json file, and it's easy to audit its contents.
You will of course need to set your host acceptance level to "Community Supported":

esxcli software acceptance set --level CommunitySupported Then apply the patch like so: esxcli software vib install -v /tmp/lolp2000.vib

I'm presently looking into the "Community Packaging Tools" (great product for our needs) and hope to present a standard .zip bundle shortly.