cannot connect ESX box to HP MSA 1000

[toofaan]

I have 2 ESX servers. Both these servers are connected to a HP MSA 1000 (this is our mini san)

On connecting to one of the box from the vi console, I see the server and 3 guest under it.

On creating a new VM, I can connect to a datastore. There is connectivity from this box to the array.

On connecting to the 2nd ESX server via the VI Client, all the guests show as unavailable.

I cannot restart or do anything with these guests.

Also, on trying to add a new VM, I see nothing in the datastore screen. (next becomes disabled)

Is this an issue with conectivity between the ESX server (in whihc the guest are not available) and the HP Array?

What commands can I use to tets connectivity to a datastore from a ESX server and check if datastores are available?

[SuryaVMware]

It looks like you are not able to see the LUNs on the Array or the Physical Connectivity is lost.

Run the command esxcfg-vmhbadevs -q if you dont see your SAN LUNs try run esxcfg-rescan vmhbaX (where X is the HBA number of the FC cards)

If you still don't see the SAN LUNs. Past the last few lines from your /var/log/vmkernel here.

-Surya

[toofaan]

Surya,

Our VM servers are part of serers in a blade. We have about 6 Windows Servers and 2 ESX Servers in the Blade .

We have fiber connections connected (not from each individual server) but from the blade chasis to the HP Mass Storage Array 1000.

The array, LUNs and the ESX servers were configured before I joined this new place.

Is it possible that the connectivity from server to array is lost only for one server (server on which we cannot see the guests),?

because we can see the guests hosted on the other VM Server (good server) - that means other server can see and connect to the LUNs.

I have an engineer from HP coming to look at the array controller and upgrade the firmware.

But I dont think that can be a reason for the loss of connectivity from only one of the ESX servers.

To try your solution, I am thinking these will be the steps that I will have to take:

- Connect to the console of the bad server (ESX server on which we are not seeing the guests)

- login as root

- run your commands

- What should I keep my eye for? (I am not an expert at interpreting the linux responses

I will let you know the results after running the commands

thank you,

[toofaan]

Surya,

I ran the commands:

esxcfg-rescan vmhba1 and esxcfg-rescan vmhba2

it showed: On scsi0, removing and then On scsi0, adding

On the other command: esxcfg-vmhbadevs -q, it showed:

vmhba0:0:0 /dev/cciss/c0d0

not sure what to interpret from those responses

Also, How to export the logs?

I am very new to the ESX Linux world.

[toofaan]

Last 4 Lines from: tail -f /var/log/messages

Setting network parameters: succeeded

bringing up loopback interface: succeeded

SIOCSIFHWADDR: Device or resource busy

Bringing up interface vswif0: succeeded

[SuryaVMware]

With the esxcfg-vmhbadevs -q output i am certain that you are not seeing the LUNs. Originally I was suspecting that you could be seeing the LUNs but not the VMFs volumes.

Since this is a blade in chase we can rule out the connectivity issue. However, I am sure you are connecting the Blade center throught the fibre switch.

Did you check the zonning? when did the Array firmware upgraded? before noticing the problem or after.

-Surya

[mdaubaras]

I'm not sure if this is helpful or not, but recently we experienced an HBA failure on an ESX server that was attached to an MSA1500 via a SAN fabric. There is a specific "connection" configured on the MSA1500 separate from SAN zoning at the fabric level that must be configured for each connection from ESX host to MSA. The only method for performing this configuration is at the command line interface. There should be a console connection on the MSA1000 (which I believe is very similar to and MSA1500) that can be used for this purpose. Not only does the connection need to be configured with the correct profile (linux), but the server needs access granted to each LUN. This sounds more complicated that it is. Your first step should be to get your hands on the manuals from HP for the MSA1000 and get console access to the MSA so you can see what is going on.

[toofaan]

The system board and the power was updated in the last 2 days .

Could this have changed something? THE HBA cards stayed the same.

We have an engineer coming in right now to upgrade the firmware on the array controller.

I am not sure how to check the zoning on the HP Array controller?

Can the LUN's be re-created for the guests in concern? Would that affect the guests and LUNs for those guests on the other good ESX server?

[toofaan]

Surya,

We have an HP guy coming to connect to that console. He is going to upgrade the firmware of the box. Please do let me know what exactly should I be looking for on the HP console?

I really really need to get those VMs hosted on the ESX server up and available. I am being thrown in this environment that was setup by someone in 2006. If there is any help and support that you can provide, please keep me updated...with the next steps. If therre is a direct number ot reach you, please do let me k now..

I thank you a lot.

[SuryaVMware]

well one thing can be upgrade the firmware on the blade. If you are comfortable doing it by yourself you could download the FW840.2009_0209.17.iso from HP site and boot the blade with the ISO connected VIA iLO

A couple of reboots will complet the firmware upgrade of your hardware to the latest version. If it is a firmware issue that you are not able to get connected to the array. It should be resolved with this.

Let me know how it goes. I can further troubleshoot this issue with you.

-Surya

[toofaan]

what is the way to find the current version of the firmware on this ESX server with no connectivity to the guests?

[SuryaVMware]

Well since there is no connectivity you can shutdown the ESX server and verify the BIOS version of the server, FC card and iLO versions manually or boot from the FW ISO image via iLO before it proceed with upgrade it will list the current versions.

However, the upgrade process is not automatic. You can choose not to proceed with the upgrade once the necessary information is displayed. or you can proceed with the upgrade process for selective hardware.

-Surya

[Texiwill]

Hello,

I used a MSA 1000 SAN for years. You need to login to the COnsole or from a non-ESX host access the cpqarray software and setup the connections. The Connection Type for ESX will be Linux and not WIndows. Newer versions of the firmware may have an ESX setting. Once you do that then you need to look at your ACLs on the SAN. If there are ANY ACLs set you need to add some specifically for your ESX LUNs.

Once you do that, it just works. Connection Type is very important however.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[toofaan]

All,

Thank you for the responses.

The HP guys came and they could not detect a heartbeat from the MSA 1000 unit. They asked us to verify if we really had connectivity to the unit.

We used VI client to connect to the ESX box (in which we could see the guests), but when we clicked on the Host server, clicked on one of the guest and went to the console tab, it gave an error that configuration file could not be seen, and the console for those guest servers was not coming up.

As that point we realized that we really did not have any connectivity to the unit. The HP folks rebooted the unit, and voila we could get into the consoles.

So, I guess the unit was down and that needed to be reset. The HP guys did not upgrade the firmware of the unit as they wanted to make sure that there are no compatibilty issues with the HBA card firmware and the firware of the MSA 1000 box.

One wierd thing is that this MSA 1000 has 2 controllers, but one of the controller was never connected to the box. It is kept unseated in the MSA1000. So, it appears there is no array redundancy (Active-passive, active-active) setup. That is something I would like to do moving forward.

At this point we can connect via Web, VI and ILO to the two ESX server boxes and yes we also had one Windows box connected (this was never conveyed to me), so now from the windows box, I started the Compaq array utility and I can see the raid configurations.

Oh boy! IF anyone of you can guide me towards a nice guide for configuring and connections between MSA1000 SAN and ESX hosts it would be wonderful.

Another question: I noticed that the version of ESX is 3.0 and VI client is 2.0. Also when I connect via the VI client , I only see the inventory tab and adminstration, I do not see options like cluster options and also enabling the plug-ins for VMotion, and VMupdate manager. when I connect via the VI client, I login as the root account.

Should I be upgrading the infrastructure of the ESX to 3.5 and the VI Client to 2.5 to enable all those options?

Thank you, for all your help

we could connect

[Texiwill · Accepted solution]

Hello,

So, I guess the unit was down and that needed to be reset. The HP guys did not upgrade the firmware of the unit as they wanted to make sure that there are no compatibilty issues with the HBA card firmware and the firware of the MSA 1000 box.

Interesting I never had to reboot my Unit to get connectivity. I would upgrade to the Active/Active firmware.

One wierd thing is that this MSA 1000 has 2 controllers, but one of the controller was never connected to the box. It is kept unseated in the MSA1000. So, it appears there is no array redundancy (Active-passive, active-active) setup. That is something I would like to do moving forward.

Just insert it. Make sure however that the primary HBA port on each host is connected to the primary controller within the MSA. If they are not you get flip flopping unless you are using the Active/Active firmware.

At this point we can connect via Web, VI and ILO to the two ESX server boxes and yes we also had one Windows box connected (this was never conveyed to me), so now from the windows box, I started the Compaq array utility and I can see the raid configurations.

That is the way to configure the MSA. You can not do it from ESX or the ILO, only a separate Windows Box with qlogic HBAs or the Serial Cable. WHich I generally connected to my ESX host so I could control things without the need for a separate windows box.

Oh boy! IF anyone of you can guide me towards a nice guide for configuring and connections between MSA1000 SAN and ESX hosts it would be wonderful.

HP should have one.

Another question: I noticed that the version of ESX is 3.0 and VI client is 2.0. Also when I connect via the VI client , I only see the inventory tab and adminstration, I do not see options like cluster options and also enabling the plug-ins for VMotion, and VMupdate manager. when I connect via the VI client, I login as the root account.

Should I be upgrading the infrastructure of the ESX to 3.5 and the VI Client to 2.5 to enable all those options?

Those options exist in 3.0 as well as 3.5. Not really sure what you are asking here. But I think this is a separate issue and should have its own thread.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[toofaan]

Thank you