11 Replies Latest reply on Oct 12, 2011 7:47 AM by JohnLambie

    FC HBA's Losing connectivy after updating

    manfriday Hot Shot

      Hi,

       

      So last Friday I applied patch ESXi410-201107401-BG via the update manager.

      This brought my ESXi hosts up to build 433742.

       

      Since then, I have been periodically losing HBA paths.

       

      My servers are HP DL580 G5's, with ISP2532 8gb HBA's connected to Compellent FC storage.

      Everything had been running fine for the past couple years. The patch I installed is the only change that I made.

       

      So, if I look in configuration->Storage adapters_>vmhba2 I can see the WWN of the adapter that fails.

      It is something like 50:01:43:etc.. etc.

      All of the ISP2532 WWN's start with a 50:01..

       

      When the HBA dropps connection, i still see this same WWN via the VI Client.

      However, if I ssh into my esxi box and do a

       

      cat /proc/iscsi/qla2xxx/6

       

      I can see that the WWN has changed to 21:00:00 etc...

       

      I am not sure where this WWN is coming from.

      Rebooting the host restores connectivity and the card assumes the correct WWN.

       

      Any ideas?

       

      Thanks guys

       

      Jason

        • 1. Re: FC HBA's Losing connectivy after updating
          JohnLambie Lurker

          Jason,

           

          We are experiencing the exact same issue as you. We are running IBM x3580's with Qlogic 2562 HBA's and upgraded to 433742 a couple weeks ago. Since then our Host's sporadically lose their connection to the LUN's (they show up as "inactive" in vCenter). If we happen to be at work when it happens we can put the machine in maintenance mode and rescan the LUN's and they come back right away. If not, the guests slowly drop off and the only solution is to hard boot the Host. We have opened a ticket with VMware, nothing so far except "make sure the HBA's firmware is the latest". We are in the process of upgraded the firmware right now...I'll let you know what becomes of the open ticket.

           

          John

          • 2. Re: FC HBA's Losing connectivy after updating
            RParker Guru

            So, if I look in configuration->Storage adapters_>vmhba2 I can see the WWN of the adapter that fails.

            It is something like 50:01:43:etc.. etc.

            All of the ISP2532 WWN's start with a 50:01..

             

            When the HBA dropps connection, i still see this same WWN via the VI Client.

            However, if I ssh into my esxi box and do a

             

            cat /proc/iscsi/qla2xxx/6

             

            I can see that the WWN has changed to 21:00:00 etc...

             

            I am not sure where this WWN is coming from.

            Rebooting the host restores connectivity and the card assumes the correct WWN.

             

            Any ideas?

             

            WWN is assigned by vendor.  Emulex starts with 10:00, Qlogic starts with 21:00, and Brocade starts with 28:00.  If you have Qlogic, assuming the qla2xxx path and 21:00 that should be the SAME WWN it's always been..

             

            you can have Virtual nodes on a Fiber HBA, but it SHOULD be zoned as the REAL WWN.  There is a WWPN (port) and WWNN (node).  21: is the port, 20: is the node.  port is the sender and node is the destination (something close to that, I may have them reversed).

             

            So if you are using vmhba2 that is the port assigned by VM Ware because you are using a VIRTUAL port rather than the actual adapter, that's why it's changed.  I would zone the adapter 21:00:00 and rescan the storage adapter rather than virtual, and you won't have this problem anymore.

             

            Yes, I know it's been 2 years.. but as you see virtual ports CAN and DO change.

            • 3. Re: FC HBA's Losing connectivy after updating
              jamesrico Novice

              Hi,

               

              We've got the same problem. Running DL580 G7s with the HP branded QLogic HBA cards ISP2532

               

              I have checked and the physical cards have WWN's starting 50:01 printed on labels.

               

              Got a call logged with HP but I think I may have to remove the patch.

               

              Isn't actually causing a problem though, it just causes the server to loose storage resiliancy for about 1 min every few days. The two broken paths reconnect without having to reboot the server.

              • 4. Re: FC HBA's Losing connectivy after updating
                JohnLambie Lurker

                We upgraded all HBA's firmware last week and it happened again early Saturday morning, one host lost connection to all it's LUN's. All guests were effected. VMware sent us a link to the latest Qlogic driver (841.k1.16.2-1vmw) which we arre applying now.

                 

                http://downloads.vmware.com/d/details/dt_esxi4x_qlogic_qla2xxx_841k12811vmw/dHdlYnQqaGhiZGhwZA==

                 

                Keep you posted.

                 

                John

                • 5. Re: FC HBA's Losing connectivy after updating
                  jamesrico Novice

                  Great, be useful to know how you get on.

                   

                  Just checked our firmware:

                   

                  50014380029D62E1

                  Description:FC HBA Option ROM detail for QLogic HPAK344A
                  Version:v.2.15
                  Manufacturer:QLogic Corporation
                  Type:FC HBA Option ROM

                   

                  50014380029D62E1

                  Description:FC HBA Firmware details for HPAK344A
                  Version:v. 5.02.00
                  Manufacturer:QLogic Corporation
                  Type:FC HBA Firmware

                   

                  50014380029D1D81

                  Description:FC HBA Firmware details for HPAK344A
                  Version:v. 5.02.00
                  Manufacturer:QLogic Corporation
                  Type:FC HBA Firmware

                   

                  50014380029D1D81

                  Description:FC HBA Option ROM detail for QLogic HPAK344A
                  Version:v.2.15
                  Manufacturer:QLogic Corporation
                  Type:FC HBA Option ROM
                  • 6. Re: FC HBA's Losing connectivy after updating
                    RParker Guru

                    A firmware update won't affect the connectivity if we are talking about a path problem on the host, 2 different issues.  One is the port reported by VM ware as 50:00 which is virtual vs the actually WWN of the HBA card, the firmware ONLY matters if you can't connect at all, it won't all of a sudden stop connecting after working for a time, that's not firmware.  You have some other problem.

                    • 7. Re: FC HBA's Losing connectivy after updating
                      jamesrico Novice

                      Our HBA cards are HP branded and have stickers on the physical cards with WWNs starting 50:00, its the same WWN as configured in HP CV and displayed in the vi client.

                      • 8. Re: FC HBA's Losing connectivy after updating
                        JohnLambie Lurker

                        We don't have a pathing problem, this was a stable environment (for years) UNTIL we updated to patch 433732.

                        • 9. Re: FC HBA's Losing connectivy after updating
                          JohnLambie Lurker

                          It's been 9 days since we updated the Qlogic drivers to 841.k1.16.2-1vmw and we've had no disconnects since. I've closed our ticket with VMware and hopefully this will solve the problem for good..

                           

                          John

                          • 10. Re: FC HBA's Losing connectivy after updating
                            jamesrico Novice

                            Interesting, thanks John.

                             

                            I rolled our servers back to the previous build (381591) on Saturday using the very useful shift-r option (http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.upgrade.doc_40/vhuu_help/t_roll_back_an_esxi_update_patch_or_upgrade.html)

                             

                            All servers have been fine since.

                             

                            I will probably leave it another week then re-apply the patch and use the QLogic update.

                             

                            James

                            • 11. Re: FC HBA's Losing connectivy after updating
                              JohnLambie Lurker

                              Good luck to you James.. I’ll let you know if we lose our LUN’s again.

                               

                              John