10 Replies Latest reply on May 14, 2014 12:56 PM by MaciekTX

    iSCSI LUNs Connected but Datastore Not Visible/Available

    uninspired Novice

      Hey all-

       

      We've got a cluster consisting of 3 ESXi hosts. All three are configured indentically, with three BroadCom ports all connecting to an Equallogic array with 8 datastores (7x 2TB LUNs for VMs, and 1x LUN for ISOs) via software iSCSI adapter (using Dell multi-pathing module). I installed the latest firmware and tools updates (from the end of January) today, and now two of the datastores are missing (not that it matters, but disks 1 and 5). The LUNs are visible on the Static Discovery tab of the iSCSI Software Adapter properties page (and in the Details pane in Configuration > Storage Adapters when the iSCSI Software Adapter is selected), and the Equallogic shows active connections from all three IP addresses.

       

      I've tried removing all of the datastores and re-discovering them all, I've tried rescanning the adapters, and I've tried Add Storage (Configuration > Storage), but I can't get the datastores to appear in the inventory.

       

      Any thoughts on how to get the datastore to re-appear?

       

      Thanks

      -Chris

        • 1. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
          AndySimmons Hot Shot

          Any chance a rogue Windows server stomped on your VMFS volume? If you can see the LUN, but the datastore is gone, the VMFS is likely wiped out.

           

          That doesn't mean you can't recover it, sometimes you just need to change the partition type back to VMFS. See this post. You'll need to enable the shell if you haven't done so already, but this should still work in ESXi 5.

          • 2. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
            uninspired Novice

            Thanks for the input. I forgot about a crucial piece of information: The other two ESXi hosts still see the datastore and there are active, operating, VMs on that datastore. It's only the single host (that was just updated) not seeing the datastore. So, I know the integrity of the vmdk is solid, it's just not popping into the inventory. And, of course, now I can't migrate any VMs on that datastore to that host.

             

            So, long story long, the datastore is not gone, I just can't get this host to recognize it and add it to inventory.

             

            Thanks again

            -Chris

            • 3. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
              christianZ Virtuoso

              Have you check the vmkernel log file for any errors/warnings?

               

              Reg

              Christian

              • 4. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
                AndySimmons Hot Shot

                Just to be clear, if the partition type is inadvertently changed, any ESXi hosts that had already identified that datastore as VMFS will continue to be able to see that datastore. However, if those hosts are rebooted, they will no longer be able to see them.

                 

                If you check with fdisk -l from the problematic ESXi host, that will tell you for sure.

                1 person found this helpful
                • 5. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
                  uninspired Novice

                  The results of the fdisk show something different about the two affected volumes, but I'm unable to interpret.

                   


                  ~ # fdisk -l

                   

                  Disk /dev/disks/naa.6090a04870aeb18352d28456f5771e0b: 2199.0 GB, 2199036887040 bytes
                  255 heads, 63 sectors/track, 267351 cylinders
                  Units = cylinders of 16065 * 512 = 8225280 bytes

                   

                                                             Device Boot      Start     End      Blocks  Id System
                  /dev/disks/naa.6090a04870aeb18352d28456f5771e0bp1             1         2     13 195+  fb  VMFS

                   

                  Disk /dev/disks/naa.6090a04870aea1d636d2a445b577ee74: 2199.0 GB, 2199036887040 bytes
                  255 heads, 63 sectors/track, 267351 cylinders
                  Units = cylinders of 16065 * 512 = 8225280 bytes

                   

                                                             Device Boot      Start     End      Blocks  Id System
                  /dev/disks/naa.6090a04870aea1d636d2a445b577ee74p1             1         2     13 195+  fb  VMFS
                  /dev/disks/naa.6090a04870aea1d636d2a445b577ee74p2             3    267351 2147480779+  fb  VMFS

                   

                  Disk /dev/disks/naa.6090a04870aea14371d4746b387b0e02: 2199.0 GB, 2199036887040 bytes
                  255 heads, 63 sectors/track, 267351 cylinders
                  Units = cylinders of 16065 * 512 = 8225280 bytes

                   

                                                             Device Boot      Start     End      Blocks  Id System
                  /dev/disks/naa.6090a04870aea14371d4746b387b0e02p1             1         2     13 195+  fb  VMFS
                  /dev/disks/naa.6090a04870aea14371d4746b387b0e02p2             3    267351 2147480779+  fb  VMFS

                   

                  Disk /dev/disks/naa.6090a04870ae81f7e3d224b8f378eeb3: 2199.0 GB, 2199036887040 bytes
                  255 heads, 63 sectors/track, 267351 cylinders
                  Units = cylinders of 16065 * 512 = 8225280 bytes

                   

                                                             Device Boot      Start     End      Blocks  Id System
                  /dev/disks/naa.6090a04870ae81f7e3d224b8f378eeb3p1             1         2     13 195+  fb  VMFS
                  /dev/disks/naa.6090a04870ae81f7e3d224b8f378eeb3p2             3    267351 2147480779+  fb  VMFS

                   

                  Disk /dev/disks/naa.6090a04870ae410990d2343a47783e53: 1099.5 GB, 1099526307840 bytes
                  255 heads, 63 sectors/track, 133676 cylinders
                  Units = cylinders of 16065 * 512 = 8225280 bytes

                   

                                                             Device Boot      Start     End      Blocks  Id System
                  /dev/disks/naa.6090a04870ae410990d2343a47783e53p1             1    133676 1073752406   fb  VMFS

                   

                  Disk /dev/disks/naa.6090a04870aea1f7b2e1742bef93ce2c: 2199.0 GB, 2199036887040 bytes
                  255 heads, 63 sectors/track, 267351 cylinders
                  Units = cylinders of 16065 * 512 = 8225280 bytes

                   

                                                             Device Boot      Start     End      Blocks  Id System
                  /dev/disks/naa.6090a04870aea1f7b2e1742bef93ce2cp1             1         2     13 195+  fb  VMFS
                  /dev/disks/naa.6090a04870aea1f7b2e1742bef93ce2cp2             3    267351 2147480779+  fb  VMFS

                   

                  Disk /dev/disks/naa.6090a04870ae710493e0f46d3c923e1d: 2199.0 GB, 2199036887040 bytes
                  255 heads, 63 sectors/track, 267351 cylinders
                  Units = cylinders of 16065 * 512 = 8225280 bytes

                   

                                                             Device Boot      Start     End      Blocks  Id System
                  /dev/disks/naa.6090a04870ae710493e0f46d3c923e1dp1             1         2     13 195+  fb  VMFS
                  /dev/disks/naa.6090a04870ae710493e0f46d3c923e1dp2             3    267351 2147480779+  fb  VMFS

                  • 6. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
                    AndySimmons Hot Shot

                    Those look a little strange to me. Every LUN that you haven't put in bold starts off with a 2-block VMFS partition. That's 512B useable after you lose 512B for the VMFS header. After that, there's a larger VMFS partition that takes up the remaining blocks, which is obviously where all of your VM data is stored. It sounds like those are the LUNs that are working right now. Out of curiosity, were these partitioned by hand? I'm just not sure what purpose those tiny partitions could serve.

                     

                    At any rate, here's what's going on with the first device you put in bold text:

                     

                    /dev/disks/naa.6090a04870aeb18352d28456f5771e0b

                     

                    This just has a 2-block VMFS partition on it right now. If a working ESXi host shows a different partition layout, chances are you've lost the partition table, and need to rebuild it to match what the working hosts are showing before they reboot and re-read the new table.

                     

                    Here's the 2nd device you put in bold:

                     

                    /dev/disks/naa.6090a04870ae410990d2343a47783e53

                     

                    This is a "normal" looking VMFS partition table, although it sounds like it's inaccurate. The entire LUN is partitioned as a VMFS volume. Again, I'd be curious to see what a working host shows for the partition table here.

                     

                    Be extremely careful, but there is a VMKB article explaining how to use fdisk to restore VMFS partition tables. They recommend you call VMware support if you are uncomfortable with any part of the process.

                     

                    It wouldn't surprise me if you created a 2nd VMFS partition on naa.6090a04870aeb18352d28456f5771e0b to use blocks 3 - 267351 and that one started working.

                     

                    Likewise, I wouldn't be surprised if you overwrote the table for naa.6090a04870ae410990d2343a47783e53, creating VMFS partitions from blocks 1 - 2, and blocks 3 - 133676, and that one starts working. That is purely speculation, based off of what I see working on the other LUNs, and I would only act on that hunch if you can't get the correct layout from the other hosts, and you have no other option.

                     

                    Good luck!

                    • 7. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
                      uninspired Novice

                      My guess is the vss-control partitions are the small blocks at the beginning of each volume.

                       

                      Good idea to check the volumes on the *working* hosts. Results are interesting, if not particular useful. It looks the same on the ESXi hosts that are currently functional. It is, in fact, misreporting the size. The Config > Storage section shows it as 2TB.

                       

                      Disk /dev/disks/naa.6090a04870ae410990d2343a47783e53: 1099.5 GB, 1099526307840 bytes
                      255 heads, 63 sectors/track, 133676 cylinders
                      Units = cylinders of 16065 * 512 = 8225280 bytes

                       

                                                                 Device Boot      Start         End      Blocks  Id System
                      /dev/disks/naa.6090a04870ae410990d2343a47783e53p1             1    133676 1073752406   fb  VMFS

                       

                      I'm going to migrate all of the VMs to another datastore so I can safely try repairing the partition table without having to worry about losing anything.

                       

                      Thanks for your input. I'll report back here with results once I've safely moved all of the VMs elsewhere.

                      • 8. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
                        AndySimmons Hot Shot

                        Interesting -- good luck!

                         

                        As a side note, VMFS partitions are typically aligned on a 64KB boundary for performance reasons. I just figured I'd mention that now since it sounds like you're pulling all of the VMs off of those datastores, and can afford to make some destructive partition changes on those LUNs. This alignment is done automatically when creating a new VMFS volume from the vSphere client, or this doc explains how to do it with fdisk.

                        • 9. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
                          sunny1986 Enthusiast

                          Hi What did  you do to get the datastore back?

                          • 10. Re: iSCSI LUNs Connected but Datastore Not Visible/Available
                            MaciekTX Lurker

                            I have the same issue. How was that resolved?