12 Replies Latest reply on Sep 22, 2009 3:44 AM by pratkai

    Removed VMFS3 volume from iSCSI target - am I screwed?

    robotdog Novice

      Hi. I was "trying" to move some virtual machines around between servers and SANs. On one ESXi 3.5 server, I "had" a VMFS volume on an iSCSI target on a basic SAN device. 



      The volume "was" listed on the server/configuration tab under storage. I removed it thinking it would just be removing the reference to the volume from that host's perspective but it appears to have deleted the volume from the iSCSI target.



      Is that correct - is my volume gone? Am I screwed? Is there anyway to recover the volume?



      Any help or guidance would be really appreciated. Thanks in advance.

        • 1. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
          VMmatty Master
          vExpertUser Moderators

          Did you try to rescan your adapters to see if the volume is still available?  What happens if you go into Storage and click Add?  Do you still see the volume?

          • 2. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
            robotdog Novice


            I should have mentioned in my first post...



            I did try a rescan for volumes but nothing showes up.



            When I went to add a storage volume, it says the LUN has no volumes on it and all the space is free.



            It sounds like there are some options to recover based on the research but I don't see any clear step-by-step instructions.









            • 3. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
              AndreTheGiant Guru
              User ModeratorsvExpert

              After the rescan operation, what is the output of this command?

              fdisk -l



              • 4. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                robotdog Novice

                Thank you for your help.


                The first /dev entry is the iSCSI LUN I believe.



                Disk /dev/disks/vmhba32:1:0:0: 214.7 GB, 214748364800 bytes

                255 heads, 63 sectors/track, 26108 cylinders

                Units = cylinders of 16065 * 512 = 8225280 bytes



                                  Device Boot    Start       End    Blocks   Id  System



                Disk /dev/disks/vmhba1:0:0:0: 146.6 GB, 146695782400 bytes

                64 heads, 32 sectors/track, 139900 cylinders

                Units = cylinders of 2048 * 512 = 1048576 bytes



                                 Device Boot    Start       End    Blocks   Id  System

                /dev/disks/vmhba1:0:0:1             5       750    763904    5  Extended

                /dev/disks/vmhba1:0:0:2           751      4845   4193280    6  FAT16

                /dev/disks/vmhba1:0:0:3          4846    139900 138296320   fb  VMFS

                /dev/disks/vmhba1:0:0:4   *         1         4      4080    4  FAT16 <32M

                /dev/disks/vmhba1:0:0:5             5        52     49136    6  FAT16

                /dev/disks/vmhba1:0:0:6            53       100     49136    6  FAT16

                /dev/disks/vmhba1:0:0:7           101       210    112624   fc  VMKcore

                /dev/disks/vmhba1:0:0:8           211       750    552944    6  FAT16



                Partition table entries are not in disk order

                • 5. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                  AndreTheGiant Guru
                  vExpertUser Moderators

                  Bad news... I do not see the VMFS partition in your LUN (it was a 214G LUN?).

                  Call VMware to try to restore the partition (if possible).



                  • 6. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                    robotdog Novice


                    Thanks for your help.  After searching, I finally came upon some instructions that worked and it ended up being pretty easy.  I lifted the following and pasted it here for others.  I also uploaded the PDF they mention although their version of the instructions worked great.  Whew!!!









                    Recovering VMFS partitions with VMware ESX troubleshooting

                                   Edward Haletky, Contributor


                                   Rating: --- (out of 5)

                    http://rss.techtarget.com/520.xml Enterprise IT tips and expert advice\

                    [Enterprise IT tips and expert advice\

                    http://digg.com/submit?phase=2&url=http%3A%2F%2FsearchVMware%2Etechtarget%2Ecom%2Ftip%2F0%2C289483%2Csid179%5Fgci1347841%5Fmem1%2C00%2Ehtml&title=RecoveringVMFSpartitionswithVMwareESXtroubleshooting&topic=tech_news&bodytext=Reinstalling%20servers%20is%20a%20common%20task%20for%20system%20administrators%2C%20but%20when%20a%20typical%20process%20goes%20awry%2C%20it%20can%20wreak%20havoc%2E%20In%20this%20two%2Dpart%20series%20on%20troubleshooting%20VMware%20ESX%2C%20a%20VMware%20expert%20mistakenly%20deletes%20additional%20partitions%20that%20show%20up%20after%20he%20performed%20a%20VMware%20Consolidated%20Backup%20proxy%20host%20reinstall%2C%20which%20nearly%20destroys%20the%20Virtual%20Machine%20File%20System%20and%20RDM%20data%2E] Digg This!\    [Digg This!\

                    http://www.stumbleupon.com/submit?url=http%3A%2F%2FsearchVMware%2Etechtarget%2Ecom%2Ftip%2F0%2C289483%2Csid179%5Fgci1347841%5Fmem1%2C00%2Ehtml&title=RecoveringVMFSpartitionswithVMwareESXtroubleshooting] StumbleUpon\    [StumbleUpon\

                    http://del.icio.us/post?v=4&noui&jump=close&url=http%3A%2F%2FsearchVMware%2Etechtarget%2Ecom%2Ftip%2F0%2C289483%2Csid179%5Fgci1347841%5Fmem1%2C00%2Ehtml&title=RecoveringVMFSpartitionswithVMwareESXtroubleshooting] Del.icio.us\    [Del.icio.us\


                    +In the course of reinstalling a VMware Consolidated Backup (VCB) proxy host, our expert encountered two problems that can be major headaches for accessing files and data in the Virtual Machine File System (VMFS). First, he deleted partitions from his ESX logical unit numbers (LUNs), which could have meant losing access to all data in the VMFS. The following article features his workaround to that problem. He then discovered he couldn't restore virtual raw device mapping (RDM) in the same way, which also meant potentially losing a massive amount of valuable historical data. Part two of this series features his workaround to that issue.





                                                  As a VMware expert, I am supposed to know that I should disconnect my storage area network (SAN) from a server before reinstalling the server. The server in question was a backup host instead of an ESX host, so I mistakenly thought that Windows would disconnect the SAN.



                    Deleting mystery partitions is a bad idea

                                                  After the reinstall, my server had more partitions than before the reinstall. Without too much thought, I deleted them -- bad idea. The partitions I deleted happened to be those used by my VMware ESX LUNs. So if I were to reboot my ESX servers, they would permanently lose access to their data.



                                                  At this point, I knew that the prior day at least one part of the irreplaceable data had been backed up and taken off-site. But I still needed to restore the Virtual Machine File System (VMFS) and raw device mapping (RDM).



                    VMware ESX troubleshooting 101: Check the VMware ESX hosts

                                                  The first step in troubleshooting a VMware ESX problem is checking the ESX hosts. I discovered that the virtual machines (VMs) were still running. Apparently, ESX is robust enough to continue operation without its physical partitions. Since ESX was still running, and wanting to discover the damage, I went to the service console and ran the command fdisk -l. I ended up with a lot of blank partitions, as you can see in this file\ (click on link to see text).



                                                  I expected to see something like this\ instead (click on link to see text).



                                                  The difference between the two was that the first three partitions (highlighted in the second file) were missing.



                    Restoring the VMFS

                                                  After some research and some help from the tech folks who follow me on Twitter\, I discovered a link to a presentation on . The PDF proved invaluable.



                                                  To summarize the information in that document, to restore the VMFS you need to use the command fdisk to rebuild the partition table. Then move the start block to the proper alignment. Then refresh and re-scan to access the partitions.



                                                  Using fdisk, however, can be dangerous and requires caution. Instructions for using fdisk:


                    - Add a new primary partition number 1
                    - Take default first and last cylinders
                    - Change a partition's system id to fb or the VMFS partition id
                    - Move the beginning of the data in the partition to have an offset of 128 used for VMFS
                    - Write the new partition table to the disk and exit
                    - Repeat for all lost VMFS partitions.

                                                  This is what those instructions translated into for /dev/sda and /dev/sdc within my ESX host's service console:

                    # fdisk /dev/sda
                    Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
                    Building a new DOS disklabel.
                    Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.
                    The number of cylinders for this disk is set to 39162.
                    There is nothing wrong with that, but this is larger than 1024, and could, in certain setups, cause problems with two things: software that runs at boot time (e.g., old versions of LILO) and/or booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)
                    Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
                    Command (m for help): n
                    Command action
                    e extended
                    p primary partition (1-4)
                    Partition number (1-4): 1
                    First cylinder (1-39162, default 1):
                    Using default value 1
                    Last cylinder or +size or +sizeM or +sizeK (1-39162, default 39162):
                    Using default value 39162
                    Command (m for help): t
                    Selected partition 1
                    Hex code (type L to list codes): fb
                    Changed system type of partition 1 to fb (Unknown)
                    Command (m for help): x
                    Expert command (m for help): b
                    Partition number (1-4): 1
                    New beginning of data (63-629137529, default 63): 128
                    Expert command (m for help): w
                    The partition table has been altered!


                                                  I tested my handiwork by rebooting the single ESX host that wasn't running any VMs and rescanned its LUNs. Thankfully, the VMFS had been successfully restored.












                    • 7. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                      AndreTheGiant Guru
                      vExpertUser Moderators

                      You have to thank Edward Haletky, not me

                      In the VMware Communities his nick is Texiwill.



                      • 8. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                        robotdog Novice


                        Edward AKA Texiwill - you saved me - thank you, thank you!



                        • 9. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                          TimPhillips Hot Shot

                          And what iSCSI target do you use?

                          • 10. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                            robotdog Novice

                            I use a QNAP appliance. Does that make a difference?  I had deleted the volume through the Virtual Infrastructure Client.

                            • 11. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                              TimPhillips Hot Shot

                              Just interesting, cause I also one day have accidientally deleted my disk from target, but then connected it back without any problem (i use starwind`s offer)

                              • 12. Re: Removed VMFS3 volume from iSCSI target - am I screwed?
                                pratkai Lurker

                                Hi All,


                                After reading thru all the possible solutions I decided to ask for help, since none of these scenarios apply to me perfectly.


                                In my case a storage partition "fall out" from vmware host because of a faulty FC switch. Yes, it is a 1T mirrored disk attached via FC from a storage subsystem.


                                FC Switch was replaced, and the vmware host could see the FC storage pratition, unfortunately the guest OS's were gone. I have choosen to restart the vmware host, just in case.., hoping it might clear things up before I try to run the guest VMs. It might have been a bad decision, since it cannot attach the disk to the datastore anymore. However it can see it under storage controllers, but when I do a rescan nothing happens. It shows the correct size, LUN, etc however...


                                When I try to add as a storage it wants to reformat it, which I obviously don't want to happen. It also shows that the disk currently contains a VMFS partition...


                                fdisk output:


                                :~# fdisk -l /dev/sdd


                                Disk /dev/sdd: 892.5 GB, 892553134080 bytes

                                255 heads, 63 sectors/track, 108513 cylinders

                                Units = cylinders of 16065 * 512 = 8225280 bytes

                                Disk identifier: 0x2324134e


                                   Device Boot      Start         End      Blocks   Id  System

                                /dev/sdd1               1      108513   871630608+  fb  Unknown



                                Seems like the partition is okay, but how can I remount it to be a datastore again? I might be able to restore data from it.


                                Thanks much,