VMware Cloud Community
robotdog
Contributor
Contributor
Jump to solution

Removed VMFS3 volume from iSCSI target - am I screwed?

Hi. I was "trying" to move some virtual machines around between servers and SANs. On one ESXi 3.5 server, I "had" a VMFS volume on an iSCSI target on a basic SAN device.

The volume "was" listed on the server/configuration tab under storage. I removed it thinking it would just be removing the reference to the volume from that host's perspective but it appears to have deleted the volume from the iSCSI target.

Is that correct - is my volume gone? Am I screwed? Is there anyway to recover the volume?

Any help or guidance would be really appreciated. Thanks in advance.

Reply
0 Kudos
1 Solution

Accepted Solutions
AndreTheGiant
Immortal
Immortal
Jump to solution

You have to thank Edward Haletky, not me Smiley Wink

In the VMware Communities his nick is .

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro

View solution in original post

Reply
0 Kudos
12 Replies
VMmatty
Virtuoso
Virtuoso
Jump to solution

Did you try to rescan your adapters to see if the volume is still available? What happens if you go into Storage and click Add? Do you still see the volume?

Matt | http://www.thelowercasew.com | @mattliebowitz
Reply
0 Kudos
robotdog
Contributor
Contributor
Jump to solution

I should have mentioned in my first post...

I did try a rescan for volumes but nothing showes up.

When I went to add a storage volume, it says the LUN has no volumes on it and all the space is free.

It sounds like there are some options to recover based on the research but I don't see any clear step-by-step instructions.

Reply
0 Kudos
AndreTheGiant
Immortal
Immortal
Jump to solution

After the rescan operation, what is the output of this command?

fdisk -l

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
Reply
0 Kudos
robotdog
Contributor
Contributor
Jump to solution

Thank you for your help.

The first /dev entry is the iSCSI LUN I believe.

Disk /dev/disks/vmhba32:1:0:0: 214.7 GB, 214748364800 bytes

255 heads, 63 sectors/track, 26108 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Disk /dev/disks/vmhba1:0:0:0: 146.6 GB, 146695782400 bytes

64 heads, 32 sectors/track, 139900 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/disks/vmhba1:0:0:1 5 750 763904 5 Extended

/dev/disks/vmhba1:0:0:2 751 4845 4193280 6 FAT16

/dev/disks/vmhba1:0:0:3 4846 139900 138296320 fb VMFS

/dev/disks/vmhba1:0:0:4 * 1 4 4080 4 FAT16 <32M

/dev/disks/vmhba1:0:0:5 5 52 49136 6 FAT16

/dev/disks/vmhba1:0:0:6 53 100 49136 6 FAT16

/dev/disks/vmhba1:0:0:7 101 210 112624 fc VMKcore

/dev/disks/vmhba1:0:0:8 211 750 552944 6 FAT16

Partition table entries are not in disk order

Reply
0 Kudos
AndreTheGiant
Immortal
Immortal
Jump to solution

Bad news... I do not see the VMFS partition in your LUN (it was a 214G LUN?).

Call VMware to try to restore the partition (if possible).

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
Reply
0 Kudos
robotdog
Contributor
Contributor
Jump to solution

Thanks for your help. After searching, I finally came upon some instructions that worked and it ended up being pretty easy. I lifted the following and pasted it here for others. I also uploaded the PDF they mention although their version of the instructions worked great. Whew!!!

Recovering VMFS partitions with VMware ESX troubleshooting

Edward Haletky, Contributor

02.19.2009

Rating: --- (out of 5)

Enterprise IT tips and expert advice\

[Enterprise IT tips and expert advice\

http://digg.com/submit?phase=2&url=http%3A%2F%2FsearchVMware%2Etechtarget%2Ecom%2Ftip%2F0%2C289483%2...VMFSpartitionswithVMwareESXtroubleshooting&topic=tech_news&bodytext=Reinstalling%20servers%20is%20a%20common%20task%20for%20system%20administrators%2C%20but%20when%20a%20typical%20process%20goes%20awry%2C%20it%20can%20wreak%20havoc%2E%20In%20this%20two%2Dpart%20series%20on%20troubleshooting%20VMware%20ESX%2C%20a%20VMware%20expert%20mistakenly%20deletes%20additional%20partitions%20that%20show%20up%20after%20he%20performed%20a%20VMware%20Consolidated%20Backup%20proxy%20host%20reinstall%2C%20which%20nearly%20destroys%20the%20Virtual%20Machine%20File%20System%20and%20RDM%20data%2E] Digg This!\ [Digg This!\

http://www.stumbleupon.com/submit?url=http%3A%2F%2FsearchVMware%2Etechtarget%2Ecom%2Ftip%2F0%2C28948...VMFSpartitionswithVMwareESXtroubleshooting] StumbleUpon\ [StumbleUpon\

http://del.icio.us/post?v=4&noui&jump=close&url=http%3A%2F%2FsearchVMware%2Etechtarget%2Ecom%2Ftip%2...VMFSpartitionswithVMwareESXtroubleshooting] Del.icio.us\ [Del.icio.us\

http://fusion.google.com/add?feedurl=http://rss.techtarget.com/520.xml]

+In the course of reinstalling a VMware Consolidated Backup (VCB) proxy host, our expert encountered two problems that can be major headaches for accessing files and data in the Virtual Machine File System (VMFS). First, he deleted partitions from his ESX logical unit numbers (LUNs), which could have meant losing access to all data in the VMFS. The following article features his workaround to that problem. He then discovered he couldn't restore virtual raw device mapping (RDM) in the same way, which also meant potentially losing a massive amount of valuable historical data. Part two of this series features his workaround to that issue.

+

As a VMware expert, I am supposed to know that I should disconnect my storage area network (SAN) from a server before reinstalling the server. The server in question was a backup host instead of an ESX host, so I mistakenly thought that Windows would disconnect the SAN.

Deleting mystery partitions is a bad idea

After the reinstall, my server had more partitions than before the reinstall. Without too much thought, I deleted them -- bad idea. The partitions I deleted happened to be those used by my VMware ESX LUNs. So if I were to reboot my ESX servers, they would permanently lose access to their data.

At this point, I knew that the prior day at least one part of the irreplaceable data had been backed up and taken off-site. But I still needed to restore the Virtual Machine File System (VMFS) and raw device mapping (RDM).

VMware ESX troubleshooting 101: Check the VMware ESX hosts

The first step in troubleshooting a VMware ESX problem is checking the ESX hosts. I discovered that the virtual machines (VMs) were still running. Apparently, ESX is robust enough to continue operation without its physical partitions. Since ESX was still running, and wanting to discover the damage, I went to the service console and ran the command fdisk -l. I ended up with a lot of blank partitions, as you can see in this file\ (click on link to see text).

I expected to see something like this\ instead (click on link to see text).

The difference between the two was that the first three partitions (highlighted in the second file) were missing.

Restoring the VMFS

After some research and some help from the tech folks who follow me on Twitter\, I discovered a link to a presentation on . The PDF proved invaluable.

To summarize the information in that document, to restore the VMFS you need to use the command fdisk to rebuild the partition table. Then move the start block to the proper alignment. Then refresh and re-scan to access the partitions.

Using fdisk, however, can be dangerous and requires caution. Instructions for using fdisk:

- Add a new primary partition number 1
- Take default first and last cylinders
- Change a partition's system id to fb or the VMFS partition id
- Move the beginning of the data in the partition to have an offset of 128 used for VMFS
- Write the new partition table to the disk and exit
- Repeat for all lost VMFS partitions.

This is what those instructions translated into for /dev/sda and /dev/sdc within my ESX host's service console:

# fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel.
Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.
The number of cylinders for this disk is set to 39162.
There is nothing wrong with that, but this is larger than 1024, and could, in certain setups, cause problems with two things: software that runs at boot time (e.g., old versions of LILO) and/or booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-39162, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-39162, default 39162):
Using default value 39162
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fb
Changed system type of partition 1 to fb (Unknown)
Command (m for help): x
Expert command (m for help): b
Partition number (1-4): 1
New beginning of data (63-629137529, default 63): 128
Expert command (m for help): w
The partition table has been altered!

I tested my handiwork by rebooting the single ESX host that wasn't running any VMs and rescanned its LUNs. Thankfully, the VMFS had been successfully restored.

|

Reply
0 Kudos
AndreTheGiant
Immortal
Immortal
Jump to solution

You have to thank Edward Haletky, not me Smiley Wink

In the VMware Communities his nick is .

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
Reply
0 Kudos
robotdog
Contributor
Contributor
Jump to solution

Edward AKA Texiwill - you saved me - thank you, thank you!

Reply
0 Kudos
TimPhillips
Enthusiast
Enthusiast
Jump to solution

And what iSCSI target do you use?

Reply
0 Kudos
robotdog
Contributor
Contributor
Jump to solution

I use a QNAP appliance. Does that make a difference? I had deleted the volume through the Virtual Infrastructure Client.

Reply
0 Kudos
TimPhillips
Enthusiast
Enthusiast
Jump to solution

Just interesting, cause I also one day have accidientally deleted my disk from target, but then connected it back without any problem (i use starwind`s offer)

Reply
0 Kudos
pratkai
Contributor
Contributor
Jump to solution

Hi All,

After reading thru all the possible solutions I decided to ask for help, since none of these scenarios apply to me perfectly.

In my case a storage partition "fall out" from vmware host because of a faulty FC switch. Yes, it is a 1T mirrored disk attached via FC from a storage subsystem.

FC Switch was replaced, and the vmware host could see the FC storage pratition, unfortunately the guest OS's were gone. I have choosen to restart the vmware host, just in case.., hoping it might clear things up before I try to run the guest VMs. It might have been a bad decision, since it cannot attach the disk to the datastore anymore. However it can see it under storage controllers, but when I do a rescan nothing happens. It shows the correct size, LUN, etc however...

When I try to add as a storage it wants to reformat it, which I obviously don't want to happen. It also shows that the disk currently contains a VMFS partition...

fdisk output:

:~# fdisk -l /dev/sdd

Disk /dev/sdd: 892.5 GB, 892553134080 bytes

255 heads, 63 sectors/track, 108513 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x2324134e

Device Boot Start End Blocks Id System

/dev/sdd1 1 108513 871630608+ fb Unknown

Seems like the partition is okay, but how can I remount it to be a datastore again? I might be able to restore data from it.

Thanks much,

Peter

Reply
0 Kudos