Recover VMFS / iSCSI datastore - ESXi 4?

Titans99 · ‎04-07-2010

Hi, my theory is the partition table is corrupt and I'm not exactly sure how this happened. I have a 1TB LUN presented to a ESXi 4 host via iSCSI but the datastore no longer shows up? I can see the iSCSI target and LUN via the Vsphere client. If I click "Add Storage" -> Choose the 1TB lun -> Assign a new signature, I get the following error...

Resignature unresolved VMFS volume. Error during the configuration of the host: SysinfoException: Node (5014) ; Status(bad0021)= Timeout; Message=Module: lvmdriver Instance(0): Input(1) naa.60014050a3832a2db316d4d47db632d6:1

fdisk -lu shows the following for the LUN:

Disk /dev/disks/naa.60014050a3832a2db316d4d47db632d6: 1073.7 GB, 1073741824512 bytes 255 heads, 63 sectors/track, 130541 cylinders, total 2097152001 sectors Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System /dev/disks/naa.60014050a3832a2db316d4d47db632d6p1 128 2097141164 1048570518+ fb VMFS

I am not sure how to attempt repair on ESXi. I tried the command: dosfsck -v /dev/disks/naa.60014050a3832a2db316d4d47db632d6p1, but it shows no such file of directory (not sure of the syntax). Any CLI help is really appreciated!

golddiggie · ‎04-07-2010

What's the iSCSI storage device you're using? It could be a problem based on the storage which will recur even if you fix it (for now)... Is the iSCSI device using the latest (supported) firmware version? Is the iSCSI device on the HCL for ESX/ESXi 4? When was the last time you updated the ESXi host? Have you checked to make sure there are not additional updates that address the situation you are in (combination of hardware and such)?

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

Titans99 · ‎04-07-2010

All good questions golddiggie. The server/storage is in a dev environment and we're just using a QNAP TS-639 storage device which doesn't appear to be on the HCL (the TS-659 is on there). There is a firmware update out for the QNAP that I haven't installed yet, but will. I've searched for hours trying to figure out what commands to run to attempt repair but maybe I'm searching for the wrong thing. (I finally figured out how to enable SSH on ESXi)

In the meantime, any ideas on how to repair the vmfs partition so I can try to copy off the vmdk's?

Titans99 · ‎04-07-2010

Just FYI - I know enough to be dangerous in terms of linux administration, but I'm learning.

golddiggie · ‎04-07-2010

I'd reach out to QNAP to find out if it's at all possible... You might be able to telnet into the device and copy the contents of one LUN into another (without destroying it)...

I was looking at the QNAP products, for a while, as a solution for my iSCSI/storage needs... I did communicate with them and while they claimed the devices (older ones at the time) were compatible, I was leery since they didn't have the VMware Ready logo/graphic on them. The newer devices do have that certification, which gives higher confidence that they would work. Still, I'll probably end up with a more enterprise/business class product when it comes time. I don't care about a lot of the bells and whistles they toss into their devices. I just want a solid iSCSI device that I can depend upon to run 24x7... I'll put it onto one of my UPS units (or get a new one for it) and make sure it never shuts down hard/wrong...

Did you have any power glitches recently?

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

golddiggie · ‎04-07-2010

Just FYI - I know enough to be dangerous in terms of linux administration, but I'm learning.

</div>But do you know enought to know you don't know enough and then stop before causing damage/harm??? If I had a dollar for every person who's systems I've fixed that thought they knew what they were doing, I wouldn't need to work another day... Ever.

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

Titans99 · ‎04-07-2010

No power outages that I know of. It all started yesterday when a couple of VM's 'locked up" and would not shut down or power off. I couldn't even see them in the VI client maching console. I had to hard reset the host and it all went to hell after that. I logged in to the QNAP and saw it was "resyncing" which took took about 6 hours. After that, the datastore disappeared.

But do you know enought to know you don't know enough and then stop before causing damage/harm??

Most of the time yes! I've had to call on linux experts quite of few times

though.

Are there any partition related commands you can point me to? I can call QNAP but a disk is a disk is a partition is a partition. It's either repairable or not.

golddiggie · ‎04-07-2010

I've not dived into trying to hack/fix VMFS LUNs so far... Are all the disks in the array 100%? Have you checked to make sure it's not in any kind of degraded state? What RAID level did you configure the device to use?

Do you have the CLI tools available to you? Such as through the vMA? You could try something with the du commands, but I've not gone too far into those either... Use the man command to investigate... Maybe you'll get lucky...

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

PaulSvirin · ‎04-07-2010

Some good links that can help you:

http://communities.vmware.com/message/1477368

Recovering a lost partition table on a VMFS volume

Anyway try running disk checkonthe target side to be sure that disks on hardware level are alright.

---

iSCSI SAN software

http://www.starwindsoftware.com

--- iSCSI SAN software http://www.starwindsoftware.com

golddiggie · ‎04-07-2010

KB article looks like it will destroy any data existing on the LUN as part of the 'fix'... Or am I misreading it?

The information in the first link could provide a solution though... Except that part of it also linksback to the same KB article... :smileyshocked:

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

Titans99 · ‎04-09-2010

@PaulSvirin, when I get to step 6c here (under the ESX 4 heading) and choose "partition 1" , I get the following:

"Partition 1 is already defined, delete it before re-adding"

I'm a little hesitant of deleting the partition first, but maybe I have to? Any suggestions? Thanks

kormen · ‎07-18-2011

Hi!

i have exactly same problem. I have data that VMFS i dont reformat it. But i see this is the only possibility. I don't understand, why not good the ext filesys? That's have repir tool.

Anyway...

The problem is solved without data loss? If yes, Tell me!

Kormen