Remount VMFS Partition - no format

[gesturgis]

Good afternoon all,

We just had a problem with our RAID controller which caused some SCSI errors and a crash of ESX. Since then, we have replaced the disk, rebuilt the array, and everything more or less appears normal except for one big problem.

We now have no vmfs volumes according to Host > Summary (datastore) or Host > Configuration > Storage

fdisk -l shows that our partitions are intact and available (type fb) but there is nothing at all in /vmfs/volumes

So - do we have any hope? Can we "re-mount" those volumes?

The Host > Configuration > Storage Adapters page shows the SCSI targets as we would expect to see them.

Any suggestions are most welcome.

Thanks,

Grant

-

---

[fakber]

From the VI Client you can select "Rescan" and select scan for volumes.

From the command-line run:

vmkfstools -V

That should remount all of your VMFS volumes that are currently presented to your ESX server. (Assuming you do not have any other issues present.)

[gesturgis]

No such luck (but no errors):

# vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 2.3G 2.4G 50% /

/dev/sda1 99M 29M 66M 31% /boot

none 132M 0 132M 0% /dev/shm

/dev/sda6 2.0G 82M 1.8G 5% /var/log

/vmfs/devices 136G 0 136G 0% /vmfs/devices

# vmkfstools -V

# vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 2.3G 2.4G 50% /

/dev/sda1 99M 29M 66M 31% /boot

none 132M 0 132M 0% /dev/shm

/dev/sda6 2.0G 82M 1.8G 5% /var/log

/vmfs/devices 136G 0 136G 0% /vmfs/devices

[fakber]

What version of ESX are you using?

Also, was the partition ever formatted before? If not then you may have to format it with vmkfstools -C ...

[gesturgis]

This is ESX 3.0.0. Yes, the partitions previously were working fine. We have ton of VMs out there that we need to get back online, that is the problem.

So we know the partitions are there, we can see them. It just as if ESX isn't allowing us access to them.

[wondab]

Have you tried rescanning yet as suggested earlier?

From the VI client select the ESX host, click on the properties tab and select "configuration".

Select Storage Adapter and click on

"rescan" (right top). Select both check boxes on the next screen.

Then go to "add storage" under storage, selet vmfs volume (or LUN-not sure) and see if the LUNs show up there.

You can also scan from the terminal: find out the vmhba numbers, I think it is under proc/vmware/pci, starting with the lower numberd vmhba type

esxcfg-rescan vmhbaX

Good luck...

[gesturgis]

Thanks for the info. That all works except when I get to "add storage",

the VI client warns me that the "current disk layout will be destroyed".

I have several VMs on that disk that I would like to preserve (i.e. not

format that drive).

[wondab]

If you do a ls -l in the vmfs/volumes directory, do you see anything at all?

Is this a standalone ESX server with only local storage?

[gesturgis]

Nothing at all in /vmfs/volumes directory. Yes, standalone system, all internal storage.

[fakber]

Do you see any messages in /var/log/vmkernel?

Specifically, look for lines with SCSI: ... status = / 0x* 0x* 0x*

Also, look in /proc/scsi/scsi to ensure that all disks can be seen.

In /proc/vmware/scsi/vmhba0/* look for any Command Aborts (cmdsAbrts) or Bus Resets (busRsts). They should be equal to 0.

You may want to use the storageMonitor utility to see any errors during a rescan or the vmkfstools -V operation.

[wondab]

You still have the disk that you replaced-if everything fails and it was a mirror I would try to go from there as a last resort.

[wondab]

And I would post the same thing on the general discussion forum-it seems like more people respond to that one

[gesturgis]

Now I believe we are getting somewhere, this shows up in vmkernel when I

do a vmkfstools -V:

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.620 cpu3:1038)LVM: 5670:

Device vmhba0:0:0:3 is a snapshot:

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.621 cpu3:1038)LVM: 5676:

disk ID:

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.665 cpu3:1038)WARNING:

LVM: 4844: detected as a snapshot device. Disallowing

access to the LUNsince resignaturing is turned off.

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.732 cpu3:1038)LVM: 5670:

Device vmhba0:1:0:1 is a snapshot:

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.732 cpu3:1038)LVM: 5676:

disk ID:

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.745 cpu3:1038)WARNING:

LVM: 4844: detected as a snapshot device. Disallowing

access to the LUNsince resignaturing is turned off.

[lukas_radil]

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.665 cpu3:1038)WARNING:

LVM: 4844: detected as a snapshot device. Disallowing

access to the LUNsince resignaturing is turned off.

Dec 6 08:45:06 vm01bldr vmkernel: 0:19:47:02.745 cpu3:1038)WARNING:

LVM: 4844: detected as a snapshot device. Disallowing

access to the LUNsince resignaturing is turned off.

This messages says that, that your system find two partitions with same signatures as is already in config, You can try allow resignaturing. VI Client - ESX Configuration - Advanced Settings -

LVM.EnableResignature setting to 1.

Then restart esx server.

Lukas

[wondab · Accepted solution]

Check out the SAN CFG Guide page 113

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=http--wwwvmwarecom-pd...

Mounting Original and Snapshot VMFS Volumes

You can mount both, the original and snapshot VMFS volumes on the same ESX Server

host.

To mount original and snapshot VMFS volumes

1 In the VI Client, select the host in the inventory panel.

2 Click the Configuration tab and click Advanced Settings.

3 Perform the following tasks repeatedly, as needed:

a Make the snapshot.

b Add the snapshot to the storage array.

c Select LVM in the left panel, then set the LVM.EnableResignature option to 1.

4 Rescan the volume.

After rescan, the LUN appears as /vmfs/volumes/snap-/ paths, you must change

these items to reflect the resignatured volume path.

5 If necessary, set the LVM.EnableResignature option to 0 after resignaturing is

complete.

[fakber]

Before moving any further, I would ensure that the LUN numbers for your LUNs as presented on each ESX host is identical.

That is if you have 3 ESX servers named ESXA, ESXB, and ESXC, the LUNs should be numbered identically from the array.

ESXA - vmhba1:0:1 (SAN LUN 1), vmhba1:0:2 (SAN LUN 2)

ESXB - vmhba1:0:1 (SAN LUN 1), vmhba1:0:2 (SAN LUN 2)

ESXC - vmhba1:0:1 (SAN LUN 2), vmhba1:0:2 (SAN LUN 1) <=== This is wrong and can cause your snapshot LUN messages and prevent you from seeing the LUN(s).

Therefore ensure that you have the LUNs presented identially for each host.

Look at the "Id" field in /proc/vmware/scsi/vmhba1/0:1 files to and compare on each host. (Run grep Id: /proc/vmware/scsi/vmhba/ to get a quick list of LUN ID's and vmhba numbers.)

If you have identical LUNs then you need to resignature but not before this has been verified.

[gesturgis]

Thanks much wondab, that is exactly right. Also addressed here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100157...