Help please datastore1 is missing.

[jabz10]

Hi I run a Mac Pro 5,1 with ESXi 5.5

I had Windows 7 and OS X server guests on Datastore1 which is a 1TB Sata hard drive (t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR). 

My Esxi install is also on this 1TB Sata drive.

I have another 1TB Sata drive (t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LH4D)

that was not mounted yet. This setup was working fine for the last 8 months.

Recently I needed to install Windows 2012 Server, I thought I would create another datastore for it, so using vSphere client, I proceeded to Add storage, and chose the spare 1TB sata drive, I stepped through the wizard choosing VMFS 5, then named this Datastore, "Datastore2".  It added it the Datastore view, but then my Datastore 1 disappeared!    I tried a refresh etc but it still won't show up. 

Before trying anything extreme I thought this maybe bug or something, so I decided to enter maintenance mode, and reboot the Esxi.  When it came back up I connected via vSphere client  and it shows both the VM Guests are "Unknown" and (Inaccessible).  Arrrghh!!

I connected to the console via SSH and tried to list my volumes/partitions using fdisk.  Here is what is shows..

Seeing fdisk is depreciated I tried listing via partedUtil, the output of that is below the fdisk output.

So where to from here please?  I thought I would seek help before trying googled suggestions.  I would like to get my Datastore1 and it's guest VM installs back.

I assume the 1TB Sata drive itself is fine since ESXi is installed to it, and it boots fine to the console.  Any help appreciated thanks.

Attached my vmkernel.log

~ # fdisk -lu

***

*** The fdisk command is deprecated: fdisk does not handle GPT partitions.  Please use partedUtil

***

Found valid GPT with protective MBR; using GPT

Disk /dev/disks/t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LH4D: 1953525168 sectors, 1863M

Logical sector size: 512

Disk identifier (GUID): f4295ca1-1683-4da6-9277-487dddc2d2bd

Partition table holds up to 128 entries

First usable sector is 34, last usable sector is 1953525134

Number  Start (sector)    End (sector)  Size       Code  Name

   1            2048      1953523711       1863M   0700

Found valid GPT with protective MBR; using GPT

Disk /dev/disks/t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR: 1953525168 sectors, 1863M

Logical sector size: 512

Disk identifier (GUID): b3c4f185-38c5-46e4-905c-2017d64f0091

Partition table holds up to 128 entries

First usable sector is 34, last usable sector is 1953525134

Number  Start (sector)    End (sector)  Size       Code  Name

   1              64            8191        8128   0700

   2         7086080        15472639       8190K   0700

   3        15472640      1953525134       1848M   0700

   5            8224          520191        499K   0700

   6          520224         1032191        499K   0700

   7         1032224         1257471        219K   0700

   8         1257504         1843199        571K   0700

   9         1843200         7086079       5120K   0700

~ #

~ # ls /vmfs/devices/disks/

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LH4D

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LH4D:1

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:1

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:2

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:3

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:5

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:6

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:7

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:8

t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR:9

vml.01000000002020202020202020202020205a3144364c483444535431303030

vml.01000000002020202020202020202020205a3144364c483444535431303030:1

vml.01000000002020202020202020202020205a3144364c485a52535431303030

vml.01000000002020202020202020202020205a3144364c485a52535431303030:1

vml.01000000002020202020202020202020205a3144364c485a52535431303030:2

vml.01000000002020202020202020202020205a3144364c485a52535431303030:3

vml.01000000002020202020202020202020205a3144364c485a52535431303030:5

vml.01000000002020202020202020202020205a3144364c485a52535431303030:6

vml.01000000002020202020202020202020205a3144364c485a52535431303030:7

vml.01000000002020202020202020202020205a3144364c485a52535431303030:8

vml.01000000002020202020202020202020205a3144364c485a52535431303030:9

~ # partedUtil getptbl "/vmfs/devices/disks/t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LH4D"

gpt

121601 255 63 1953525168

1 2048 1953523711 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

~ # partedUtil getptbl "/vmfs/devices/disks/t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR"

gpt

121601 255 63 1953525168

1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

3 15472640 1953525134 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

~ #

When I try to Re-Add the datastore, I see the error as shown in the Screenshot..

[jabz10]

So no suggestions?

Also the partition tables aren't missing, so I cannot use the following

VMware KB: Recovering a lost partition table on a VMFS volume

VMware KB: Recreating a missing VMFS datastore partition in VMware vSphere 5.0/5.1/5.5

See output shows the tables intact.

~ # partedUtil getptbl /vmfs/devices/disks/t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LH4D

gpt

121601 255 63 1953525168

1 2048 1953523711 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

~ # partedUtil getptbl /vmfs/devices/disks/t10.ATA_____ST1000DX0012D1CM162__________________________________Z1D6LHZR

gpt

121601 255 63 1953525168

1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

3 15472640 1953525134 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

[jabz10]

Seeing as there has been no suggestions/help.  How can I copy the VMDK files and any associated files out of datastore1 and into datastore2 (which is mounted) so I can resume using my VMs?

[jabz10]

I can even list everything in the volumes, datastore 1, I just can't mount the freaking thing and I don't know why!

/vmfs/volumes # ls

33cb2471-d7433393-9b8a-499a3f5a18b4  53158f98-7004b68e-114b-002500f3d6ed

4e44a47b-1bcc3c23-32af-2d18ebe0590c  539c3241-43368a5a-89c4-002500f3d6ed

53158f8b-fdfb6ffd-c390-002500f3d6ed  datastore1

53158f94-ef55ceca-f657-002500f3d6ed  datastore2

/vmfs/volumes # cd datastore1

/vmfs/volumes/53158f94-ef55ceca-f657-002500f3d6ed # ls

ISO                Mac OS X Server    Windows 7 x64      Windows 7 x64 8GB

/vmfs/volumes/53158f94-ef55ceca-f657-002500f3d6ed # cd Mac\ OS\ X\ Server/

/vmfs/volumes/53158f94-ef55ceca-f657-002500f3d6ed/Mac OS X Server # ls

Mac OS X Server-flat.vmdk  Mac OS X Server.vmsd       vmware-1.log

Mac OS X Server.nvram      Mac OS X Server.vmx        vmware.log

Mac OS X Server.vmdk       Mac OS X Server.vmxf

/vmfs/volumes/53158f94-ef55ceca-f657-002500f3d6ed/Mac OS X Server # ls -l

total 12900360

-rw-------    1 root     root     536870912000 Jun 14 12:04 Mac OS X Server-flat.vmdk

-rw-------    1 root     root         74232 Jun 14 12:04 Mac OS X Server.nvram

-rw-------    1 root     root           531 Jun 14 08:16 Mac OS X Server.vmdk

-rw-r--r--    1 root     root             0 Mar  7 23:25 Mac OS X Server.vmsd

-rwxr-xr-x    1 root     root          3319 Jun 14 12:04 Mac OS X Server.vmx

-rw-r--r--    1 root     root           373 Jun 14 08:16 Mac OS X Server.vmxf

-rw-r--r--    1 root     root        129992 Jun 14 06:00 vmware-1.log

-rw-r--r--    1 root     root        158960 Jun 14 12:04 vmware.log

/vmfs/volumes/53158f94-ef55ceca-f657-002500f3d6ed/Mac OS X Server #

[grasshopper]

You can try using vmkfstools to copy the vmdk from the 'bad' datastore to the 'good' datastore.

1.  Create a folder on the target datastore (i.e. the name of the VM)

2.  vmkfstools -i <source> <destination>

     Note:  Ensure that you use the full path (i.e. /vmfs/volumes/datastorex/my.vmdk) for both source and destination

3.  cp the .vmx file to the new folder and right click add to inventory, then modify via Edit Settings to point to the new location of the disks

     Note:  (Alternatively, create new VM pointing to the disks copied to datastore2)

4.  Power on the VM and answer the question as "I Moved it" if you opted for the cp option in Step 3 instead of creating a new VM shell

Let us know how it goes.  Also ensure that the BIOS on your 5,1 is at least 'MP51.88Z.007F.B03.1010071432' as that is the supported level per the HCL.

[grasshopper]

Based on the vmkernel logs, it appears that the underlying disk (Z1D6LHZR) for datastore1 has gone bad.

failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x4

Here's how the above translates...

ESXi Host = OK

Device = Check Condition

VMware Plugin = No error

Sense Key = MEDIUM ERROR

ASC/ASCQ = UNRECOVERED READ ERROR - AUTO REALLOCATE FAILED

If you get lucky and vmkfstools is able to get your VMs onto a healthy datastore, consider then moving your VMs onto an external NFS or iSCSI device and run that way going forward.  The MTBF of your disk type is a risk.  I would also USB boot the host as well instead of using local disk if changing them out is not an option.

[jabz10]

Great thanks I will try that out.  I didnt think the MacPro could boot from USB? Do you have instructions for this?

When you mean moving the VMs on external do you mean a RAID setup i.e. install a Raid card into Macpro?

thanks

[grasshopper]

Ok, I won't confuse you with the USB booting of the hypervisor, that is less important than getting your VMs onto external storage.  The ESXi host can be rebuilt time and again without risk if the VMs are external to the MAC.

By external storage, I mean a dedicated NAS solution (i.e. LenovoEMC ,QNAP, Synology, etc.).  Alternatively, you could install OpenFiler or FreeNAS on a spare server.  Either way, you would then connect your ESXi host via NFS or iSCSI to that NAS device.  Your VMs (i.e. vmdk's, etc.) would then live on that external device, but they would consume CPU and Memory like they normally would from your MAC 5,1.

[jabz10]

Ok I have a Thecus 5200XXX NAS, would iSCSI not be slower than the internal drives on Sata3, if I have multiple guest VM's etc?  Currently only 2 bays (mirrored 2x2tb) drives for data storage are occupied in my NAS.

One of the Guest OS VM's is run Windows 7 which has direct PCI passthrough of the Radeon and USB ports.  It runs games well.

If its going to be slow, how about booting off USB and storing the VM on the internal drives, and backup the VM to NAS?

[grasshopper]

jabz10 wrote:

One of the Guest OS VM's is run Windows 7 which has direct PCI passthrough of the Radeon and USB ports.  It runs games well.

Nice!  Yeah if you are gaming with that VM the performance would be less than desirable based on the drives in that external unit.  Of course, with the right external device and disks (i.e. SSD, etc.) you would be fine with the vmdk's remote over iSCSI or NFS.

Your last idea was a good one.  If you perform consistent backups (i.e. VEEAM Free Backup) to that external NAS device you would be safe to keep using the internal disk on your MAC going forward.

Let us know how the copy job goes with vmkfstools.

[jabz10]

Thanks, yea I will try the copy job over the weekend.  Do you have any links to how to create the bootable USB for the mac pro?  ( I will have to get a internal USB card for PCI Passthrough since I am currently using the onboard ones)

[grasshopper]

Hi jabz10,

TBH, I would not worry about USB booting ESXi.  Although I mentioned it as an option, it may be overkill.  The ESXi hypervisor is really lightweight and truly has become a disposable appliance.  If it dies, just rebuild and point to the VMs on your NAS again.

If you do want to play around for geek factor, you can use something like the free Unetbootin utility which is pretty popular for creating bootable USB images.  Also notable, you should choose a decent USB stick that's at least 16GB which gives the system the required space to save a core dump in the event the host experiences a PSoD down the road.  Also, USB sticks that are less than 8GB in size cause issues with the /scratch partition.  More details available at VMware KB2004784.

Edit1:  Added USB size requirements and related KB

Edit 2: Also check out this thread about Rufus which is another popular tool for creating bootable USB