Esxi 6.7 No more Datastore after Raid 1+0 rebuild (HP micro server Gen 8)

[sebaheretic]

Hi Everybody ,

i need help to connect again my datastore at ESXI 6.7

After a HDD fail in my Raid 1+0  in my (HP micro server Gen 😎  i have no more datastore connected i can look at harddrives and i see the the drive with partition in web interface and also in

CLI

[root@xxxxx:~] esxcfg-scsidevs -m

[root@xxxxxx:~] cd /dev/disks

[root@xxxxx:/dev/disks] ls -lisa | grep -v vml

total 3918219898

    171 1953458904 -rw-------    1 root     root     2000341917696 May 13 17:54 naa.600508b100xxxxxx55d9cdfxxxxxxxxx

    169 1953456128 -rw-------    1 root     root     2000339075584 May 13 17:54 naa.600508b100xxxxxx55d9cdfxxxxxxxxx:1

how can i fix this  ?

Thank you all  in advance !

[SureshKumarMuth]

What does vmkernel log say ? Are you getting any error in vmkernel logs related to the lun id (naa.xxxxxxx)

[continuum]

First of all check wether the VMFS-volume is detected as a snapshot.

If yes - follow this knowledgebase article:

VMware Knowledge Base

If that does not help - read
Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

If you provide sucha dump I maybe able to help you recover your VMs.

But first check the KB - if you are lucky that should help you to mount the datastore again.

Ulli

[sebaheretic]

thats is all i have in vmkernel.log about naa

Z cpu0:2097463)ScsiDeviceIO: 9297: Get VPD 86 Inquiry for device "naa.60xxxxxxxxxxxxxxxxxxxxb80f" from Plugin "NMP" failed. Not supported

2019-05-13T17:51:55.122Z cpu0:2097463)ScsiDeviceIO: 7998: QErr is correctly set to 0x0 for device naa.6xxxxxxxxxxxxxxxxxxxxxxxxf.

2019-05-13T17:51:55.146Z cpu0:2097463)ScsiDeviceIO: 8495: Could not detect setting of sitpua for device naa.60xxxxxxxxxxxxxxxxxxxxxxxxx0f. Error Failure.

2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 40, Parameter: 0x4302b4db7180, Registered!

2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 200, Parameter: 0x4302b4db7180, Registered!

2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 800, Parameter: 0x4302b4db7180, Registered!

2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 400, Parameter: 0x4302b4db7180, Registered!

2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 8, Parameter: 0x4302b4db7180, Registered!

2019-05-13T17:51:55.313Z cpu0:2097463)ScsiDevice: 4944: Successfully registered device "naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx80f" from plugin "NMP" of type 0

2019-05-13T17:51:55.313Z cpu0:2097463)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 180, Parameter: 0x430869010010, Registered!

2019-05-13T17:51:55.400Z cpu0:2097463)ScsiUid: 276: Path 'vmhba32:C0:T0:L0' does not support VPD Device Id page.

Z cpu0:2097668)ScsiDeviceIO: 3015: Cmd(0x459a40bf28c0) 0x85, CmdSN 0x0 from world 2098977 to dev "naa.60xxxxxxxxxxxxxxxxxxxxxx0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

thank you

[sebaheretic]

I'm sorry about my lack of knowledge  ,

how can i check if there is a snapshot about my harddrive ?

naa.6xxxxxxxxxxxxxxxxxxxxxx                         vml.0xxx

naa.6xxxxxxxxxxxxxxxxxxxf:1                        vml.0xxx:1

thank you

[sebaheretic]

Ps ,

when Esxi is booted  i can notice a really heavy hard drives activity but i don't know why .

HDD led is  always blinking

thank you

[continuum]

> how can i check if there is a snapshot about my harddrive ?
read the knowledgebase article I linked.

The esxcli command is used on the command line. To list the volumes detected as snapshots, run this command:

# esxcli storage vmfs snapshot list

[sebaheretic]

sadly no output with that command

+

[root@:~] esxcli storage vmfs snapshot list

[root@dae:~]

thank you

[continuum]

Not good ....

Please read the instructions
Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

and create a dump like described.

[sebaheretic]

thank you , i was like i imagine .

Now i have another problem, if i create the dump with DD i just have a 256Mb file 100% due to no space left in the filesystem ( i can say that because if i run dd twice the second time creat almost 0kb file casenamexx )

how can i save the dd command in another path like an usb  or network path ?

thank you

[continuum]

first do

cd /tmp

rm *.1536

then use the option with gzip

dd if=/dev/disks/Device:1 bs=1M count=1536 | gzip -c >  /tmp/sebastiano.1536.gz

or use another datastore if available

dd if=/dev/disks/Device:1 bs=1M count=1536 | gzip -c >  /vmfs/volumes/unaffected-datastore/sebastiano.1536.gz

[sebaheretic]

thank you i will connect another disk in the internal sata slot and i will create a new datastore to save the DD

just 2 quick question before i prepare the temp datastore

1) why the activity of the disks is so high ? i  see (and i hear ) the disk reads/writing all the time , also led hd activity is alway blinking ? (i have only the raid 10 and the micro sd with esxi installed )

2)what could be the cause of this snapshot missing and datastore missing situation ? maybe a boot esxi before the raid rebuild ?

thank you

[sebaheretic]

Hi continuum,

i have the dd dump but inside there a lot of data and information in  clear txt that i can not share  like password ecc.

which part of the file you need ? or can i recreate the snap my my own ?

sorry fo the inconvenient

thank you

ps i can just read the wmfs partition and backup my data al re-create the datastore   ?

[continuum]

> sorry fo the inconvenient

No need to aplogize - for me this just means that I do not  spend / waste 1 - 2 hours while analysing your chances to recover your data.
Maybe I even manage to go to bed before 3 or 4 tonight.

> which part of the file you need ?
All of it.

Datastores that suddenly disappear usually are caused by:

- power failure destroys the VMFS metadata which is used to store the location of all vmdk-fragments stored on that datastore

- misbehaving RAID-controllers

- bad decisions of local admins trying to deal with Raid-controllers error messages or warnings

I use the dump-file to

- recover vmx-files and vmdk-descriptorfiles

- check if there is any vmfs-metadata left to find the locations of vmdk-fragments (this is required to recover flat, delta and sesparse vmdk - all the big files)

If I am successful I provide vmx-files and vmdk-descriptorfiles and create sh-scripts
that will extract the lost vmdk-fragments and assembles them to new VMDKs on another datastore.

Your other options are:

- call vmware support - they will tell you that you have a recovery-request and that they do not help with recovery-requests

- try commercial recovery tools such as Diskinternals and UFSexplorer (this is the complete list of useful tools - both charge 500 - 1000 dollars and both will have poor results for the first-time user )

- call Ontrack - and prepare for a 10.000 dollar bill

- call HP-support - may help to fix the current problems but will probably be too late to fix the VMFS-volume.

- try freeware recovery tools - none of them is designed for this job so you can only recover files by signatures - typical result when searching for database files: 10.000 false detections, 20.000 corrupt files , handful of good files - but all too old ,,,,

- try to google for instructions - well this is probably the worst thing you can do: 99,9 % chance to fail during the first 10 attempts

- go to your boss and report that no files have been shared with some freak in germany and that also nobody else will ever be able to read that data as it is finally gone forever.
You see - there really is no need to apologize - your decision is only inconvenient for you.

I recommend that you call HP-support next.

Call me via skype if you have further questions.
Good night and good luck

Ulli