VMware Global Community
johinVM
Contributor
Contributor

Missing datastore after powerfail, Datastore ist verschwunden

Hallo

Nach einem Stromausfall ist Datastore (/vmfs/volumes/5b045940-70f55f40-02e4-f8db88fc6e18/) verschwunden.

vmkernel.log:

....

2020-02-04T17:45:39.953Z cpu0:65888)Activating Jumpstart plugin mount-disk-fs.

2020-02-04T17:45:40.004Z cpu0:65681)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba0:C0:T4:L0

2020-02-04T17:45:40.004Z cpu0:65681)FSS: 5763: No FS driver claimed device 'mpx.vmhba0:C0:T4:L0': No filesystem on the device

2020-02-04T17:45:40.006Z cpu0:65682)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba1:C0:T0:L0:2

2020-02-04T17:45:40.006Z cpu2:65677)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba1:C0:T0:L0:5

2020-02-04T17:45:40.008Z cpu4:65676)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba1:C0:T0:L0:6

2020-02-04T17:45:40.008Z cpu5:65679)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba1:C0:T0:L0:8

2020-02-04T17:45:40.040Z cpu0:65870)VC: 4511: Device rescan time 50 msec (total number of devices 5)

2020-02-04T17:45:40.040Z cpu0:65870)VC: 4514: Filesystem probe time 36 msec (devices probed 5 of 5)

2020-02-04T17:45:40.040Z cpu0:65870)VC: 4516: Refresh open volume time 0 msec

2020-02-04T17:45:40.167Z cpu0:65870)WARNING: NFS: 1183: Invalid volume UUID 5b04593c-34f29520-afe2-f8db88fc6e18

2020-02-04T17:45:40.396Z cpu5:65780)NMP: nmp_ThrottleLogForDevice:3616: Cmd 0x28 (0x439500bdab80, 65869) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Act:NONE

2020-02-04T17:45:40.396Z cpu5:65780)ScsiDeviceIO: 3015: Cmd(0x439500bdab80) 0x28, CmdSN 0x17 from world 65869 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.

2020-02-04T17:45:40.396Z cpu0:65870)WARNING: HBX: 2355: Failed to initialize VMFS distributed locking on volume 5b045940-70f55f40-02e4-f8db88fc6e18: I/O error

2020-02-04T17:45:40.483Z cpu0:65870)FSS: 5763: No FS driver claimed device '5b04593c-34f29520-afe2-f8db88fc6e18': No filesystem on the device

2020-02-04T17:45:40.483Z cpu0:65870)LVM: 16602: File system '' (LV 5b04593c-34f29520-afe2-f8db88fc6e18) mounted in 'rw' mode.

2020-02-04T17:45:40.484Z cpu7:65888)Jumpstart plugin mount-disk-fs activated.

....

Wie kann des Volume repariert werden?

Nach dem Befehl " vmfstools -V " winrd folgendes dem vmkernel.log hinzugefügt:

2020-02-04T18:55:04.753Z cpu4:65676)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba1:C0:T0:L0:5

2020-02-04T18:55:04.753Z cpu2:68809)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba1:C0:T0:L0:6

2020-02-04T18:55:04.755Z cpu0:68829)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba1:C0:T0:L0:8

2020-02-04T18:55:05.015Z cpu3:68808)WARNING: NFS: 1183: Invalid volume UUID mpx.vmhba0:C0:T4:L0

2020-02-04T18:55:05.478Z cpu3:68808)FSS: 5763: No FS driver claimed device 'mpx.vmhba0:C0:T4:L0': No filesystem on the device

2020-02-04T18:55:05.479Z cpu4:68826)VC: 4511: Device rescan time 81 msec (total number of devices 6)

2020-02-04T18:55:05.479Z cpu4:68826)VC: 4514: Filesystem probe time 728 msec (devices probed 5 of 6)

2020-02-04T18:55:05.479Z cpu4:68826)VC: 4516: Refresh open volume time 0 msec

2020-02-04T18:55:05.530Z cpu7:66385)NMP: nmp_ThrottleLogForDevice:3616: Cmd 0x28 (0x439500bf0980, 66949) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Act:NONE

2020-02-04T18:55:05.530Z cpu7:66385)ScsiDeviceIO: 3015: Cmd(0x439500bf0980) 0x28, CmdSN 0x17 from world 66949 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.

2020-02-04T18:55:05.530Z cpu3:67653)WARNING: HBX: 2355: Failed to initialize VMFS distributed locking on volume 5b045940-70f55f40-02e4-f8db88fc6e18: I/O error

2020-02-04T18:55:05.530Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 1 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 0 gen 0 :I/O error

2020-02-04T18:55:05.530Z cpu3:67653)WARNING: Fil3: 1389: Failed to reserve volume f530 28 1 5b045940 70f55f40 dbf802e4 186efc88 0 0 0 0 0 0 0

2020-02-04T18:55:05.530Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 2 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 4 gen 1 :I/O error

2020-02-04T18:55:05.581Z cpu7:66385)ScsiDeviceIO: 3015: Cmd(0x439500b45500) 0x28, CmdSN 0x17 from world 66949 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.

2020-02-04T18:55:05.581Z cpu3:67653)WARNING: HBX: 2355: Failed to initialize VMFS distributed locking on volume 5b045940-70f55f40-02e4-f8db88fc6e18: I/O error

2020-02-04T18:55:05.581Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 1 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 0 gen 0 :I/O error

2020-02-04T18:55:05.581Z cpu3:67653)WARNING: Fil3: 1389: Failed to reserve volume f530 28 1 5b045940 70f55f40 dbf802e4 186efc88 0 0 0 0 0 0 0

2020-02-04T18:55:05.581Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 2 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 4 gen 1 :I/O error

2020-02-04T18:55:05.631Z cpu7:65952)ScsiDeviceIO: 3015: Cmd(0x439500b50b00) 0x28, CmdSN 0x17 from world 66949 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.

2020-02-04T18:55:05.631Z cpu3:67653)WARNING: HBX: 2355: Failed to initialize VMFS distributed locking on volume 5b045940-70f55f40-02e4-f8db88fc6e18: I/O error

2020-02-04T18:55:05.631Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 1 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 0 gen 0 :I/O error

2020-02-04T18:55:05.631Z cpu3:67653)WARNING: Fil3: 1389: Failed to reserve volume f530 28 1 5b045940 70f55f40 dbf802e4 186efc88 0 0 0 0 0 0 0

2020-02-04T18:55:05.631Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 2 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 4 gen 1 :I/O error

2020-02-04T18:55:05.673Z cpu7:65952)ScsiDeviceIO: 3015: Cmd(0x4395009ee300) 0x28, CmdSN 0x17 from world 66949 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.

2020-02-04T18:55:05.673Z cpu3:67653)WARNING: HBX: 2355: Failed to initialize VMFS distributed locking on volume 5b045940-70f55f40-02e4-f8db88fc6e18: I/O error

2020-02-04T18:55:05.673Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 1 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 0 gen 0 :I/O error

2020-02-04T18:55:05.673Z cpu3:67653)WARNING: Fil3: 1389: Failed to reserve volume f530 28 1 5b045940 70f55f40 dbf802e4 186efc88 0 0 0 0 0 0 0

2020-02-04T18:55:05.673Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 2 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 4 gen 1 :I/O error

2020-02-04T18:55:05.715Z cpu7:66385)ScsiDeviceIO: 3015: Cmd(0x4395009f8780) 0x28, CmdSN 0x17 from world 66949 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.

2020-02-04T18:55:05.716Z cpu3:67653)WARNING: HBX: 2355: Failed to initialize VMFS distributed locking on volume 5b045940-70f55f40-02e4-f8db88fc6e18: I/O error

2020-02-04T18:55:05.716Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 1 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 0 gen 0 :I/O error

2020-02-04T18:55:05.716Z cpu3:67653)WARNING: Fil3: 1389: Failed to reserve volume f530 28 1 5b045940 70f55f40 dbf802e4 186efc88 0 0 0 0 0 0 0

2020-02-04T18:55:05.716Z cpu3:67653)Vol3: 3155: Failed to get object 28 type 2 uuid 5b045940-70f55f40-02e4-f8db88fc6e18 FD 4 gen 1 :I/O error

2020-02-04T18:55:42.488Z cpu0:65990)NMP: nmp_ResetDeviceLogThrottling:3458: last error status from device mpx.vmhba1:C0:T0:L0 repeated 4 times

2020-02-04T18:55:42.488Z cpu0:65990)NMP: nmp_ResetDeviceLogThrottling:3452: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x2 0x3a 0x1 from dev "mpx.vmhba0:C0:T4:L0" occurred 5 times(of 5 commands)

2020-02-04T19:00:01.321Z cpu2:65538)NMP: nmp_ThrottleLogForDevice:3616: Cmd 0x28 (0x439500bf6b80, 65848) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Act:NONE

2020-02-04T19:00:01.321Z cpu2:65538)ScsiDeviceIO: 3015: Cmd(0x439500bf6b80) 0x28, CmdSN 0x8b84 from world 65848 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.

Reply
0 Kudos
9 Replies
a_p_
Leadership
Leadership

Willkommen im Forum,

schwer zu sagen, was die Ursache ist.

Liefert der Befehl

esxcli storage vmfs snapshot list

ein Ergebnis?

André

Reply
0 Kudos
johinVM
Contributor
Contributor

Mit Putty seh ich keine Antwort

[root@ESXi:~] esxcli storage vmfs snapshot list

[root@ESXi:~]

und im vmkernel.log auch nicht

Der Datasore enthält eine NM mit dem SQL Server.

Die Sicherung davon ist auf einer anderen VM aber im gleichen Datastore!

Reply
0 Kudos
a_p_
Leadership
Leadership

Ok. das schließt eine sog. "Snapshot LUN" erst mal aus.

Bitte laß nun mal den etwas längeren "offset ...; done" Befehl aus Schritt 1 aus https://kb.vmware.com/s/article/2046610​ laufen, um zu sehen was  damit erkannt wird.

André

Reply
0 Kudos
johinVM
Contributor
Contributor

Hallo André

Das ergibt:

/vmfs/devices/cdrom/mpx.vmhba0:C0:T4:L0

Unable to get device /vmfs/devices/cdrom/mpx.vmhba0:C0:T4:L0

---------------------

/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0

gpt

91135 255 63 1464094720

1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

2 7086080 15472639 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

3 15472640 1464094686 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

---------------------

[root@ESXi:~]

Reply
0 Kudos
a_p_
Leadership
Leadership

Bevor durch Trial&Error eventuell noch mehr kaputt geht, versuch doch mal continuum​ per Skype zu erreichen. Ulli ist hier im Forum der Experte, wenn's um solche Probleme geht.

André

Reply
0 Kudos
johinVM
Contributor
Contributor

Danke André

Reply
0 Kudos
continuum
Immortal
Immortal

Hi Johannes

Wenn ich die Sache richtig verstehe ist die VMFS-Partition

/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3

das verschwundene Datastore.

Kannst du das device ueberhaupt noch per esxi ansprechen ?

Probier bitte mal

cd /tmp

od -A x -N 25m -t x /dev/disks/mpx.vmhba1:C0:T0:L0:3  | grep "00000 abcdef"

Dabei kommt dann entweder ein I/O error raus oder 4 Zeilen die etwa so aussehen sollten.

1400000 abcdef01 00300000 00000000 00000000

1500000 abcdef01 00400000 00000000 00000000

1600000 abcdef01 00500000 00000000 00000000

1700000 abcdef01 00600000 00000000 00000000

Der naechste Schritt haengt davon ab - ob die Partition ansprechbar ist.

Bitte sag Bescheid was sich bei dir ergibt ?

Gruss Ulli

skype : sanbarrow


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
johinVM
Contributor
Contributor

Guten Morgen Ulli

Danke für deine Unterstützung

Der Befehl gibt volgendes Ergebnis:

[root@ESXi:/tmp] od -A x -N 25m -t x /dev/disks/mpx.vmhba1:C0:T0:L0:3  | grep "00000 abcdef"

od: /dev/disks/mpx.vmhba1:C0:T0:L0:3: read error

[root@ESXi:/tmp]

Reply
0 Kudos
johinVM
Contributor
Contributor

Danke Ulli, dass du mir die Daten in mühsamer Kleinarbeit wiederhergestellt hast. Smiley Happy

Reply
0 Kudos