VMware Cloud Community
Day3_NOC
Contributor
Contributor

LUN not accessible vsphere

We have a Synolgy SAN. It was connected to an ESXi host. After a power outage things got back into operation but the LUN did not get connected to the ESXi. We did try several things:

1. Rescan the software HBA to discover the datastore.

2. Reboot the ESXI host and then rescan the software HBA.

3. After the reboot the LUNs get detected and gets attached to the ESXI but when we try to browse the LUN through the vsphere client we do not see any contents inside the store. Although the volume of the LUN has data inside it as we can see through the Synology.

How do we get the see the contents inside the LUN?

Any help will be appreciated

Reply
0 Kudos
7 Replies
a_p_
Leadership
Leadership

Please clarify stpe 3. Do you only see the LUN under "Storage Adapters", or do you see the datastore under "Storage", but it doesn't show data anymore?

In case you can see the LUN, but the datastore isn't mounted, follow the "Command Line" steps from https://kb.vmware.com/kb/1011387 (i.e. run esxcli storage vmfs snapshot list) to see whether the LUN has been detected as a Snapshot LUN, and can be mounted manually.

André

Reply
0 Kudos
Day3_NOC
Contributor
Contributor

HI a.p,

By Step-3, I mean I had restarted the ESXi host and tried to reconnect the LUN (with its iqn number given in the SAN). The LUN gets connected to the esxi host but this LUN did have my VM's and as I use the vsphere client to browse through the LUN it shows me moting. It get detected as a new LUN without display the contents.

Following is the log that I get from /var/log/vmkernel.log

####

2016-08-14T03:24:31.844Z cpu3:33597)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:25:01.845Z cpu5:38173)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:25:31.845Z cpu5:38173)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:26:01.846Z cpu5:38172)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:26:31.847Z cpu4:33597)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:26:38.174Z cpu2:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.249Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.356Z cpu0:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.356Z cpu0:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.356Z cpu0:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:38.357Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.366Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.368Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.370Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.372Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.374Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.378Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:38.424Z cpu1:34366)VC: 1958: Device rescan time 92 msec (total number of devices 10)

2016-08-14T03:26:38.424Z cpu1:34366)VC: 1961: Filesystem probe time 143 msec (devices probed 7 of 10)

2016-08-14T03:26:38.424Z cpu1:34366)VC: 1963: Refresh open volume time 13 msec

2016-08-14T03:26:38.429Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.462Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.482Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.487Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.489Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.491Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.493Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.495Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.499Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:38.499Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.499Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.499Z cpu1:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:38.584Z cpu7:34366)VC: 1958: Device rescan time 44 msec (total number of devices 10)

2016-08-14T03:26:38.584Z cpu7:34366)VC: 1961: Filesystem probe time 107 msec (devices probed 7 of 10)

2016-08-14T03:26:38.584Z cpu7:34366)VC: 1963: Refresh open volume time 2 msec

2016-08-14T03:26:38.654Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.686Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.753Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.753Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.753Z cpu1:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:38.753Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.759Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.761Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.763Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.765Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.767Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.771Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:38.848Z cpu4:34366)VC: 1958: Device rescan time 79 msec (total number of devices 10)

2016-08-14T03:26:38.848Z cpu4:34366)VC: 1961: Filesystem probe time 112 msec (devices probed 7 of 10)

2016-08-14T03:26:38.848Z cpu4:34366)VC: 1963: Refresh open volume time 2 msec

2016-08-14T03:26:41.731Z cpu1:35082)LVM: 13082: One or more LVM devices have been discovered.

2016-08-14T03:26:55.336Z cpu0:34366)LVM: 7525: Initialized naa.600140574ab763dde2dfd493adb910de:1, devID 57afe4ff-89c5a615-b2da-0026b94357b6

2016-08-14T03:26:55.343Z cpu4:34366)LVM: 7613: Zero volumeSize specified: using available space (2060491668992).

2016-08-14T03:26:56.717Z cpu5:34366)FS3: 224: <START fbb>

2016-08-14T03:26:56.717Z cpu5:34366)1964800 resources, each of size 0

2016-08-14T03:26:56.717Z cpu5:34366)Organized as 154 CGs, 64 C/CG and 200 R/C

2016-08-14T03:26:56.717Z cpu5:34366)CGsize 65536. 0th CG at 65536

2016-08-14T03:26:56.717Z cpu5:34366)FS3: 226: <END fbb>

2016-08-14T03:26:56.939Z cpu3:34366)FS3: 224: <START fdc>

2016-08-14T03:26:56.939Z cpu3:34366)130000 resources, each of size 2048

2016-08-14T03:26:56.939Z cpu3:34366)Organized as 11 CGs, 64 C/CG and 200 R/C

2016-08-14T03:26:56.939Z cpu3:34366)CGsize 26279936. 0th CG at 65536

2016-08-14T03:26:56.939Z cpu3:34366)FS3: 226: <END fdc>

2016-08-14T03:26:56.980Z cpu1:34366)FS3: 224: <START pbc>

2016-08-14T03:26:56.980Z cpu1:34366)64512 resources, each of size 4096

2016-08-14T03:26:56.980Z cpu1:34366)Organized as 63 CGs, 64 C/CG and 16 R/C

2016-08-14T03:26:56.980Z cpu1:34366)CGsize 4259840. 0th CG at 65536

2016-08-14T03:26:56.980Z cpu1:34366)FS3: 226: <END pbc>

2016-08-14T03:26:57.088Z cpu0:34366)FS3: 224: <START sbc>

2016-08-14T03:26:57.088Z cpu0:34366)32000 resources, each of size 8192

2016-08-14T03:26:57.088Z cpu0:34366)Organized as 8 CGs, 64 C/CG and 64 R/C

2016-08-14T03:26:57.088Z cpu0:34366)CGsize 33619968. 0th CG at 65536

2016-08-14T03:26:57.088Z cpu0:34366)FS3: 226: <END sbc>

2016-08-14T03:26:57.144Z cpu3:34366)FS3: 224: <START pb2>

2016-08-14T03:26:57.144Z cpu3:34366)256 resources, each of size 4096

2016-08-14T03:26:57.144Z cpu3:34366)Organized as 1 CGs, 64 C/CG and 16 R/C

2016-08-14T03:26:57.144Z cpu3:34366)CGsize 4259840. 0th CG at 65536

2016-08-14T03:26:57.144Z cpu3:34366)FS3: 226: <END pb2>

2016-08-14T03:26:57.173Z cpu0:34366)Res3: 5734: SDDir: type: 0x5, fileLength: 0x118, numBlocks: 1

2016-08-14T03:26:57.176Z cpu0:34366)Vol3: 3434: Created VMFS-5.60 with config 0x16 on vol 'iSCSI-Mel-Black-SAN-1-Ergo-VM'

2016-08-14T03:26:57.176Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.228Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.381Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.386Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.388Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.390Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.392Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.394Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.398Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:57.398Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:57.398Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:57.398Z cpu1:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:57.415Z cpu1:34366)VC: 1958: Device rescan time 68 msec (total number of devices 10)

2016-08-14T03:26:57.415Z cpu1:34366)VC: 1961: Filesystem probe time 153 msec (devices probed 7 of 10)

2016-08-14T03:26:57.415Z cpu1:34366)VC: 1963: Refresh open volume time 15 msec

2016-08-14T03:26:57.514Z cpu1:34366)Vol3: 661: Unable to register file system iSCSI-Mel-Black-SAN-1-Ergo-VM for APD timeout notifications: Already exists

2016-08-14T03:26:57.514Z cpu1:34366)LVM: 13062: File system '[iSCSI-Mel-Black-SAN-1-Ergo-VM, 57afe500-482444ee-1305-0026b94357b6]' (LV 57afe4ff-5820eaf4-7436-0026b94357b6) mounted in 'rw' mode.

2016-08-14T03:26:58.341Z cpu0:38173)LVM: 13082: One or more LVM devices have been discovered.

####

Just to mention. I am running VMware ESXi 5.5.0 and there are no snapshots in any of the VM's.

I just want to reconnect the LUN to the host and see its contents. SAN volume shows that it has data inside it.

The naa-id is naa.600140574ab763dde2dfd493adb910de:1

I ran the command:

esxcli storage vmfs snapshot list

and it does not return anything.

Reply
0 Kudos
continuum
Immortal
Immortal

It happens occasionally that a LUN looks empty after a power failure.
Be careful with your next steps.Probably your VMs are still there and can be extracted if you dont make mistakes.
Please create a VMFS-header dump with
dd if="/dev/disks/naa.600140574ab763dde2dfd493adb910de:1" bs=1M count=1536 of=/vmfs/volumes/some-other-datastore/day3_noc.1536
Download the *1536 file, compress it and provide a download.
With that info we can tell you how serious it is.
Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
Day3_NOC
Contributor
Contributor

Hi continuum,

I have attached the compressed file. Please have a look and let me know if you come up with anything. And thanks for your suggestion, really appreciate that.

Reply
0 Kudos
continuum
Immortal
Immortal

Hi
sorry I dont have much time at the moment - I suggest that you call me via skype in a few hours.
I think I found

Ergo VM Server-flat.vmdk

Server 2008 R2 x64bit-flat.vmdk

Server 2008 R2 x64bit_1-flat.vmdk
with descriptorfiles.
The Ergo VM seems to have a snapshot which maybe recoverable.
All 3 vmdks are highly fragmented: 120.000 gragments, 80.000 fragments and 55.000 fragments so carving them out is not trivial - but possible.
Two 200gb vmdks and one of 150gb.
There maybe one more vmdk of about 200gb but thats inconclusive - ask me later.

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
Day3_NOC
Contributor
Contributor

Hi,

I have attached data dump of another LUN as per our discussion. Please have a look.

Reply
0 Kudos
continuum
Immortal
Immortal

I may be able to extract CN-MEL-BB-WinShare_1-flat.vmdk. (4tb)
and CN-MEL-BB-WinShare-flat.vmdk

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=973d99a0

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 31457280 VMFS "CN-MEL-BB-WinShare-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "1958"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "f12aad0d484b07c7e403bac2973d99a0"

ddb.toolsVersion = "9344"

ddb.uuid = "60 00 C2 9f 15 4f ed 6b-22 a5 df bc 04 fc b9 4d"

ddb.virtualHWVersion = "8"

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=232ce3d9

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 8388608000 VMFS "CN-MEL-BB-WinShare_1-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "522166"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "7ff6ee3f0705632c582244a4232ce3d9"

ddb.toolsVersion = "9344"

ddb.uuid = "60 00 C2 95 b9 57 74 c6-77 ad e6 c6 b5 70 70 83"

ddb.virtualHWVersion = "8"


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos