Error accessing ESXI 6.5 datastore

[Nerumph]

So recently I have been trying to get a Dell r710 up and running for a VM homelab server. I have, in the past, had great success in implementing one of these servers into spec for my purposes, and have been wanting to increase my available testbenches with this new VM host.

Unfortunately, I have been running aground with regards to the storage part of this. More specifically, the inability to access any file structure within the datastore. Symptoms I have been experiencing include inability to access anything through the filebrowser, though only part of the time. What I mean by this is, I can log in, go to the file browser and actually see the directory structure. But if I close the window, go to any other tab in the web UI and then return back to the datastore browser, I get a "Error, please try again later" message popup.

I also have found it impossible to create VM instances. Partially due to the fact that I am unable to see my ISO files to create said VM's, partially because it doesn't seem to be creating anything on the datastore upon hitting "complete" on the blank VM.

I suppose the first step here would be to look through the logs to see what is going on? Does anyone have any advice as to which logset I should be looking at for datastore issues such as this? I tried looking through the logs section under monitor and have found myself a tad overwhelmed by the obscurity of the specific log levels.

[Nerumph]

Ok, so I did a little more digging around and came across the Kernel logs, which ended up shedding some enlightening information about the creation and execution of my disks:

2018-02-02T07:56:59.781Z cpu8:68471)ScsiDeviceIO: 2948: Cmd(0x439d0023f080) 0x1a, CmdSN 0x24dc from world 0 to dev "naa.60026b905201a5002206c80d09ec2613" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

This was of quite some interest. I was unable to find out what the hex values meant as I couldn't find the 6.5 diag for this.

In addition, this is the full log from when I had tried to create a new file system on a single RAID 0 disk (dissolved the previous RAID 6 config I had and instead passed the disks to VMWare on a 1:1 basis.)

2018-02-02T07:56:56.107Z cpu11:67556)ScsiDeviceIO: 2948: Cmd(0x439d003a3280) 0x1a, CmdSN 0x2475 from world 0 to dev "naa.60026b905201a5002206c81c0aca3477" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:56:59.781Z cpu8:68471)ScsiDeviceIO: 2948: Cmd(0x439d0023f080) 0x1a, CmdSN 0x24dc from world 0 to dev "naa.60026b905201a5002206c80d09ec2613" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:57:00.889Z cpu1:65566)ScsiDeviceIO: 2948: Cmd(0x43950060f200) 0x1a, CmdSN 0x251f from world 0 to dev "t10.SanDisk00Cruzer_Glide000000004C530001021216103010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:57:01.562Z cpu8:67646 opID=a9a8e467)World: 12230: VC opID 507aad8a maps to vmkernel opID a9a8e467

2018-02-02T07:57:01.562Z cpu8:67646 opID=a9a8e467)VC: 4511: Device rescan time 369 msec (total number of devices 3)

2018-02-02T07:57:01.562Z cpu8:67646 opID=a9a8e467)VC: 4514: Filesystem probe time 1412 msec (devices probed 3 of 3)

2018-02-02T07:57:01.562Z cpu8:67646 opID=a9a8e467)VC: 4516: Refresh open volume time 0 msec

2018-02-02T07:57:34.830Z cpu3:67557 opID=72cb2756)World: 12230: VC opID 507aadbd maps to vmkernel opID 72cb2756

2018-02-02T07:57:34.830Z cpu3:67557 opID=72cb2756)ScsiDeviceIO: 2948: Cmd(0x439500656e00) 0x1a, CmdSN 0x262a from world 0 to dev "naa.60026b905201a5002206c7fc08e99760" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:57:50.538Z cpu0:67642 opID=e1c3eaba)World: 12230: VC opID 507aadc0 maps to vmkernel opID e1c3eaba

2018-02-02T07:57:50.538Z cpu0:67642 opID=e1c3eaba)LVM: 10276: Initialized naa.60026b905201a5002206c7fc08e99760:1, devID 5a7419fe-ab3f9348-2433-782bcb641e9b

2018-02-02T07:57:50.541Z cpu0:67642 opID=e1c3eaba)LVM: 10370: Zero volumeSize specified: using available space (1999826235904).

2018-02-02T07:57:50.889Z cpu0:67642 opID=e1c3eaba)FS3: 183: <START fbb>

2018-02-02T07:57:50.889Z cpu0:67642 opID=e1c3eaba)1906944 resources, each of size 0

2018-02-02T07:57:50.889Z cpu0:67642 opID=e1c3eaba)Organized as 149 CGs, 64 C/CG and 200 R/C

2018-02-02T07:57:50.889Z cpu0:67642 opID=e1c3eaba)CGsize 65536. 0th CG at 65536.

2018-02-02T07:57:50.889Z cpu0:67642 opID=e1c3eaba)FS3: 185: <END fbb>

2018-02-02T07:57:50.929Z cpu0:67642 opID=e1c3eaba)FS3: 183: <START fdc>

2018-02-02T07:57:50.929Z cpu0:67642 opID=e1c3eaba)130000 resources, each of size 2048

2018-02-02T07:57:50.929Z cpu0:67642 opID=e1c3eaba)Organized as 11 CGs, 64 C/CG and 200 R/C

2018-02-02T07:57:50.929Z cpu0:67642 opID=e1c3eaba)CGsize 26279936. 0th CG at 65536.

2018-02-02T07:57:50.929Z cpu0:67642 opID=e1c3eaba)FS3: 185: <END fdc>

2018-02-02T07:57:50.939Z cpu0:67642 opID=e1c3eaba)FS3: 183: <START pbc>

2018-02-02T07:57:50.939Z cpu0:67642 opID=e1c3eaba)64512 resources, each of size 4096

2018-02-02T07:57:50.939Z cpu0:67642 opID=e1c3eaba)Organized as 63 CGs, 64 C/CG and 16 R/C

2018-02-02T07:57:50.939Z cpu0:67642 opID=e1c3eaba)CGsize 4259840. 0th CG at 65536.

2018-02-02T07:57:50.939Z cpu0:67642 opID=e1c3eaba)FS3: 185: <END pbc>

2018-02-02T07:57:50.958Z cpu0:67642 opID=e1c3eaba)FS3: 183: <START sbc>

2018-02-02T07:57:50.958Z cpu0:67642 opID=e1c3eaba)32000 resources, each of size 8192

2018-02-02T07:57:50.958Z cpu0:67642 opID=e1c3eaba)Organized as 8 CGs, 64 C/CG and 64 R/C

2018-02-02T07:57:50.958Z cpu0:67642 opID=e1c3eaba)CGsize 33619968. 0th CG at 65536.

2018-02-02T07:57:50.958Z cpu0:67642 opID=e1c3eaba)FS3: 185: <END sbc>

2018-02-02T07:57:50.966Z cpu0:67642 opID=e1c3eaba)FS3: 183: <START pb2>

2018-02-02T07:57:50.966Z cpu0:67642 opID=e1c3eaba)256 resources, each of size 4096

2018-02-02T07:57:50.966Z cpu0:67642 opID=e1c3eaba)Organized as 1 CGs, 64 C/CG and 16 R/C

2018-02-02T07:57:50.966Z cpu0:67642 opID=e1c3eaba)CGsize 4259840. 0th CG at 65536.

2018-02-02T07:57:50.966Z cpu0:67642 opID=e1c3eaba)FS3: 185: <END pb2>

2018-02-02T07:57:50.972Z cpu0:67642 opID=e1c3eaba)Res3: 6728: SDDir: type: 0x5, fileLength: 0x118, numBlocks: 1

2018-02-02T07:57:50.973Z cpu0:67642 opID=e1c3eaba)Vol3: 1268: Created VMFS-5.81 with config 0x6 on vol 'SomethingFresh'

2018-02-02T07:57:50.974Z cpu3:65737)ScsiDeviceIO: 2948: Cmd(0x439500691f00) 0x1a, CmdSN 0x26a6 from world 0 to dev "naa.60026b905201a5002206c7fc08e99760" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:57:50.976Z cpu0:67642 opID=e1c3eaba)FSS: 5749: No FS driver claimed device 'naa.60026b905201a5002206c7fc08e99760:1': No filesystem on the device

2018-02-02T07:57:51.001Z cpu0:67642 opID=e1c3eaba)LVM: 16581: File system '[SomethingFresh, 5a7419fe-ee26e3b3-cfa9-782bcb641e9b]' (LV 5a7419fe-a21f4b00-06bc-782bcb641e9b) mounted in 'rw' mode.

2018-02-02T07:57:51.050Z cpu11:66147)ScsiDeviceIO: 2948: Cmd(0x439d0031b500) 0x1a, CmdSN 0x2730 from world 0 to dev "naa.60026b905201a5002206c7cc0611c58d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:57:51.416Z cpu3:66153)ScsiDeviceIO: 2948: Cmd(0x4395007b6a00) 0x1a, CmdSN 0x2752 from world 0 to dev "naa.60026b905201a5002206c7fc08e99760" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:57:52.063Z cpu1:65566)ScsiDeviceIO: 2948: Cmd(0x439d0028a100) 0x1a, CmdSN 0x2784 from world 0 to dev "t10.SanDisk00Cruzer_Glide000000004C530001021216103010" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:57:52.920Z cpu0:67642 opID=cbcd5d0d)World: 12230: VC opID 507aadc1 maps to vmkernel opID cbcd5d0d

2018-02-02T07:57:52.920Z cpu0:67642 opID=cbcd5d0d)VC: 4511: Device rescan time 459 msec (total number of devices 4)

2018-02-02T07:57:52.920Z cpu0:67642 opID=cbcd5d0d)VC: 4514: Filesystem probe time 1411 msec (devices probed 4 of 4)

2018-02-02T07:57:52.920Z cpu0:67642 opID=cbcd5d0d)VC: 4516: Refresh open volume time 0 msec

2018-02-02T07:57:53.494Z cpu0:65974)LVM: 16600: One or more LVM devices have been discovered.

2018-02-02T07:58:34.031Z cpu7:66158)ScsiDeviceIO: 2948: Cmd(0x439d00234c00) 0x1a, CmdSN 0x284a from world 0 to dev "naa.60026b905201a5002206c7fc08e99760" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-02-02T07:58:34.031Z cpu7:66158)ScsiDeviceIO: 2948: Cmd(0x439d00234c00) 0x1a, CmdSN 0x284a from world 0 to dev "naa.60026b905201a5002206c7fc08e99760" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

I don't really have the resources to make sense of such things, so if someone could help with sourcing these, I would appreciate it.

A little more about my config before I send this off to you all. I am (again) running a Dell r710. I am currently booting ESXI off a Cruiser flashdrive from the internal port. All RAID and disk pairing services are being ran through the perc6i. Though initially I was a bit impatient and didn't let the array go through its initialization and integrity checks properly, with both the single-disk 1:1 and upon reforming the disks into a RAID 6 over the last couple of nights (which still nets the same results) I let the init and verification run to its completion.

Again, any assistance would be greatly appreciated here.

[daphnissov]

It looks like, in your case, the block device that is reporting those sense codes cannot respond to an inquiry for standard SCSI VPDs. This doesn't necessarily indicate something is wrong with the device, but it is curious. Can you please state, in detail, the information about your R710 with regard to the storage subsystem? What drive is this, what firmware, what bus type?

Also, is this a standalone ESXi host or is it joined to vCenter? If it's standalone, please show CLI output of esxcli software vib list | grep esx-ui.

[Nerumph]

First of all, thank you for your response! I did want to clarify what it is you are asking before I go out and end up seeking irrelevant information.

In regards to firmware, are you asking for the firmware of the iperc itself, or the BIOS, or both?

Drive wise, I'm assuming you'd want model number yea?

And bus type. Are you talking about interface bus type (IE: SAS, SATA, iscsi)?

[daphnissov]

I would like to know what storage controller you're using, what the firmware of that is, what drive(s) you're using, the configuration of those, the firmware of those, and any other relevant information you wish to volunteer.

[Nerumph]

Hey there. So I finally managed to get a couple pieces of information that you had requested the other day. My apologies for the delay.

Disks are in a RAID-6. They are all Seagate drives, different family of drives but the model numbers are all the same. Four Desktop ST2000DM001 and one Barracuda ST2000DM001. I was unable to retrieve the firmware version of the drives. I did try RE: How to Identify PERC controller Firmware in ES... - Dell Community to try and acquire the drive information; however was unable to get anything useful.

Upon entering this line:

/usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval  -d

I received this as a result:

"Dumping all key-value instance names:

No key-value instances registered."

RAID controller is: Dell PERC 6/i. Firmware version of: 1.22.52-1909.

I feel like this could be updated? Is there a way to do this in ESXI by chance?

Finally, the command that you asked me to enter before returned:

"esx-ui                         1.8.0-4516221                         VMware  VMwareCertified   2018-02-03"

[daphnissov · Accepted solution]

Update your PERC 6/i firmware, which, btw, that card is a piece of junk so do not expect stellar performance. Support for PowerEdge R710 | Drivers & downloads | Dell US

Also, your ESXi embedded host client is quite out of date. Update to the latest one from the Fling site here.

[Nerumph]

Ok, so the solution seems to have been to update ESXi to update one. That has seemed to help me with the RAID side of things. Thank you daphnissov for your assistance!