Re: VSAN disk group creation failed

Silvester · ‎08-26-2014

Dear all,

I've created a new cluster and enabled VSAN on it, with 3 hosts. Everything looks fine so far,

However when I select 1 SSD and 4 HDD from one of the hosts and create a disk group, the process failed with the following message:

Operation failed, diagnostics report: Disk with uuid 52b244d6-7cda-613f-46b8-eb786e6207e5 didn't appear in the vsan cluster directory within 10 seconds.

What's interesting is that, there not such a UUID existing on that host.

I've tried disabled/enabled VSAN on the cluster, and create disk group from other hosts within the cluster, with the same result.

Any idea on how to continue the troubleshooting?

Thanks!

Silvester

MightySuite · ‎08-27-2014

Please post the output of vdq -i, vdq -q and esxcfg-scsidevs -l

And what is the network speed of vmkernel nics that you used for vSAN..? Virtual SAN requires a private 1Gb network. As a best practice, use 10Gb network.

Silvester · ‎08-27-2014

I'm using the 10Gb network for the vSAN kernrnel.

The output of vdq -i:

[

{

"SSD" : "naa.600605b005a783701acbdef240ab6fb8",

"MD" : [

"naa.600605b008d44de01b7ea39bda5e9bc3",

"naa.600605b008d44de01b7ea6d30b7c9b9d",

"naa.600605b008d44de01b7e9e25870f7118",

"naa.600605b008d44de01b7eaacb4807dcc5",

]

},

]

Output of vdq -q:

[
   {
      "Name"     : "naa.600605b005a783701acbdef240ab6fb8",
      "VSANUUID" : "52ec0da9-7e39-8749-b729-62a01411c931",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "1",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea39bda5e9bc3",
      "VSANUUID" : "52756247-baf6-c8a1-10f3-2ea0d564e972",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ead416d91ca0f",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "1",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea6d30b7c9b9d",
      "VSANUUID" : "52b244d6-7cda-613f-46b8-eb786e6207e5",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7e9e25870f7118",
      "VSANUUID" : "52416344-3051-95a0-cbd2-884442f5d0d0",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "mpx.vmhba32:C0:T0:L0",
      "VSANUUID" : "",
      "State"    : "Ineligible for use by VSAN",
      "Reason"   : "Has partitions",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7eaacb4807dcc5",
      "VSANUUID" : "52098586-205a-0fe0-f6ce-8fb922daf058",
      "State"    : "In-use for VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7e8a995cc62833",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea64903437c1c",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea89f26e5579e",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7e893b47e3b64f",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7ea80d1e364892",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

   {
      "Name"     : "naa.600605b008d44de01b7eab464f5a234b",
      "VSANUUID" : "",
      "State"    : "Eligible for use by VSAN",
      "Reason"   : "None",
      "IsSSD"    : "0",
      "IsPDL"    : "0",
   },

]

Output of esxcfg-scsidevs

mpx.vmhba0:C0:T19:L0

Device Type: Enclosure Svc Dev

Size: 0 MB

Display Name: Local CISCO Enclosure Svc Dev (mpx.vmhba0:C0:T19:L0)

Multipath Plugin: NMP

Console Device: /vmfs/devices/genscsi/mpx.vmhba0:C0:T19:L0

Devfs Path: /vmfs/devices/genscsi/mpx.vmhba0:C0:T19:L0

Vendor: CISCO Model: UCS 240 Revis: 0809

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.020d000000f40f1b1e5d2600bd554353203234

VAAI Status: unsupported

mpx.vmhba32:C0:T0:L0

Device Type: Direct-Access

Size: 14336 MB

Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

Vendor: HV Model: Hypervisor_0 Revis: 1.01

SCSI Level: 2 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: true

Is Local: true Is SSD: false

Other Names:

vml.0000000000766d68626133323a303a30

VAAI Status: unsupported

naa.600605b005a783701acbdef240ab6fb8

Device Type: Direct-Access

Size: 189781 MB

Display Name: Local LSI Disk (naa.600605b005a783701acbdef240ab6fb8)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b005a783701acbdef240ab6fb8

Devfs Path: /vmfs/devices/disks/naa.600605b005a783701acbdef240ab6fb8

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: true

Other Names:

vml.0200000000600605b005a783701acbdef240ab6fb84d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7e893b47e3b64f

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7e893b47e3b64f)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e893b47e3b64f

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e893b47e3b64f

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7e893b47e3b64f4d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7e8a995cc62833

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7e8a995cc62833)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e8a995cc62833

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e8a995cc62833

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7e8a995cc628334d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7e9e25870f7118

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7e9e25870f7118)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e9e25870f7118

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e9e25870f7118

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7e9e25870f71184d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7ea39bda5e9bc3

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7ea39bda5e9bc3)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea39bda5e9bc3

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea39bda5e9bc3

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7ea39bda5e9bc34d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7ea64903437c1c

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7ea64903437c1c)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea64903437c1c

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea64903437c1c

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7ea64903437c1c4d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7ea6d30b7c9b9d

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7ea6d30b7c9b9d)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea6d30b7c9b9d

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea6d30b7c9b9d

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7ea6d30b7c9b9d4d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7ea80d1e364892

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7ea80d1e364892)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea80d1e364892

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea80d1e364892

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7ea80d1e3648924d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7ea89f26e5579e

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7ea89f26e5579e)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea89f26e5579e

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea89f26e5579e

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7ea89f26e5579e4d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7eaacb4807dcc5

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7eaacb4807dcc5)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7eaacb4807dcc5

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7eaacb4807dcc5

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7eaacb4807dcc54d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7eab464f5a234b

Device Type: Direct-Access

Size: 857353 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7eab464f5a234b)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7eab464f5a234b

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7eab464f5a234b

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: false

Other Names:

vml.0200000000600605b008d44de01b7eab464f5a234b4d5239323731

VAAI Status: unsupported

naa.600605b008d44de01b7ead416d91ca0f

Device Type: Direct-Access

Size: 189781 MB

Display Name: Local LSI Disk (naa.600605b008d44de01b7ead416d91ca0f)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ead416d91ca0f

Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ead416d91ca0f

Vendor: LSI Model: MR9271-8i Revis: 3.40

SCSI Level: 5 Is Pseudo: false Status: on

Is RDM Capable: false Is Removable: false

Is Local: true Is SSD: true

Other Names:

vml.0200000000600605b008d44de01b7ead416d91ca0f4d5239323731

VAAI Status: unsupported

Thanks a lot!

Silvester

MightySuite · ‎08-27-2014

If you see the output of vdq -q you can actually find that VSAN UUID 52b244d6-7cda-613f-46b8-eb786e6207e5, which you mentioned you couldn't find. That is attached to the disk naa.600605b008d44de01b7ea6d30b7c9b9d. So there could be some problem with that disk. To confirm that, do not select this data disk while disk group creation. If that works, then you can be sure there is some problem with that disk.

surprising factor is that, vdq output shows that VSAN cluster creation is successful. You can observe that some of the disk state shows that In-Use for VSAN. Can you also post the output for "esxcli vsan cluster get"

Silvester · ‎08-27-2014

OK, I managed to delete the disk group, and tries to create another disk group, using some other disks than "600605b008d44de01b7ea6d30b7c9b9d", now this time it fails with the following message

"Operation failed, diagnostics report: Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information"

And I notice this time no disk group is created (vdq -i returns nothing).

Silvester

paudieo · ‎08-27-2014

Be nice to see the vmkernel logs..

Can you post?

a vm-support would be great

Thanks

Silvester · ‎08-28-2014

Hi Paudieo

Here is the VMKernel/vmkwarning/vmksummary log files etracted from the ESX host. I reproduced the error twice before downloading the logs.

With my knowledge I was not able to identify anything abnormal. Any hint/suggestions will be much apprecated!

If you need other log files, please feel free to let me know.

Silvester

paudieo · ‎08-28-2014

Hi Silvester,

the vmkernel logs have some data, but maybe out of context, it would it really be great if you could send me a full vm-support

I need to check out some more details regarding the disk naa = naa.600605b005a783701acbdef240ab6fb8

there are a few events that I need to check out

vm-support not only collects log files , but also some configuration data that would be helpful

Appreciate it !!

paudieo · ‎08-28-2014

Also there is some vsan specific log files that would be included on a vm-support

this would narrow the problem considerably

Silvester · ‎08-28-2014

Here it is the vm-support output.

BTW, the disk ID you mention is the SSD I create the disk group with.

Thanks!

Silvester

Silvester · ‎08-30-2014

Hi Paudeio,

If you have reviewed my vm-support log files, would you kindly suggest how I shall proceed?

Thanks!

Silvester

All

VSAN disk group creation failed