Not able to expand data store of size 16TB

[chandan1847]

Hi, I am using vijavaNG-6.0 for provisioning vcenter.

Problem :

When i create datastore of exactly 16TB it got created. But when i am trying to expand, its not happening.

so suppose i created 16TB data store and then i try to expand it to 20TB. Data store expand failed. Showing total formatted capacity 16TB only.

This happens for exact 16TB Data store.

Logs ESX vmkernel.log :

2017-08-02T13:31:08.211Z cpu23:34189 opID=e5892109)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:08.441Z cpu23:34189 opID=e5892109)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:08.542Z cpu23:34189 opID=e5892109)WARNING: LVM: 2735: [naa.6000eb32f4b70e830000000000000849:1] Invalid extended magic 0x0 at off 17592203870208

2017-08-02T13:31:08.547Z cpu23:34189 opID=e5892109)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:08.550Z cpu23:34189 opID=e5892109)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:39.421Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000839:1] Device expanded (actual size 36507219935 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:39.425Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000839:1] Device expanded (actual size 36507219935 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:39.643Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000839:1] Device expanded (actual size 36507219935 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:39.646Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000839:1] Device expanded (actual size 36507219935 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:41.886Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:41.890Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:42.099Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:42.103Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:59.239Z cpu1:34707 opID=39891987)World: 14302: VC opID 34FC351F-00003406-73-c8 maps to vmkernel opID 39891987

2017-08-02T13:31:59.239Z cpu1:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.302Z cpu1:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.344Z cpu1:34707 opID=39891987)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:59.362Z cpu1:34707 opID=39891987)LVM: 2908: [naa.6000eb32f4b70e830000000000000839:1] Device expanded (actual size 36507219935 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:59.367Z cpu1:34707 opID=39891987)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:59.370Z cpu1:34707 opID=39891987)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:59.687Z cpu0:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.689Z cpu4:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.690Z cpu4:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.691Z cpu4:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.691Z cpu4:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.692Z cpu4:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

2017-08-02T13:31:59.694Z cpu4:34707 opID=39891987)FSS: 5099: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported

2017-08-02T13:31:59.706Z cpu4:34707 opID=39891987)Vol3: 731: Couldn't read volume header from control: Not supported

2017-08-02T13:31:59.706Z cpu4:34707 opID=39891987)Vol3: 731: Couldn't read volume header from control: Not supported

2017-08-02T13:31:59.706Z cpu4:34707 opID=39891987)FSS: 5099: No FS driver claimed device 'control': Not supported

2017-08-02T13:31:59.872Z cpu4:34707 opID=39891987)LVM: 2908: [naa.6000eb32f4b70e830000000000000839:1] Device expanded (actual size 36507219935 blocks, stored size 34360260575 blocks)

2017-08-02T13:31:59.875Z cpu4:34707 opID=39891987)LVM: 2908: [naa.6000eb32f4b70e830000000000000839:1] Device expanded (actual size 36507219935 blocks, stored size 34360260575 blocks)

2017-08-02T13:32:00.232Z cpu4:34707 opID=39891987)VC: 2052: Device rescan time 172 msec (total number of devices 22)

2017-08-02T13:32:00.232Z cpu4:34707 opID=39891987)VC: 2055: Filesystem probe time 867 msec (devices probed 21 of 22)

2017-08-02T13:32:00.232Z cpu4:34707 opID=39891987)VC: 2057: Refresh open volume time 0 msec

2017-08-02T13:32:00.255Z cpu4:34707 opID=39891987)<3>ata2.00: bad CDB len=16, scsi_op=0x9e, max=12

[dekoshal]

Run below mentioned command on esxi host and share the output.

partedUtil getptbl /vmfs/devices/disks/naa.6000eb32f4b70e830000000000000849

partedUtil getptbl /vmfs/devices/disks/naa.6000eb32f4b70e830000000000000839

If you found this or any other answer helpful, please consider the use of the Correct or Helpful to award points.

Best Regards,

Deepak Koshal

CNE|CLA|CWMA|VCP4|VCP5|CCA

[continuum]

Hi
Looks like the VMFS-partition naa.6000eb32f4b70e830000000000000839:1 has actually been expanded to 17TB
VMFS-partition naa.6000eb32f4b70e830000000000000849:1 has been expanded to 20TB.
Not sure why the VMFS-headers do not reflect that ?
To expand a VMFS-partition you have to expand the LUN but also grow the VMFS. Are you sure you did both ?
Do you have important data on any of those 2 datastores ?
If yes - I highly recommend to create a VMFS-header-dump for both before you try anything else.
dd if="/dev/disks/naa.6000eb32f4b70e830000000000000839:1" bs=1M count=1536 of=/vmfs/volumes/<some-other-datastore>/naa.6000eb32f4b70e830000000000000839_1.1536

dd if="/dev/disks/naa.6000eb32f4b70e830000000000000849:1" bs=1M count=1536 of=/vmfs/volumes/<some-other-datastore>/naa.6000eb32f4b70e830000000000000849_1.1536
Those dumps would help to extract your data if everything else goes wrong.
Next step would be to post the results of partedUtil getptbl as already suggested.
Also explain how exactly you expanded both partitions.
Can you still mount them ?
I highly recommend to do the further troubleshooting from one single ESXi host - bypassing vCenter

[chandan1847]

Thanks Guys for reply but the given solutions not able to solve this problem.

Here are the steps to reproduce :

1. Create a volume of exact size 16TB(17592186044416 Bytes) i.e multiple of 1024.

2. Export it to ESX Host and create a data store out of it.

3. Expand the same volume on Array Side by 1TB(any size).

4. Do Rescan HBAs and refresh. Expand same datastore.

While expanding data store I am getting error :

Error : Call "HostDatastoreSystem.ExpandVmfsDatastore" for object "datastoreSystem-164" on vCenter Server "linuxvc5" failed.

Operation failed, diagnostics report: Unable to grow Filesystem, please see VMkernel log fordetails : Invalid metadata

[continuum]

> Thanks Guys for reply but the given solutions not able to solve this problem.
??? - we have not suggested any solution yet.
All actions we suggested so far were troubleshooting steps.

[dekoshal]

Command i shared was not the solution. I am still waiting for the output.

Best Regards,

Deepak Koshal

CNE|CLA|CWMA|VCP4|VCP5|CCA

[chandan1847]

Here is the output :

~ # partedUtil getptbl /vmfs/devices/disks/naa.6000eb32f4b70e830000000000000862

gpt

2272469 255 63 36507222016

1 2048 36507221982 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

[dekoshal]

This is a different LUN and not what you reported previously. Can you share the vmkernel.log snippet for this LUN one which is similar to this.

2017-08-02T13:31:42.103Z cpu4:34284)LVM: 2908: [naa.6000eb32f4b70e830000000000000849:1] Device expanded (actual size 42949670879 blocks, stored size 34360260575 blocks)

Best Regards,

Deepak Koshal