VMWare Expanded Extent over 2 disks - Invalid argument on some files

[MohammadRavaghi]

Hi everyone

I have an ESX 6.7 which consists of a datastore named Vira nad this datastore was built over a 1TB Samsung SSD. after a while we needed to expand it and did it using another 1TB and expanded its storage space to about 2TB. last week we had a bad hardware restart and after restart Datastore named Vira had some issues. the first problem was its naming and I figured out that Datastore is not available correctly. I tried to power on a virtual machine that corresponds to this datastore and it didn't come up. after some investigation i found one of the disks included in 2TB Extent is not available. tried to change the missing disk SATA port and it was available in system again. after starting the ESX I could see the Datastore but after that change and start I cannot power on the machin and FLAT VMDK file is locked. every time I try to clone it, touch it , move it I get Invalid Argument error.

searche a lot in VMWare KBs and Forums and found it may be some process and checked it using VMKFSTOOLS -D. it has a lock that relates to ESX itself. restarting Services or even Host doesn't remove the lock.

vmkfstools -D W-Sharing\ SRV_2-flat.vmdk

Lock [type 10c00001 offset 134742016 v 8, hb offset 3702784

gen 15, mode 1, owner 57a3177f-dbdbe28c-4845-002590cb29c2 mtime 82378

num 0 gblnum 0 gblgen 0 gblbrk 0]

Addr <4, 52, 0>, gen 1, links 1, type reg, flags 0x9, uid 0, gid 0, mode 600

len 1869169766400, nb 1299233 tbz 0, cow 0, newSinceEpoch 1299233, zla 3, bs 1048576

affinityFD <4,52,0>, parentFD <4,35,0>, tbzGranularityShift 20, numLFB 0

lastSFBClusterNum 2364, numPreAllocBlocks 0, numPointerBlocks 218

tried to touch the file :

touch: W-Sharing SRV_2-flat.vmdk: Invalid argument

tried unmounting and checking the Datastore using VOMA and found these results:

here is the extent list when Datastore Vira is mounted:

esxcli storage vmfs extent list

Volume Name  VMFS UUID                            Extent Number  Device Name                                                               Partition

-----------  -----------------------------------  -------------  ------------------------------------------------------------------------  ---------

All OS       57892a55-88ac4e8a-d55d-002590cb29c2              0  t10.ATA_____Samsung_SSD_860_EVO_1TB_________________S4BDNG0M104285X_____          3

Kiantech     57a5fb65-e2d18e7a-36f6-002590cb29c2              0  t10.ATA_____Samsung_SSD_840_EVO_1TB_________________S1D9NEADA02146Y_____          1

Wiera1       5849d59d-f80d0f04-5805-002590cb29c2              0  t10.ATA_____Samsung_SSD_860_QVO_2TB_________________S4CYNF0M207684P_____          1

Vira         57a5fac4-a0e012b6-91f4-002590cb29c2              0  t10.ATA_____Samsung_SSD_840_EVO_1TB_________________S1D9NEADA02133Y_____          1

Vira         57a5fac4-a0e012b6-91f4-002590cb29c2              0  t10.ATA_____Samsung_SSD_840_EVO_1TB_________________S1D9NEADB01872B_____          1

and here is the result when Vira is not mounted:

esxcli storage vmfs extent list

Volume Name  VMFS UUID                            Extent Number  Device Name                                                               Partition

-----------  -----------------------------------  -------------  ------------------------------------------------------------------------  ---------

All OS       57892a55-88ac4e8a-d55d-002590cb29c2              0  t10.ATA_____Samsung_SSD_860_EVO_1TB_________________S4BDNG0M104285X_____          3

Kiantech     57a5fb65-e2d18e7a-36f6-002590cb29c2              0  t10.ATA_____Samsung_SSD_840_EVO_1TB_________________S1D9NEADA02146Y_____          1

Wiera1       5849d59d-f80d0f04-5805-002590cb29c2              0  t10.ATA_____Samsung_SSD_860_QVO_2TB_________________S4CYNF0M207684P_____          1

Vira         57a5fac4-a0e012b6-91f4-002590cb29c2              0  t10.ATA_____Samsung_SSD_840_EVO_1TB_________________S1D9NEADA02133Y_____          1

and here is the VOMA result itself:

voma -m vmfs -f check -d /vmfs/devices/disks/t10.ATA_____Samsung_SSD_840_EVO_1TB_________________S1D9NEADA02133Y_____:1

Running VMFS Checker version 2.1 in check mode

Initializing LVM metadata, Basic Checks will be done

Checking for filesystem activity

Performing filesystem liveness check..|Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).

Phase 1: Checking VMFS header and resource files

   Detected VMFS-6 file system (labeled:'Vira') with UUID:57a5fac4-a0e012b6-91f4-002590cb29c2, Version 6:82

         ERROR: Trying to do IO beyond device Size

         ERROR: Failed to check fbb.sf.

   VOMA failed to check device : Limit exceeded

Total Errors Found:           0

   Kindly Consult VMware Support for further assistance

can you please help me investigate the problem over this deployment ?

thanks everyone

Mohammad

[continuum]

Hi Mohammad

After the changes to the extent it is probably next to impossible to fix the issue.

But it may be still possible to extract your missing vmdk.

If the data is valuable read Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

I will need the dump for the parent datastore plus another one for the extent.

> ERROR: Failed to check fbb.sf.

That does not sound good ...

Ulli