ESXi 5.5 Dell VOMA gives errors with continuous stream for VMFS partition which looks fine

[nios33]

We have ESXi version: Esxi5.5.0, 8934887 on Dell PE R630 server

With the commands below we have the error report.

Looks like VOMA does not able to check the partition.

Anybody could you please help to resolve the problem?

Full output log is attached here.

# voma -m vmfs -f check -d /vmfs/devices/disks/naa.61866da0a6d9c80022da06c20f21310d:3

Checking if device is actively used by other hosts

Initializing VMFS Checker..|Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).

Running VMFS Checker version 1.0 in check mode

Initializing LVM metadata, Basic Checks will be done

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'esxi43bg_datastore_os') with UUID:5b475391-63c355f6-75bb-246e962bf144, Version 5:60

ON-DISK ERROR: Unexpected 'resourceSize' 8192, expected 65536

ON-DISK ERROR: Unexpected 'clusterGroupSize' 33619968, expected 268500992

ON-DISK ERROR: <FD c0 r4> : Invalid zla type 0x1

ON-DISK ERROR: Block count mismatch(0): PBA cow 0 cnum 0 rnum 0

ON-DISK ERROR: Block count mismatch(1): PBA cow 0 cnum 0 rnum 0

ON-DISK ERROR: Block count mismatch(2): PBA cow 0 cnum 0 rnum 0

...

ON-DISK ERROR: Block count mismatch(249): PBA cow 0 cnum 0 rnum 0

ON-DISK ERROR: Block count mismatch(250): PBA cow 0 cnum 0 rnum 0

         ERROR: Resource file values has been guessed wrong.

         May be because of wrong minor version in disk

         Number of blocks for system file 502 (expected 2001) mismatch.

ON-DISK ERROR: Corruption too severe in resource file [SB]

         ERROR: Failed to check sbc.sf.

ON-DISK ERROR: VMFS Checker has found severe errors in volume esxi43bg_datastore_os. We strongly recommend you backup your data externally and then recreate a new volume.

   VOMA failed to check device : Severe corruption detected

Total Errors Found:           255

[SureshKumarMuth]

Is the VMFS volume contains multiple extents (luns) ??

[nios33]

Hi Sureshkumar

We have solid stream erros. It is unreal besause the VMs on the VMFS looks fine. OS Internal tools on the WMs does not give any errros.

We susrecp that wrong VOMAs wrong there. Probalby metadata not foud ny VOMA or something else.

[SureshKumarMuth]

Could be an issue with VOMA tool itself. We always use to approach vmware to check the consistency of VMFS . You may do that now also. You have to provide them the disk dump.

Unless, you see some errors in the vmkernel logs with respect to the file system, you need not worry, your data is safe.

[nios33]

We use Dell customised ESXi 5.5 on Dell servers and native ESXi 5.5 on non Dell severs (such as Supermicro or Cisco, etc)

On Dell ESXi servers we see this picture. But on non Dell servers VOMA works fine.

The ~ # voma -v outputs are the same

~ # voma -v

voma version 0.2

[myounus2008]

Have we preformed the VOMA test, when it was in disconnected state?

https://ikb.vmware.com/s/article/2037429

Note: 

• It is important that no I/O is performed on the datastore after generating the disk dump. 
• If virtual machines must be moved, they must be moved prior to performing the dd command. 
• Ensure there are no virtual machines powered on when running the dd commands.

[nios33]

Unfortunately i do not have no acess to the 2037429 solution

​could you please repost me the solution

[Finikiez]

Did you unmount datastore from _all_ hosts include the one where you run VOMA?

[nios33]

myounus2008

Are you suggest to test the parttion for writable feature?

Where can we watch the logging ?

[nios33]

Finikiez,

are you saying about exclusive access for VOMA to the partition?

[Finikiez]

Be careful posting the links from internal KB

[Vijay2027]

I would recommend you to log a ticket with VMware support as they would collect 1.5GB dump against the datastore to check the consistency.

On a side note do you see any metadata or heartbeat corruption errors in vobd.log or vmkerenel.log

[Finikiez]

I mean that when you run VOMA datastore must be unmounted from _all_ ESXi hosts.

If it's not unmounted you can get various errors

VMware Knowledge Base

Before running VOMA you must ensure that:

1. All virtual machines on the affected datastore are powered off or migrated to another datastore. See Impact/Risks section.
2. For VMFS-5 datastores, the datastore is unmounted on all ESXi5.x hosts.

[SupreetK]

Looks to be an issue with the VOMA tool. On one of the affected hosts, can you upgrade it to ESXi 6.0 and run VOMA against the same datastore? If we don't see errors on version 6.0, rest assured it is an issue with the VOMA tool.

Please consider marking this answer as "correct" or "helpful" if you think your questions have been answered.

Cheers,

Supreet

[SureshKumarMuth]

From your comment, I understand that the issue occurs only on DELL servers where you used DELL OEM image, if you can take any DELL host for testing and install native ESXi image from VMware site and check if that works ? This will help us to isolate if something to do with the image or the specific to DELL hardware.

[SupreetK]

nios33​ As mentioned in my previous reply, did you get to run VOMA from a ESXi 6.0 host? If yes, do you see the same 255 errors?

Please consider marking this answer as "correct" or "helpful" if you think your questions have been answered.

Cheers,

Supreet

[nios33]

Colleagues

I have found it known issue for 5.5 ESXi. We are moving to 6.0

Thank you for paying attention for the issue !

[continuum]

Moderators note> I removed the "answered" flag as it was inapropriate.
Ulli