VMware Cloud Community
WhiteSea
Contributor
Contributor

ESXi 7.0.3 VMFS-6 Unable to mount datastore ( JBC inconsistency errors )

Hi,

I recently ( and unfortunately ) upgraded my ESXi server to 7.0.3 ... which was then pulled. Having upgraded I have been experiencing random crashes of the ESXi server. This looks to have caused a corruption in one of my VMFS-6 datastores. A voma check lists a number of lock issues but ultimately I think the 4 JBC inconsistency errors shown are the reason for the filesystem not mounting

 

Running VMFS Checker version 2.1 in check mode               

Initializing LVM metadata, Basic Checks will be done         

                                                             

Checking for filesystem activity

Performing filesystem liveness check..|Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).

         Scsi 2 reservation successful                       

Phase 1: Checking VMFS header and resource files             

   Detected VMFS-6 file system (labeled:'VMFS-2TBa') with UUID:60568465-a8e23e74-8eca-a0369fcfbc38, Version 6:82

   Found stale lock [type 10c00002 offset 11698176 v 234, hb offset 4096000

gen 161, mode 1, owner 61aa6e60-dc12f1be-594b-a0369fcfbc38 mtime 8353

num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00002 offset 15794176 v 64, hb offset 4096000

gen 211, mode 1, owner 61cf03d5-bdd708c1-ccb3-a0369fcfbc38 mtime 5487

num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00002 offset 16080896 v 52, hb offset 4096000

gen 161, mode 1, owner 61aa6e60-dc12f1be-594b-a0369fcfbc38 mtime 8391

num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00002 offset 32776192 v 20, hb offset 4096000

gen 183, mode 1, owner 61b5ef88-5f9c65cf-dfd2-a0369fcfbc38 mtime 1076828

num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00007 offset 734175232 v 132, hb offset 4100096

gen 29, mode 1, owner 61cf2ee4-900f1726-6c59-a0369fcfbc38 mtime 91

num 0 gblnum 0 gblgen 0 gblbrk 0]

Phase 2: Checking VMFS heartbeat region                      

   Stale HB state with zero journal address                  

Phase 3: Checking all file descriptors.                      

   Found stale lock [type 10c00001 offset 7577600 v 54, hb offset 4096000

gen 211, mode 1, owner 61cf03d5-bdd708c1-ccb3-a0369fcfbc38 mtime 4914

num 0 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 7593984 v 50, hb offset 4096000

gen 119, mode 2, owner 00000000-00000000-0000-000000000000 mtime 4908

num 1 gblnum 0 gblgen 0 gblbrk 0]

   Found stale lock [type 10c00001 offset 7602176 v 513, hb offset 4096000

gen 211, mode 1, owner 61cf03d5-bdd708c1-ccb3-a0369fcfbc38 mtime 4842

num 0 gblnum 0 gblgen 0 gblbrk 0]

..

..

..

Phase 4: Checking pathname and connectivity.                 

Phase 5: Checking resource reference counts.                 

 ON-DISK ERROR: JBC inconsistency found: (13,0) allocated in bitmap, but never used

 ON-DISK ERROR: JBC inconsistency found: (13,1) allocated in bitmap, but never used

 ON-DISK ERROR: JBC inconsistency found: (13,2) allocated in bitmap, but never used

 ON-DISK ERROR: JBC inconsistency found: (13,3) allocated in bitmap, but never used

                                                             

Total Errors Found:           4

Does anyone know how to fix these errors. The "fix" option of voma does nothing. To say I am pissed at VMWare for this 7.0.3 release is an understatement.

Reply
0 Kudos
2 Replies
Marcela2
VMware Employee
VMware Employee

Hello!

If you are still having issues, checking the Knowledge Base might offer solutions. 
You can also find a list of known issues and existing workarounds. For example this one https://kb.vmware.com/s/article/86281

Have you been able to find a solution? 

Digital Support Coordinator
Reply
0 Kudos
WhiteSea
Contributor
Contributor

I resolved this by booting the ESXi server off a Linux Live CD and used vmfs6-tools to mount the filesystem and copy the required data, luckily retaining 100%.

Turns out issue was due to a faulty SSD drive and not VMWare. Still it is not helpful to be unable to mount and access the VMFS6 data without having to boot an alternative operating system.

 

Reply
0 Kudos