6 Replies Latest reply: Oct 20, 2008 5:57 AM by InfinityAdmin RSS

    vmdk consistency problems

    casolorz Hot Shot

       

      We have hundreds of servers running vmware server. Every so often (several times a month) one of our servers will fail to start a vm with consistency errors on a vmdk file. We are unsure of why we are getting these errors, the hard drives on the systems are fine, and typically the vms were shutdown nicely prior to the error. Is there something we may be doing wrong? is the original vmdk that we cloned all of these vms out of somehow corrupt but still capable of running for a while until the error occurrs?

       

       

      I am copying below this part of the log of the error we usually get, thank you for any suggestions you may have.

       

       

       

       

       

      Jul 03 10:39:16: vmx| AIOGNRC: Starting 19 I/O threads.

      Jul 03 10:39:16: vmx| Msg_Hint: msg.disk.validate.dir (not shown)

      Jul 03 10:39:16: vmx| DISK: OPEN scsi0:0 '/home/user/vms/SERVER1/root.vmdk' persistent R[(null)]

      Jul 03 10:39:17: vmx| DISKLIB-DSCPTR: Opened : "root.vmdk" (0xa)

      Jul 03 10:39:17: vmx| DISKLIB-LINK  : Opened '/home/user/vms/SERVER1/root.vmdk' (0xa): monolithicSparse, 104857600 sectors / 51200 Mb.

      Jul 03 10:39:17: vmx| DISKLIB-LIB   : Opened "/home/user/vms/SERVER1/root.vmdk" (flags 0xa).

      Jul 03 10:39:17: vmx| DISK: OPEN '/home/user/vms/SERVER1/root.vmdk' Geo (6527/255/63) BIOS Geo (6527/255/63) freeSpace=160907Mb

      Jul 03 10:39:17: vmx| Msg_Hint: msg.disk.validate.dir (not shown)

      Jul 03 10:39:17: vmx| DISK: OPEN scsi0:1 '/home/user/vms/SERVER1/shared.vmdk' persistent R[(null)]

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk Grain #75635 @9694208 is orphaned.

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk Grain #75636 @9694336 is orphaned.

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk Grain #75637 @9694464 is orphaned.

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk 3 orphan(s) detected

      Jul 03 10:39:19: vmx| DISKLIB-SPARSE: "/home/user/vms/SERVER1/shared.vmdk" : failed to open (14): Disk needs repair.

      Jul 03 10:39:19: vmx| DISKLIB-DSCPTR: Failed to open extents for embedded descriptor file in normal mode

      Jul 03 10:39:19: vmx| DISKLIB-LINK  : "/home/user/vms/SERVER1/shared.vmdk" : failed to open (The specified virtual disk needs repair).

      Jul 03 10:39:19: vmx| DISKLIB-CHAIN : "/home/user/vms/SERVER1/shared.vmdk" : failed to open (The specified virtual disk needs repair).

      Jul 03 10:39:19: vmx| DISKLIB-LIB   : Failed to open '/home/user/vms/SERVER1/shared.vmdk' with flags 0x2a (The specified virtual disk needs repair).

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk Grain #75635 @9694208 is orphaned.

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk Grain #75636 @9694336 is orphaned.

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk Grain #75637 @9694464 is orphaned.

      Jul 03 10:39:19: vmx| DISKLIB-SPARSECHK: /home/user/vms/SERVER1/shared.vmdk 3 orphan(s) detected

      Jul 03 10:39:19: vmx| Msg_Question:

      Jul 03 10:39:19: vmx| http://msg.disk.scaryWithNoLostSectors The disk "/home/user/vms/SERVER1/shared.vmdk" has internal consistency errors that may be caused by partial corruption of the disk file.  It is recommended that you restore a backup of this virtual machine. If you do not have a backup, VMware Server can repair the disk, but there is a possibility that the data on the disk may be corrupt and affect the stability of the guest.

      Jul 03 10:39:19: vmx| Do you wish to repair the disk and continue?

       

       

        • 1. Re: vmdk consistency problems
          Peter_vm Guru

          Did you try to run prolonged memtest86 on those physical servers?

          Do they have local disks or SAN, NAS, iSCSI attached drives?

          • 2. Re: vmdk consistency problems
            casolorz Hot Shot

             

            We have done memtest on some of the machines, not all of them. I am certain at least some of the ones giving the errors have at some point been memtested.

             

             

            The disks are local, SATA, some are software raid on linux, other aren't.

             

             

            Thanks,

             

             

            --Carlos

             

             

            • 3. Re: vmdk consistency problems
              Peter_vm Guru

              Software RAID on Linux hosts can cause problems if hosts are suddenly powered off.

              • 4. Re: vmdk consistency problems
                casolorz Hot Shot

                 

                However these systems were rebooted politely with the vms shutdown all the way first. We have also not gotten any raid events on those systems when the corruption happens.

                 

                 

                 

                 

                 

                --Carlos

                 

                 

                • 5. Re: vmdk consistency problems
                  casolorz Hot Shot

                  We got it again on another system.

                   

                   

                   

                   

                   

                  Opened paging file anon

                  Jul 08 08:30:01: vmx| Mapped mainmem as pageable

                  Jul 08 08:30:01: vmx| MStat: Creating Stat vm.cpuusage

                  Jul 08 08:30:01: vmx| MStat: Creating Stat vm.ram

                  Jul 08 08:30:01: vmx| MStat: Creating Stat vm.uptime

                  Jul 08 08:30:01: vmx| DISK: OPEN scsi0:0 '/home/users/vms/SERVER1/root.vmdk' persistent R[(null)]

                  Jul 08 08:30:01: vmx| AIOGNRC: Starting 18 I/O threads.

                  Jul 08 08:30:01: vmx| DISKLIB-DSCPTR: Opened : "root.vmdk" (0xa)

                  Jul 08 08:30:01: vmx| DISKLIB-LINK  : Opened '/home/users/vms/SERVER1/root.vmdk' (0xa): monolithicSparse, 104857600 sectors / 51200 Mb.

                  Jul 08 08:30:01: vmx| DISKLIB-LIB   : Opened "/home/users/vms/SERVER1/root.vmdk" (flags 0xa).

                  Jul 08 08:30:01: vmx| DISK: OPEN '/home/users/vms/SERVER1/root.vmdk' Geo (6527/255/63) BIOS Geo (6527/255/63) freeSpace=151716Mb

                  Jul 08 08:30:01: vmx| DISK: OPEN scsi0:1 '/home/users/vms/SERVER1/shared.vmdk' persistent R[(null)]

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [503] 0/23682048

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [503] = 23682048

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [504] 0/23682176

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [504] = 23682176

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [505] 0/23681792

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [505] = 23681792

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [506] 0/23681920

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [506] = 23681920

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [507] 0/23681536

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [507] = 23681536

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [508] 0/23681664

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [508] = 23681664

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSE: "/home/users/vms/SERVER1/shared.vmdk" : failed to open (14): Disk needs repair.

                  Jul 08 08:30:01: vmx| DISKLIB-DSCPTR: Failed to open extents for embedded descriptor file in normal mode

                  Jul 08 08:30:01: vmx| DISKLIB-LINK  : "/home/users/vms/SERVER1/shared.vmdk" : failed to open (The specified virtual disk needs repair).

                  Jul 08 08:30:01: vmx| DISKLIB-CHAIN : "/home/users/vms/SERVER1/shared.vmdk" : failed to open (The specified virtual disk needs repair).

                  Jul 08 08:30:01: vmx| DISKLIB-LIB   : Failed to open '/home/users/vms/SERVER1/shared.vmdk' with flags 0xa (The specified virtual disk needs repair).

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [503] 0/23682048

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [503] = 23682048

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [504] 0/23682176

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [504] = 23682176

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [505] 0/23681792

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [505] = 23681792

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [506] 0/23681920

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [506] = 23681920

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [507] 0/23681536

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [507] = 23681536

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk GT Error (ZG): [508] 0/23681664

                  Jul 08 08:30:01: vmx| DISKLIB-SPARSECHK: /home/users/vms/SERVER1/shared.vmdk Resolving [508] = 23681664

                  Jul 08 08:30:01: vmx| Msg_Question:

                  Jul 08 08:30:01: vmx| http://msg.disk.scaryWithNoLostSectors The disk "/home/users/vms/SERVER1/shared.vmdk" has internal consistency errors that may be caused by partial corruption of the disk file.  It is recommended that you restore a backup of this virtual machine. If you do not have a backup, VMware Server can repair the disk, but there is a possibility that the data on the disk may be corrupt and affect the stability of the guest.

                  Jul 08 08:30:01: vmx| Do you wish to repair the disk and continue?----


                   

                  • 6. Re: vmdk consistency problems
                    InfinityAdmin Novice

                    I am having this same issue. I have converted a Sv2i file from symantec recovery product and I have gotten this same error on two different machines. If I try to directly convert one of the machines it fails.... 

                     

                    Each of these Server are a Proliant DL380 G3 server with a secondary partition that is Hardware raided.... The second partition is the one that always comes up with the internal consistency error.....  They are both the same size as well at 203 Gigs... So I cant help but start to think that it is attributed to them being a RAID 5 partition... Can anyone shed some light on this or experienced this? I am not using Software Raid.... I might ass that these are windows 2003 servers I am trying to convert over... and I am trying to run them in VMware server 1.05Build 80187