1 Reply Latest reply on Nov 22, 2016 2:23 PM by branmelton

    Corrupt VMs, still running, no backups, cannot access the "ghost"

    branmelton Lurker

      We have some VMs that are currently running but we're terrified to do anything with them, because they are basically running in memory, pointing to inodes the filesystem no longer allows you to access. Due to holes in alerts/backup-procedures we've found that these very critical servers are not being backed up due to failures, due to no access to the VMDKs. One of them got powered off due to host failure, then could no longer power back on.

       

      "lsof" on one of the ESXi hosts will show the file being accessed. Other esxi commands show the vms healthy running, and point to their configuration file locations. If you try to navigate to these locations though......file or directory not found.

       

      [root@txs0705074:/vmfs/volumes] lsof | grep TAM

      14311301    vmx                   FILE                       50   /vmfs/volumes/61be1cbe-fac9f70e/TXS9312026-TAMWebSEAL#1/TXS9312026-TAMWebSEAL#1.vmx.lck

      14311301    vmx                   FILE                       53   /vmfs/volumes/61be1cbe-fac9f70e/TXS9312026-TAMWebSEAL#1/vmware.log

      14311301    vmx                   FILE                       80   /vmfs/volumes/61be1cbe-fac9f70e/TXS9312026-TAMWebSEAL#1/TXS9312026-TAMWebSEAL#1-flat.vmdk

      14311301    vmx                   FILE                       81   /vmfs/volumes/61be1cbe-fac9f70e/TXS9312026-TAMWebSEAL#1/TXS9312026-TAMWebSEAL#1-ctk.vmdk

      14311301    vmx                   FILE                      108   TXS9312026-TAMWebSEAL#1.nvram

      14311301    vmx                   FILE                      121   /vmfs/volumes/61be1cbe-fac9f70e/TXS9312026-TAMWebSEAL#1/TXS9312026-TAMWebSEAL#1.vmx~

      3579985     vmx                   FILE                       53   /vmfs/volumes/07858b50-59529eb5/TXS9312029–TAMWebSEAL#3/vmware.log

      3579985     vmx                   FILE                       74   /vmfs/volumes/01f25e6d-1f0d3efa/TXS9312029–TAMWebSEAL#3/TXS9312029–TAMWebSEAL#3-flat.vmdk

      3579985     vmx                   FILE                       78   /vmfs/volumes/07858b50-59529eb5/TXS9312029–TAMWebSEAL#3/TXS9312029–TAMWebSEAL#3-flat.vmdk

      3579985     vmx                   FILE                       79   /vmfs/volumes/07858b50-59529eb5/TXS9312029–TAMWebSEAL#3/TXS9312029–TAMWebSEAL#3-ctk.vmdk

      3579985     vmx                   FILE                      106   TXS9312029–TAMWebSEAL#3.nvram

      3579985     vmx                   FILE                      120   /vmfs/volumes/01f25e6d-1f0d3efa/TXS9312029–TAMWebSEAL#3/TXS9312029–TAMWebSEAL#3-ctk.vmdk

      54321       vmx                   FILE                       53   /vmfs/volumes/46fd421e-73a97551/TXS9312025–TAMPolicyServer/vmware.log

      54321       vmx                   FILE                       78   /vmfs/volumes/46fd421e-73a97551/TXS9312025–TAMPolicyServer/TXS9312025–TAMPolicyServer-flat.vmdk

      54321       vmx                   FILE                       79   /vmfs/volumes/46fd421e-73a97551/TXS9312025–TAMPolicyServer/TXS9312025–TAMPolicyServer-ctk.vmdk

      54321       vmx                   FILE                      105   TXS9312025–TAMPolicyServer.nvram

      54321       vmx                   FILE                      118   /vmfs/volumes/01f25e6d-1f0d3efa/TXS9312025–TAMPolicyServer/TXS9312025–TAMPolicyServer-flat.vmdk

      54321       vmx                   FILE                      119   /vmfs/volumes/01f25e6d-1f0d3efa/TXS9312025–TAMPolicyServer/TXS9312025–TAMPolicyServer-ctk.vmdk

       

      Here's an example of some of the directories we can/cannot access:

      [root@txs0705074:/vmfs/volumes] cd /vmfs/volumes/01f25e6d-1f0d3efa/TXS9312029–TAMWebSEAL#3

      [root@txs0705074:/vmfs/volumes/01f25e6d-1f0d3efa/TXS9312029–TAMWebSEAL#3] ls

      TXS9312029–TAMWebSEAL#3-ctk.vmdk   TXS9312029–TAMWebSEAL#3-flat.vmdk  TXS9312029–TAMWebSEAL#3.vmdk

       

      [root@txs0705074:/vmfs/volumes/01f25e6d-1f0d3efa/TXS9312029–TAMWebSEAL#3] cd /vmfs/volumes/07858b50-59529eb5/TXS9312029–TAMWebSEAL#3

      -sh: cd: can't cd to /vmfs/volumes/07858b50-59529eb5/TXS9312029–TAMWebSEAL#3

       

      [root@txs0705074:/vmfs/volumes/01f25e6d-1f0d3efa/TXS9312029–TAMWebSEAL#3] cd /vmfs/volumes/01f25e6d-1f0d3efa/TXS9312025–TAMPolicyServer

       

      [root@txs0705074:/vmfs/volumes/01f25e6d-1f0d3efa/TXS9312025–TAMPolicyServer] cd /vmfs/volumes/46fd421e-73a97551/TXS9312025–TAMPolicyServer

      -sh: cd: can't cd to /vmfs/volumes/46fd421e-73a97551/TXS9312025–TAMPolicyServer

       

      The locations we cannot access are where the configuration files are located.....you know the important stuff. I'm hoping there's a way to restore access to these locations or at least safely copy them to a new place.

       

      Configurations:

       

      TXS9312026-TAMWebSEAL#1

         World ID: 14311302

         Process ID: 0

         VMX Cartel ID: 14311301

         UUID: 42 0e 7d 78 d3 51 33 43-f2 9e a6 b4 54 b6 22 89

         Display Name: TXS9312026-TAMWebSEAL#1

         Config File: /vmfs/volumes/61be1cbe-fac9f70e/TXS9312026-TAMWebSEAL#1/TXS9312026-TAMWebSEAL#1.vmx

       

      TXS9312029–TAMWebSEAL#3

         World ID: 3579986

         Process ID: 0

         VMX Cartel ID: 3579985

         UUID: 42 04 f6 ba 0c 34 dd 70-26 e8 b4 00 91 be 34 b4

         Display Name: TXS9312029–TAMWebSEAL#3

         Config File: /vmfs/volumes/07858b50-59529eb5/TXS9312029–TAMWebSEAL#3/TXS9312029–TAMWebSEAL#3.vmx

       

      TXS9312025–TAMPolicyServer

         World ID: 54322

         Process ID: 0

         VMX Cartel ID: 54321

         UUID: 42 04 51 04 97 9a ae 7f-c5 91 42 34 31 34 15 e4

         Display Name: TXS9312025–TAMPolicyServer

         Config File: /vmfs/volumes/46fd421e-73a97551/TXS9312025–TAMPolicyServer/TXS9312025–TAMPolicyServer.vmx

       

      Here's an example of an "ls" in a directory and it coming back knowing that the TAM directory is missing without me even specifying it:

       

      [root@txs0705074:/vmfs/volumes] cd 2552_02_esx_tier2_02

      [root@txs0705074:/vmfs/volumes/07858b50-59529eb5] ls

      ls: ./TXS9312029–TAMWebSEAL#3: No such file or directory

      CTYDC02-AD-DC                                   TXS9312081 - Replicator Producti                TXS9314239-DocumakerProdServer2

      TXS0703091-DynaTraceCASServer                   TXS9312083-FCPDDRINT_2_2                        TXS9316128-IBMCDCAccessControl

      TXS0703185-DynaTraceProdServer                  TXS9312152-FCPDDRINT_2-2                        TXS9316133-CRM2013ProdSRV2

      TXS10387-Infonet 2                              TXS9312160-ERAS Server                          TXS9316151-IISVideo

      TXS6811059-SharePointServerInternetProduction1  TXS9312170-WebFocus3                            TXS9316168-ADFProxyPRDServer

      TXS6811098-DocLinkMobileApp                     TXS9312201vm - Premium Finance                  TXS9316170-ShareNetSrchServer1

      TXS9210034-TAMWEBSeal7                          TXS9312229-4SightApplicationProductionServe1    TXS9316203-OfficeWebApp2

      TXS9210165-KeyLicenseServer2012-1               TXS9312243 - DocuMaker Server 2                 TXS9316205-Kofax10ProdReleaseSvr

      TXS9210191-TXFB-CAS-01                          TXS9314049-NewKofaxProduction2                  TXS9316218-SharePointSP1

      TXS9210192-TXFB-CAS-02                          TXS9314061-DB2ConnectProd10_2                   TXS9316248-TCI_2_Prod

      TXS9210198-PrintServerWin2012r2                 TXS9314160-TivoliWorkloadScheduler850           TXS9316249-DocNET_FE_1

      TXS9210205-ZeacomRepeater1                      TXS9314165-DocFinityProductionWeb1              TXS9830035-DR_AD

      TXS9210224-MicrosoftIPAM                        TXS9314167-DocFinityProductionWeb2              VeeamRestoreFolder

      TXS9312050-ActiveDirectoryDC-B                  TXS9314213-ServicedeskWeb                       txs9316084-DrivveMgr1

      TXS9312076-LANDesk95                            TXS9314213-ServicedeskWeb_1