Re: Disaster with RAW Device!!

stony007_de · ‎08-29-2008

hey VM-Guys,

yesterday night, i get an error with my one vm in my infrastructure.

The vcb backup was made trouble..

The VM was creating a snapshot and than going down (hard power off). The Vm consists of one vmdk and one rdm!

If i tried to start the vm manually i get the following error:

"Cannot open the disk '/vmfs/volumes/..../MyVM/MyVM-01-000045.vmdk' or one of the snapshot disks it depends on.Reason: Too many levels of redo logs."

Try to apply Snapshot´s --> but in VC does not show any snapshot!

On the cli tried it with vcbSnapshot .... but this shows no snapshots, too.

The workaround to create a new snapshot and remove all, does not!

after some "googeling" i found a solution to get the machine on again..

1.) export MyVM
2.)  cd /vmfs/volumes/.../
      mv MyVM MyVM.old
      mkdir MyVM
      cd MyVM
3.) # vcbExport -M 1 -F 1 -d ${VMNAME}_1.vmdk -s ../${VMNAME}.old/${VMNAME}_1.vmdk
4.) connect the RDM to the MyVM
5.) start MyVM --&gt; machine is running with RAW Device

Now my problem:

 i miss files from one day! i got all *delta* file and die copy of the original vm folder.

is ther any possibility to restore the missing files, from the delta files?

malaysiavm · ‎08-29-2008

I had not very clear with your issue. Let me explain how I can solve this. If you use the RDM in your storage, even you VM is corrupted, you will still able to get the content in the storage by assigning the Physical LUN to another VM or reattach should work. you can even do a zoning for that particular LUN to another physical server, the content in the storage will still around. If file system corrupted, that will be operating system related, and you should recover by Operating System recovery method.

Malaysia VMware Communities -

Craig vExpert 2009 & 2010 Netapp NCIE, NCDA 8.0.1 Malaysia VMware Communities - http://www.malaysiavm.com

oreeh · ‎08-29-2008

Post a listing of all the files you have (including file sizes)

stony007_de · ‎08-29-2008

the corrupt MyMV-01.vmdk was exported with vcbExport, than a new machine was created with the MyVM-01.vmdk as hdd.

the raw device on san was not touched, only diconect from currupt an reconnect to the new created machine...

ls -lh -->

total 21G drwxrwxrwx 1 root root 560 Aug 29 14:03 bck -rwxrwxrwx 1 root root 64M Aug 25 07:41 MyVM-000001-delta.vmdk -rwxrwxrwx 1 root root 227 Aug 25 06:06 MyVM-000001.vmdk -rwxrwxrwx 1 root root 16M Aug 25 07:41 MyVM-000002-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 07:41 MyVM-000002.vmdk -rwxrwxrwx 1 root root 32M Aug 25 08:18 MyVM-000003-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 07:41 MyVM-000003.vmdk -rwxrwxrwx 1 root root 16M Aug 25 08:22 MyVM-000004-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 08:18 MyVM-000004.vmdk -rwxrwxrwx 1 root root 16M Aug 25 08:48 MyVM-000005-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 08:22 MyVM-000005.vmdk -rwxrwxrwx 1 root root 16M Aug 25 08:57 MyVM-000006-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 08:48 MyVM-000006.vmdk -rwxrwxrwx 1 root root 16M Aug 25 09:04 MyVM-000007-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 08:57 MyVM-000007.vmdk -rwxrwxrwx 1 root root 16M Aug 25 09:13 MyVM-000008-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 09:05 MyVM-000008.vmdk -rwxrwxrwx 1 root root 16M Aug 25 09:22 MyVM-000009-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 09:13 MyVM-000009.vmdk -rwxrwxrwx 1 root root 16M Aug 25 09:42 MyVM-000010-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 09:22 MyVM-000010.vmdk -rwxrwxrwx 1 root root 32M Aug 25 10:15 MyVM-000011-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 09:42 MyVM-000011.vmdk -rwxrwxrwx 1 root root 32M Aug 25 10:42 MyVM-000012-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 10:15 MyVM-000012.vmdk -rwxrwxrwx 1 root root 16M Aug 25 10:52 MyVM-000013-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 10:42 MyVM-000013.vmdk -rwxrwxrwx 1 root root 16M Aug 25 11:03 MyVM-000014-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 10:52 MyVM-000014.vmdk -rwxrwxrwx 1 root root 16M Aug 25 11:12 MyVM-000015-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 11:03 MyVM-000015.vmdk -rwxrwxrwx 1 root root 16M Aug 25 11:22 MyVM-000016-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 11:12 MyVM-000016.vmdk -rwxrwxrwx 1 root root 32M Aug 25 11:37 MyVM-000017-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 11:22 MyVM-000017.vmdk -rwxrwxrwx 1 root root 16M Aug 25 11:48 MyVM-000018-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 11:37 MyVM-000018.vmdk -rwxrwxrwx 1 root root 48M Aug 25 12:41 MyVM-000019-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 11:48 MyVM-000019.vmdk -rwxrwxrwx 1 root root 16M Aug 25 12:46 MyVM-000020-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 12:41 MyVM-000020.vmdk -rwxrwxrwx 1 root root 16M Aug 25 13:02 MyVM-000021-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 12:47 MyVM-000021.vmdk -rwxrwxrwx 1 root root 80M Aug 25 16:30 MyVM-000022-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 13:03 MyVM-000022.vmdk -rwxrwxrwx 1 root root 16M Aug 25 16:37 MyVM-000023-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 16:30 MyVM-000023.vmdk -rwxrwxrwx 1 root root 128M Aug 25 20:25 MyVM-000024-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 16:37 MyVM-000024.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:37 MyVM-000025-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:25 MyVM-000025.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:41 MyVM-000026-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:37 MyVM-000026.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:46 MyVM-000027-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:41 MyVM-000027.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:46 MyVM-000028-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:46 MyVM-000028.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:52 MyVM-000029-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:46 MyVM-000029.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:53 MyVM-000030-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:52 MyVM-000030.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:53 MyVM-000031-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:53 MyVM-000031.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:56 MyVM-000032-delta.vmdk -rwxrwxrwx 1 root root 260 Aug 25 20:54 MyVM-000032.vmdk -rwxrwxrwx 1 root root 16M Aug 25 20:57 MyVM-000033-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:56 MyVM-000033.vmdk -rwxrwxrwx 1 root root 22K Aug 25 20:57 MyVM-000034-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 20:57 MyVM-000034.vmdk -rwxrwxrwx 1 root root 22K Aug 25 21:04 MyVM-000035-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 21:04 MyVM-000035.vmdk -rwxrwxrwx 1 root root 22K Aug 25 21:17 MyVM-000036-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 21:17 MyVM-000036.vmdk -rwxrwxrwx 1 root root 22K Aug 25 21:24 MyVM-000037-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 21:24 MyVM-000037.vmdk -rwxrwxrwx 1 root root 22K Aug 25 21:31 MyVM-000038-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 21:31 MyVM-000038.vmdk -rwxrwxrwx 1 root root 22K Aug 25 21:39 MyVM-000039-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 21:39 MyVM-000039.vmdk -rwxrwxrwx 1 root root 22K Aug 25 21:37 MyVM-000040-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 21:37 MyVM-000040.vmdk -rwxrwxrwx 1 root root 22K Aug 25 21:47 MyVM-000041-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 21:47 MyVM-000041.vmdk -rwxrwxrwx 1 root root 22K Aug 25 22:19 MyVM-000042-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 22:19 MyVM-000042.vmdk -rwxrwxrwx 1 root root 22K Aug 25 22:42 MyVM-000043-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 22:42 MyVM-000043.vmdk -rwxrwxrwx 1 root root 22K Aug 25 23:37 MyVM-000044-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 23:37 MyVM-000044.vmdk -rwxrwxrwx 1 root root 22K Aug 25 22:50 MyVM-000045-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 22:50 MyVM-000045.vmdk -rwxrwxrwx 1 root root 22K Aug 25 23:37 MyVM-000046-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 23:37 MyVM-000046.vmdk -rwxrwxrwx 1 root root 22K Aug 25 23:41 MyVM-000047-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 23:41 MyVM-000047.vmdk -rwxrwxrwx 1 root root 22K Aug 25 23:41 MyVM-000048-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 25 23:41 MyVM-000048.vmdk -rwxrwxrwx 1 root root 22K Aug 26 00:07 MyVM-000049-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 26 00:07 MyVM-000049.vmdk -rwxrwxrwx 1 root root 22K Aug 26 00:07 MyVM-000050-delta.vmdk -rwxrwxrwx 1 root root 234 Aug 26 00:07 MyVM-000050.vmdk -rwxrwxrwx 1 root root 801M Aug 25 07:41 MyVM_1-000001-delta.vmdk -rwxrwxrwx 1 root root 232 Aug 25 06:06 MyVM_1-000001.vmdk -rwxrwxrwx 1 root root 17M Aug 25 07:41 MyVM_1-000002-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 07:41 MyVM_1-000002.vmdk -rwxrwxrwx 1 root root 289M Aug 25 08:18 MyVM_1-000003-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 07:41 MyVM_1-000003.vmdk -rwxrwxrwx 1 root root 17M Aug 25 08:22 MyVM_1-000004-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 08:18 MyVM_1-000004.vmdk -rwxrwxrwx 1 root root 129M Aug 25 08:48 MyVM_1-000005-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 08:22 MyVM_1-000005.vmdk -rwxrwxrwx 1 root root 97M Aug 25 08:57 MyVM_1-000006-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 08:48 MyVM_1-000006.vmdk -rwxrwxrwx 1 root root 81M Aug 25 09:04 MyVM_1-000007-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 08:57 MyVM_1-000007.vmdk -rwxrwxrwx 1 root root 17M Aug 25 09:13 MyVM_1-000008-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 09:05 MyVM_1-000008.vmdk -rwxrwxrwx 1 root root 33M Aug 25 09:22 MyVM_1-000009-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 09:13 MyVM_1-000009.vmdk -rwxrwxrwx 1 root root 161M Aug 25 09:42 MyVM_1-000010-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 09:22 MyVM_1-000010.vmdk -rwxrwxrwx 1 root root 97M Aug 25 10:15 MyVM_1-000011-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 09:42 MyVM_1-000011.vmdk -rwxrwxrwx 1 root root 385M Aug 25 10:42 MyVM_1-000012-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 10:15 MyVM_1-000012.vmdk -rwxrwxrwx 1 root root 17M Aug 25 10:52 MyVM_1-000013-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 10:42 MyVM_1-000013.vmdk -rwxrwxrwx 1 root root 145M Aug 25 11:02 MyVM_1-000014-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 10:52 MyVM_1-000014.vmdk -rwxrwxrwx 1 root root 81M Aug 25 11:12 MyVM_1-000015-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 11:03 MyVM_1-000015.vmdk -rwxrwxrwx 1 root root 65M Aug 25 11:22 MyVM_1-000016-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 11:12 MyVM_1-000016.vmdk -rwxrwxrwx 1 root root 289M Aug 25 11:37 MyVM_1-000017-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 11:22 MyVM_1-000017.vmdk -rwxrwxrwx 1 root root 129M Aug 25 11:48 MyVM_1-000018-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 11:37 MyVM_1-000018.vmdk -rwxrwxrwx 1 root root 369M Aug 25 12:41 MyVM_1-000019-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 11:48 MyVM_1-000019.vmdk -rwxrwxrwx 1 root root 129M Aug 25 12:46 MyVM_1-000020-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 12:41 MyVM_1-000020.vmdk -rwxrwxrwx 1 root root 97M Aug 25 13:02 MyVM_1-000021-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 12:47 MyVM_1-000021.vmdk -rwxrwxrwx 1 root root 1.5G Aug 25 16:30 MyVM_1-000022-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 13:03 MyVM_1-000022.vmdk -rwxrwxrwx 1 root root 17M Aug 25 16:37 MyVM_1-000023-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 16:30 MyVM_1-000023.vmdk -rwxrwxrwx 1 root root 3.2G Aug 25 20:25 MyVM_1-000024-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 16:37 MyVM_1-000024.vmdk -rwxrwxrwx 1 root root 609M Aug 25 20:37 MyVM_1-000025-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:25 MyVM_1-000025.vmdk -rwxrwxrwx 1 root root 401M Aug 25 20:41 MyVM_1-000026-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:37 MyVM_1-000026.vmdk -rwxrwxrwx 1 root root 353M Aug 25 20:46 MyVM_1-000027-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:41 MyVM_1-000027.vmdk -rwxrwxrwx 1 root root 33M Aug 25 20:46 MyVM_1-000028-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:46 MyVM_1-000028.vmdk -rwxrwxrwx 1 root root 401M Aug 25 20:52 MyVM_1-000029-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:46 MyVM_1-000029.vmdk -rwxrwxrwx 1 root root 17M Aug 25 20:53 MyVM_1-000030-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:52 MyVM_1-000030.vmdk -rwxrwxrwx 1 root root 33M Aug 25 20:53 MyVM_1-000031-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:53 MyVM_1-000031.vmdk -rwxrwxrwx 1 root root 97M Aug 25 20:56 MyVM_1-000032-delta.vmdk -rwxrwxrwx 1 root root 265 Aug 25 20:54 MyVM_1-000032.vmdk -rwxrwxrwx 1 root root 49M Aug 25 20:57 MyVM_1-000033-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:56 MyVM_1-000033.vmdk -rwxrwxrwx 1 root root 802K Aug 25 20:57 MyVM_1-000034-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 20:57 MyVM_1-000034.vmdk -rwxrwxrwx 1 root root 802K Aug 25 21:04 MyVM_1-000035-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 21:04 MyVM_1-000035.vmdk -rwxrwxrwx 1 root root 802K Aug 25 21:17 MyVM_1-000036-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 21:17 MyVM_1-000036.vmdk -rwxrwxrwx 1 root root 802K Aug 25 21:24 MyVM_1-000037-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 21:24 MyVM_1-000037.vmdk -rwxrwxrwx 1 root root 802K Aug 25 21:31 MyVM_1-000038-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 21:31 MyVM_1-000038.vmdk -rwxrwxrwx 1 root root 802K Aug 25 21:39 MyVM_1-000039-delta.vmdk -rwxrwxrwx 1 root root 239 Aug 25 21:39 MyVM_1-000039.vmdk -rwxrwxrwx 1 root root 239 Aug 25 21:37 MyVM_1-000040.vmdk -rwxrwxrwx 1 root root 400G Aug 11 11:09 MyVM_1-rdm.vmdk -rwxrwxrwx 1 root root 420 Aug 29 13:39 MyVM_1.vmdk -rwxrwxrwx 1 root root 420 Aug 29 12:36 MyVM_1.vmdk.bck -rwxrwxrwx 1 root root 802K Aug 25 22:19 MyVM_2-000001-delta.vmdk -rwxrwxrwx 1 root root 232 Aug 25 22:19 MyVM_2-000001.vmdk -rw------- 1 root root 400G Aug 25 22:03 MyVM_2-rdm.vmdk -rwxrwxrwx 1 root root 417 Aug 28 16:49 MyVM_2.vmdk -rwxrwxrwx 1 root root 417 Aug 28 17:38 MyVM_2.vmdk.bck -rwxrwxrwx 1 root root 37 Aug 10 19:06 MyVM-74872033.hlog -rwxrwxrwx 1 root root 10G Aug 25 06:06 MyVM-flat.vmdk -rwxrwxrwx 1 root root 402 Aug 24 06:04 MyVM.vmdk -rwxrwxrwx 1 root root 419 Aug 26 00:10 MyVM.vmsd -rwxrwxrwx 1 root root 2.8K Aug 29 14:09 MyVM.vmx -rwxrwxrwx 1 root root 264 Aug 29 14:08 MyVM.vmxf -rwxrwxrwx 1 root root 2.9K Aug 29 13:56 MyVM.vmx_mod -rwxrwxrwx 1 root root 2.8K Aug 25 21:51 MyVM.vmx.org -rwxrwxrwx 1 root root 68K Aug 25 21:38 vmware-35.log -rwxrwxrwx 1 root root 57K Aug 25 21:48 vmware-36.log -rwxrwxrwx 1 root root 57K Aug 25 22:08 vmware-37.log -rwxrwxrwx 1 root root 57K Aug 25 22:37 vmware-38.log -rwxrwxrwx 1 root root 58K Aug 25 22:47 vmware-39.log -rwxrwxrwx 1 root root 59K Aug 25 22:51 vmware-40.log -rwxrwxrwx 1 root root 59K Aug 25 23:35 vmware.log

oreeh · ‎08-29-2008

the raw device on san was not touched

That was a wise decision!

Zip up all the descriptor files (the short VMDK files), the log files, the VMSD file and the VMX and post it.

stony007_de · ‎08-29-2008

got a private message!

Mega_Watt · ‎08-29-2008

stony,

What is the backup software you're using for what seems like vm image level backups? I just had the exact same issue with the snapshot creation and the issue was 1) The snap resulted in depleting my LUN space during the process and 2) I had to repoint the .vmx file to reflect the correct path to the .vmdk files. I can give you more info and specifics if you're still having the issue.

All

Disaster with RAW Device!!