Drive - Predictive Failure, Snapshots won't delete, cannot clone

[Tolahouse1]

Ok guys I have a serious issue.

I moved 5 VM's on to a single disk to resolve ISCSI storage problems. ISCSI issues have been resolved but I cannot get my VM's off this single disk due to all the failed backup snapshots on the drive.

• I have created  a new snapshot, then tried deleting all - no joy
  
• I have shutdown, tried cloning - no joy
  
• I have tried manually copying files to another storage - no joy
  
• I've seen "Consolidate Helper-0" a few times
  
• Snapshot Manager currently shows no snapshopts but the datastore directory shows tons of them
  
• 4 of the VM's are small (40GB but now pushing 100+ GB due to snapshots) and the 5th VM has 3 disks to it (40,130 & 500)
  

I've attached a shot of one of the datastore directory's.

Any idea of how I can resolve this issue.

Thanks

[a_p_]

Welcome to the Community,

please explain "no joy" for the steps you took so far.

If possible attach the VM's .vmx file as well as the vmware.log file to your next replay post. Maybe this contains some hints.

Is the "Drive - Predictive Failure" in the discussion's topic related to the local disk?

André

[Tolahouse1]

Andre, thanks for the follow up.

Yes, the drive predictive failure is related to a disk on the server in question.

No joy - this revolved around error messages at different points in time

• "Error caused by file [LocalDisk115Two} GAVMISA01/GAVMISA01-000009.vmdk"
  
• "Error caused by file [LocalDisk115Two} GAVMISA01/GAVMISA01-000010.vmdk"
  
• " Unable to access file <unspecified filename>"
  

Thanks

[a_p_]

This really doesn't sound promising. What you could try to do is to clone the virtual disk with its snapshots to another drive, starting with the latest snapshot as the source and - in case of errors - going back to the next snapshot in the chain.

To clone the virtula disk, run the following command from the VM's current directory:

vmkfstools -i GAVMISA01-000010.vmdk /vmfs/volumes/<targetdatastore/<New-Folder>/GAVMISA01.vmdk

According to the log file the snapshot chain is:

000010, 000009, 000006, 000004, 000003, 000002, 000008, 000007, 000005, 000001, GAVMISA01.vmdk

so if the above command does not run successfully, run it with "GAVMISA01-000009.vmdk" as the source and so on.

André

[Tolahouse1]

Thanks for this.

I will be able to test this after hours this weekend when I can shut systems down. But on the other hand is it possible to just delete all these snapshots and go (boot from) to the original install. The information on 3 of the 5 servers is static so I really don't need the snapshots because there's been no actual data change on these VM's. The snapshots are just from failed backups. my fear is that I delete the snapshots and the systems don't boot anymore.

If I can get these 3 moved then I'll come to the other two which are the real serious contenders (Exchange 2007 and Blackberry servers).

Thanks.

[a_p_]

... my fear is that I delete the snapshots and the systems don't boot anymore.

It's actually the other way around. Snapshots in VMware products work like a chain where each chain link is in use. Manually deleting a snapshot (e.g. from the datastore browser) would lead to data loss. Please take a look at http://kb.vmware.com/kb/1015180 for details.

You could certainly attach the base disks to the VM and delete the snapshots manually (technically no problem), but you would end up with the data of 08/2012 in this case.

André

[Tolahouse1]

You could certainly attach the base disks to the VM and delete the snapshots manually (technically no problem), but you would end up with the data of 08/2012 in this case

I don't mind this for the three static servers. How do I go about doing this.

[a_p_]

Shut down the VM, remove the virtual disk from the VM's settings (make note of the SCSI ID and do NOT select delete from disk) and save the settings. Then open the settings again and add an existing virtual disk (GAVMISA.vmdk) to the VM using the same SCSI ID.

Once done, open the datastore browser, create a new sub-directory and move all .vmdk files (except for the one you attached to the VM) as well as the .vmsd file to this folder. Then remove the VM from the inventory by right-clicking it (don't delete from disk) and re-add it by right clicking the VM's .vmx file in the datastore browser to make ESXi aware of the removed .vmsd file (this file contains the information for the snapshots).

If the VM powers on without issues you can delete the backup folder.

André

[Tolahouse1]

Andre, sorry for my latelate reply - been troubleshooting this and many other issues.

With some stroke of luck I was able to migrate the Exchange Server off this host - didn't do anything fancy just tried to migrate it and it worked. Thought I was in luck and tried it on the Blackberry Server, totally forgot abot the SQL server on it and that freaked out big time. So I have had to rebuild that server and repopulate the users, they were not happy at all.

That now leaves me with this last server which is the firewall. Finding time to shut it down to try the steps you mention is proving quite hard and your last sentence gives me the chills "If the VM powers on without issues" IF oh man it's got to.