vsphere Esxi 4.1 - hard disk reconnected lost data

[mdkberry]

Hi

I have ESXI 4.1 running on a HP Proliant managed by Vsphere

the VM is a single Windows Server 2008 R2 Std

it is running on a datastore which is a 250GB hard disk on the proliant

I have a 1TB disk also which I use as the data drive and this is added as a second disk.

this allows me to disconnect the 1TB disk. run and export of the shutdown VM and effectively backup just the VM

the data on the 1TB disk I backup from within Windows on a daily schedule.

I run this procedure of shutting down the VM

disconnecting Hard Disk 2 which is the 1TB data store

running the export

and then reconnecting the Hard Disk 2 and restarting the VM

this has worked great until last night.

when I restored the data store it has restored one that is date about 2nd February 2013.

I cannot figure out what has happened and really need some help to avoid running into this again

luckily I can recover all the data from the nightly backups as I ran a backup before doing the above procedure.

I am mystifed.

here is what I noticed when disconnecting the Hard DIsk 2 last night.

I had last done this procedure on the 2nd of Feb and backup up the VM

after shutting down the VM and exporting it then I went to add the Hard Disk back in I could only see one dated 2nd Feb

the thing is I assumed this meant the date it was last created so didnt think too much about it. The datastore is most all of the 1TB disk so it isnt like it can store two of them on the disk. there was just two files vmdk files and that is what I expected to see.

I booted it all up and checked the Windows server to be certain the data failes were available and it looked fine

I didnt check dates but once staff came in this morning it started be clear that nothing past 2nd feb existed and effectively I had added Hard Disk 2 back in from that date.

I have two questions I cannot answer

1. where the hell is the data store frrom last night before disconnection? I just disconnected it and did not click the option to delete the data.

2. how can a 1TB datastore from 2nd Feb suddnely show up in its place.

if anyone can help me solve this I would really appreciate it.

[a_p_]

This definitely looks like the VM has a snapshot (created on Feb, 2nd) and when you re-attached the virtual disk you selected the base disk as it is not possible to connect a snapshot using the GUI.

Please take a look at the VM's datasatore folder using the Datastore Browser and post a screenshot of the files you see (including, sizes, time stamps, ...) and attach the VM's latest vmware.log file to a reply post.

André

[mdkberry]

thanks for your response André, I really appreciate your help

attached are screen shots of the datastore (250GB drive holding the VM with Windows Server 2008 on it) and DISK_1 which is what I thought was holding the second 1TB disk drive and is data only.

I traced the time to about 18:30 on 21st when I disconnected the hard disk DISK_1 and then ran export on the VM datastore disk, and then reconnected it at about 19:50 which is the time stamp on the vmdk disk in DISK_1 screen shot. It looks like it is sticking to that time as the last modified time too which is what I noticed before seeing the 2nd of Feb on it. I am guessing the same thing will happen again if I disconnect it.

am I right in thinking it is for some reason now storing the data on the 250GB datastore disk and not really actually on the 1TB disk, what's that about?

for the record I recall running out of space around Feb 2nd due to snapshots and I learned that they grow, I hadnt been aware of that prior to finding the server crashing and had to delete all the snapshots and restart the machine. I am not sure if I made any setting changes at that time it would have been based on support I searched on the problem.

I need to rename this DISK_1 vmdk as well, as it is confusing since I named both disk instances 'windows server 2008'  but right now I am a bit lost with what I am looking at, so any advice on proceeding would really help.

I am about to start the process of replacing all the data on the DISK_1 with the backup data I made from within Windows beforethis happened as I need to have their data back by monday AM. luckily it is only about 80 GB, but I need to get this done so will start that process today.

Is there a better way to export a copy of the current VM on its own without the DISK_1 data, and without having to disconnect the DISK_1 for future reference?

I have also included the vmware logs from around the time it occured

thanks again André

rgds

Mark

[a_p_]

That's exactly what I expected to see. The VM has an active snapshot dated Feb. 2nd and instead of adding the snapshot (...-000001.vmdk) you added the base disk, which resulted in loosing the deltas in the snapshot file. Although it would now be possible to re-chain the second virtual disk's snapshot file to the base disk, this would most likely result in data corruption and since you have a backup which can be restored, I'd suggest you go this way.

What you should do to cleanup, is to use the Snapshot Manager and "Delete All" snapshots, which will merge the deltas of the first disk to its base disk. The second disk's snapshot file - which is now obsolete - will not be merged and can basically be deleted from the datastore (or moved to a sub-folder and deleted later on as a precaution).

Please take a look at http://kb.vmware.com/kb/1015180 to see how snapshots work in VMware products.

André

[mdkberry]

Hi

thanks I have deleted all from the snapshot manager and there was something in there.

it has now consolidated the vmdk on DISK_1 but when I look an hour later it is still showing the same time as when I performed the snapshot deletion.

I will observe it for now and see how it goes.

I am concerned it will happen again if I have to disconnect the DISK_1 how can I ensure it is the latest version before disconnecting in the future?

also is there a way that I can export just the VM on the 250GB 'datastore' disk without having to disconnect the 1TB 'DISK_1' hard disk that is used purely for file storage, as this is where the problem occured in the first place, I was trying to export just the Windows Server 2008 instance and not all the users data files.

I have looked through the link you sent, thanks I knew some of that already after the issue I ran into previously but that helps understand it a little better.

[a_p_]

it has now consolidated the vmdk on DISK_1 ...

Are you sure? That's not what I expected from the log files and the time stamps!? I expected "Windows Server 2008 Std R2-000001.vmdk" to be merged into "Windows Server 2008 Std R2.vmdk" and "Windows Server 2008 Std R2_1-000001.vmdk" to stay as it is as it should be an obsolete file. Do you still see no changes in the datastore browser after clicking the Refresh button?

What you should basically do to avoid such issues in the future is to check the .vmdk file name of the virtual disk (either in the VM's disk settings or in the .vmx file) and if it's name contains "...-00000x.vmdk" it has an active snapshot which you should delete prior to disconnecting the virtual disk. Another option would be to set the data disk to "Independent-Persistent", this excludes the virtual disk from snapshots.

André

[mdkberry]

Hi

the change has updated the vmdk on DISK_1 but only at the time I ran consolidation.

there are no more snapshots listed in snapshot manager

and as you can see from the screenshots I took just now the date for the vmdk on DISK_1 is still showing the time I ran the consolidation and hasnt updated.

I just want to be sure it isnt keeping updates on the Datastore for the vmdk that is on the DISK_1 hard disk.

do the attached screen shots suggest it is correct now and would be safe to remove the DISK_1 and add it back in without risk of this happening again?

I would have expected to see the DISK_1 vmdk dates updating rather than files only on the datastore

the time now is around 11:25am 25th March for reference to what is showing in the screen shots.

thanks again for your time on this

Rgds

Mark

[a_p_ · Accepted solution]

there are no more snapshots listed in snapshot manager

Although most of the time this is ok, you should not necessarily rely on this (it's only a text file which contains the snapshot information). Always check the VM's settings to ensure the virtual disk has no snapshot name (i.e. ...-00000x.vmdk). The time shown for the files, is the time when the descriptor .vmdk file (virtual disks consist of two .vmdk files, a header/descriptor files and a data file, but only the name and time stamp of the header file along with the size of the data file is shown in the Datastore Browser) was last modified, so this is ok if it shows the time when you ran the consolidation.

do the attached screen shots suggest it is correct now and would be safe  to remove the DISK_1 and add it back in without risk of this happening  again?

I'm pretty sure DISK_1 in the VM's settings point to the "Windows Server 2008 R2 Std_1.vmdk" and "Windows Server 2008 R2 Std_1-000001.vmdk" now obsolete and can be deleted. As mentioned earlier, I'd suggest you move this file to a new sub-directory and delete it later if everything runs as expected.

André

[Noonan1970]

It seems like your backup process is very manual.  Have you ever thought about utilizing an image based backup solution.  There are some very powerful yet inexpensive backup solutions on the market today, that are purpose built just for your virtual environment. I use PHD Virtual Backup and Replication and I am very happy with it.  It is very simple to install and manage, along with providing a high level of deduplication and application consistent backups at the sametime.  If you want more information, checkout http://www.phdvirtual.com

[mdkberry]

thanks André you have been a great help and guided me to a better understanding of this.

The good news is the backup data was all available and restored for the client, so this has been a lesson without being a disaster.

[mdkberry]

Hi Noonan1970

thanks for that information I had not yet found anything to replace the manual option I have been using until now. I will definitely look at that solution