VMware Cloud Community
DJPanic
Contributor
Contributor

Server down, space issue, stuck at 99% and I'm screwed...

So our Virtual server crashed. The guy that set it up has left the country.  I'm an IT guy but pretty thin on VMware, but I can tell you we have Vsphere 4.1 and I'm able to get into the console. Several other servers are all up and running fine.  This one just won't do anything.  Everything is greyed out since I tried to delete an old Snapshot.  Sitting at the glorious 99%   For about an hour now.  Sure it could take forever. But worse, they're could be more problems when and if it does finish!  And people will be arriving for work in about 3 hours.  Somebody, anybody please help!  Very appreciative and greatful for any help or direction anyone can give me.  Thanks in advance.


Also, found an error in the log that says "Cannot power on Fileserver blahblah01.  A general system error occurred.  No related events.  😕

Tags (2)
25 Replies
brunofernandez1

we need more information:

do you use vcenter or standalone esxi server?

what kind of storage? FC, iSCSI, DAS

What kind of vm is it? windows? linux?

the snapshot is probabelly very old right?

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

vcenter

iSCSI I believe

Windows Server 2008

Windows 7 Pro client

The snapshots are from 2013. Most of them. I just picked a random one and tried to delete. Thank God I didn't hit delete all!

Reply
0 Kudos
brunofernandez1

a snapshot from 2013? ok, it's clear that it needs his time :smileygrin:

I would recommend to wait until the snapshot is completetly deleted. you may run into problems (vm corruption) if you cancel the job.

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
DJPanic
Contributor
Contributor

Hey thanks for the replies, I really appreciate it.  Do you think its not hung?  Because its 2 years old it will take this long?  Like over 3 hours?  Where do i draw the line that its hung or not?  I got about 2 hours til Engineers show up and I'm a dead man...  :smileycry:

Reply
0 Kudos
brunofernandez1

when you create a snapshot of the vm and disk, vmware creates a delta disk.

from now on, the new writes to the disk go all to this delta disk.

when you delete the snapshot, vmware tries to merge all the information on the delta disk to the original vm disk.

you said that the snapshot is from 2013 so it will have probabelly a LOT of data to merge. this is one of the reason because you shouldn't keep snapshots that long.

depending on the disk system performance this can keep up to hours...

do you have the possibility to look how big are your snapshots? maybe with RV tools or something simiar...

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

Those are the vmdk files right?  Can't I see them browsing the datastore?  If so, they go from 7mb to 1.063GB (only 1 that big)

Reply
0 Kudos
brunofernandez1

right, the vmdk files...

i dont know if you can see the deltas by browsing the datastores...

but with RV tools you can see everything very detailed. Size, Date created and so on....

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

There's 124 files in there and about 60 of them are Virtual disk files. I know there's a way to easily wipe all the snapshots, but I don't know how to do that...    Is there another way to increase the size of the available space?  I'm rapidly running out of time   :-((    

Reply
0 Kudos
brunofernandez1

you are running out of space on the datastore or on the vm?

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

The datastore. At one point it said I was out of space on the datastore.

Reply
0 Kudos
brunofernandez1

is it the datastore where the VM is running on it?

do you have the possibility to resize the datastore?

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

Unsure.  I don't know how to do that.  I somehow got it to come up.  Not feeling confident though.  Says there's 12.17GB free in the datastore, yet I still can't access the virtual data...    Correction!  It came up!!!!  But its very slow....  I feel my minutes are numbered...  I'm gonna owe you big time if you help me stabilize this...  I will make that happen somehow.  That I know for sure!

Reply
0 Kudos
brunofernandez1

if you have multiple VMs on this datastore you can move one VM away to another datastore to have more space...

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

The datastore in question has only 1 VM on it.  It says Provisioned storage - 2.87TB  Not shared Storage - 1.75TB Used - 1.75  but then under the Datastore it says Capacity 1.76TB   12.17GB free and the server says 393GB free of useable space. 

Reply
0 Kudos
DJPanic
Contributor
Contributor

Oh crap.  "There is no more space for virtual disk XXXXXX.vmdk. You might be able to continue this session by freeing disk space on the relevent volume, and clicking retry.  Click Cancel to terminate this session."   Like I said the virtual volume says 393GB and the Datastore says 12.17GB free.  Any suggestions??  yikes

Reply
0 Kudos
brunofernandez1

is the free space of 12.17gb getting smaller when you click on refresh? if yes, you have to resize the datastore...we have no other possibility

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

12.17GB hasn't moved an inch (knock on wood) even after several refreshes, but I feel my time is still numbered.  How do I resize the datastore?

Reply
0 Kudos
brunofernandez1

as it is iSCSI you have to do this first on the disksystem...

then you can use this guide: VMware KB: Growing or expanding a VMFS volume or datastore

------------------------------------------------------------------------------- If you found this or any other answer helpful, please consider to award points. (use Correct or Helpful buttons) Regards from Switzerland, B. Fernandez http://vpxa.info/
Reply
0 Kudos
DJPanic
Contributor
Contributor

UGGH!  It crashed again....

Reply
0 Kudos