How can I get an alarm for VM in bad state when running out of space on datastore

[Lisbjerg]

Hi

After ignoring alarm messages about diskspace almost used, a VM stops working with this error message:

"Message from VMHost1.test.net: There is no more space for virtual disk

/vmfs/volumes/4fb7ee7c-9c2aceee-a0df-00215acb6f4a/VMHost1/VMHost1_1_1.vmdk. You might be able to

continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this

session. "

At this point i would like to have another alarm to ignore 😉

I'm trying to find a VMware alarm that can alert me in this case. (Like the "Datastore usage on disk" alarm set up to warn me when 90% is used)

[JarryG]

I do not know how to do it in ESXi but you can still set-up whatever alarm you want on your log-collector server. Just create filter with given string-pattern...

But if you like ignoring alarms so much then you can set it up using cron, every minute or so...  😉

[Lisbjerg]

@JarryG: Thanks

But would prefer to have it in ESXi as an alarm, if possible

[MGlasson]

you need vcenter for alarms,

then just create a custom alarm for datastore usage.

[Lisbjerg]

@MGlasson

I have vCenter (Sorry, forgot to mention that in the original post)

I also have the alarm for the datastore usage - thats not what I'm looking for.

I want the alarm when the VM stops, because it has run out of diskspace.

[MGlasson]

there is no alarm for a paused ( stunned ) vm ( pausing and suspending are different ) that I am aware of.

setting an alarm for the messages when a vm is paused might work.

these triggers are under " Monitor for specific events occuring on this object, for example, VM powered On " in alarm settings.

I have never attempted this so I cant be of any more help than this,

except to advise don't ignore storage space alarms

Good Luck

[Lisbjerg]

@MGlasson

Can't find the paused VM alarm anywhere.

Thanks for the advice about the space alarms though 🙂

But I would really like that extra alarm, when the VM stops working.

Isn't it possible to get an alarm based on the text/message in the Event log on the VM?

Because I get the message in the Event log:

"Message from VMHost1.test.net: There is no more space for virtual disk

/vmfs/volumes/4fb7ee7c-9c2aceee-a0df-00215acb6f4a/VMHost1/VMHost1_1_1.vmdk. You might be able to

continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this

session. "

[Shrikant_Gavhan]

Hi ,

You can configure it please check below steps.

[MGlasson]

Shrikant_Gavhane

suspended vm's and paused ones are 2 different functions,

When a vm is suspended it writes it's state to disk and frees up the memory.

When one is paused due to a full disk all activity is stopped and the current state is held in memory.

I don't think it shows as a state of suspended but I could be wrong, might give it a test on one of my servers.

[Lisbjerg]

Hi

The "VM being suspended" alarm didn't work.

[Shrikant_Gavhan]

Hi Lisbjerg,

I didnt find any information on VM PAUSE operation but there is one for suspeded VM's.

I have added this trigger in my environment and it worked fine for me. in the triggers window there are two triggers VM being suspended and VM suspended.

I tried VM suspended trigger and it worked for me

I have email alerts configured in my env and I getting it for VM suspend operation.

Here is content from email alert.

Target: datastoretest

Previous Status: Gray

New Status: Red

Alarm Definition:

([Event alarm expression: Guest OS shut down; Status = Red] OR [Event alarm expression: VM powered off; Status = Red] OR [Event alarm expression: Guest reboot; Status = Red] OR [Event alarm expression: Upgrading VM; Status = Red] OR [Event alarm expression: VM suspended; Status = Red])

Event details:

datastoretest on XXXXXXXXXXXXXXXXX in XXXXXX is suspended

Thanks and Regards,

Shrikant Gavhane.

[tomtom901 · Accepted solution]

Basically, when the datastore is full, and a VM on it is requesting I/O, due to the fact that it is thin provisioned, it is being paused by ESXi. All VM's that are not requesting I/O can still run. When this happens, you get an error message like this:

What the VM throws at you is basically a VM Message. Using vCenter, you can create an alarm that triggers when the VM has (a specific) information message. The message I used was something like this:

When I then let a VM on a thin provisioned datastore, using thin provisioned disks run out of space, my custom alarm was triggered.

N.B. I only saw the VM's icon in alarm state after I clicked retry on the popup. If you don't do this, the VM's icon will be in question state, but the custom alarm will still be triggered.

Hope this helps,

Tom

[Lisbjerg]

Hi Tom

Sorry for the late reply - was in Italy skiing

Thanks - it worked like a charm.

I even added text to the advanced section, to be sure I only got these kind of alarms.

Again: Thanks.

Regards

Jan