stuartcallison
Contributor
Contributor

VM shows alert icon with no alarms triggered

I have been having phantom alarms lately. Virtual machines will show the red alert icon with no alarms triggered. I checked the VM, the host, the cluster, the folder, the datacenter, and vcenter but none of those objects have any triggered alarms either. The cluster does not have HA or DRS enabled.

I had several VMs in this condition so I restarted vCenter server, which usually solves the problem for a while. However, this time two of the VMs have retained their alert state through the vCenter server restart.

I ran this powercli script on the trouble VMs and every alarm state was either gray or green. How can I find out what is causing the VMs to red exclamation point alert symbol?

$vm = Get-VM -Name VM | Get-View
foreach($state in $vm.DeclaredAlarmState){
    $alarm = Get-View $state.Alarm
    Write-Host "name:"
    Write-Host $alarm.info.name
    Write-Host "overall status:"
    Write-Host $state.OverallStatus
}
14 Replies
stuartcallison
Contributor
Contributor

I forgot to mention, the hardware status tab of each host shows all green checkmarks, there is nothing indicated wrong there.

0 Kudos
mschau
Contributor
Contributor

My Lab environment did this to me today.   I had recabled my environment so I had shutdown the SAN and my two lab hosts.  When they came up I manually started my DCs, SQL and vCenter etc directly on the hosts. When I signed into vCenter I saw alerts on ALL of my VMs that were located on my SAN. Some how the VMs did not register correctly. (Maybe my hosts came up before the SAN).  A quick vmotion between hosts cleared the alert without having to remove the VM and re-adding it to Inventory.

withanh
Contributor
Contributor

I had the same issue in my v5 lab and a vMotion was just what the Dr ordered.

Thanks!!

0 Kudos
Jt3rry
Contributor
Contributor

I've also found that vMotion clears the phantom alarm... Is this an ESX 5 bug? We're running 5.0.0 build 504890 and have seen this behavior on 4 hosts running 2008 R2 VMs.

The left-hand red alert doesn't clear on the VM, yet when you click into the VM's alarms tab the event is missing making it hard to troubleshoot what causes the alarm in the first place.

Any ideas?

0 Kudos
kruddy
Enthusiast
Enthusiast

I saw this in our DR site this morning after upgrading the site to ESXi 5 Update 1 and also upgrading the SAN's firmware as well.

I think mschau has it right, I think the host came up before the SAN did and all of the systems were probably disconnected for a couple seconds.

Eitherway, a reboot fixed everything and cleared all the alerts.

Please, don't forget the awarding points for "helpful" and/or "correct" answers.
0 Kudos
mschau
Contributor
Contributor

If you don't want to reboot I found that just vmotioning the VMs from one host to another cleared up the alerts.

0 Kudos
onerobertone
Contributor
Contributor

I had this problem today.  The fix was to reconfigure HA at the cluster level and then the alarm could be seen and then removed.

0 Kudos
lm31
Contributor
Contributor

I saw this problem as well - services.sh restart fixed it for me.

0 Kudos
Northsaloplad
Contributor
Contributor

I had the same issue and it turned out to be licencing

0 Kudos
richydom
Contributor
Contributor

"Restart Management Agents" on the esx host, where the alarm vm is running solves the problem.

0 Kudos
allison_gibb
Contributor
Contributor

vMotion didn't clear my alarm but unticking HA cleared the alarm and then I was able to tick HA again OK. Thanks

0 Kudos
govandsinjari
Contributor
Contributor

One of my ESXis went offline after hardware failure, after fixing it, the red alert triangle appeared on all VMs, and there is no way to clear it, this is if I am logged in through the vCenter, BUT there were no alerts if you login directly to ESXi.

Solution:

From the vCenter, click on the ESXi host "Configuration" tab, select "Security Profile", then from Services properties I have restarted VPXA services which is vCenter account on the ESXi, I lost connection to ESXi host temporary but it recovered after "Reconnect" and the alerts cleared!

cgprime
Contributor
Contributor

I have confirmed that the answer provided by govandsinjari will resolve the issue.

In my case, I brought up the ESXi host before the HNAS storage had come back online.

- Restart VPXA for the ESXi host the VM's that display the alert reside

- A new alert will show up for the disconnected host, Connect the host to vCenter

- Clear the disconnected host alert and you will be clear

0 Kudos
nhickey
Enthusiast
Enthusiast

This is a valuable thread as I have found myself digging this out twice before.

As govandsinjari states below you can do this method from within the vSphere C# client or you can also perform the same function via SSH to your ESXi host.

govandsinjari wrote:

Solution:

From the vCenter, click on the ESXi host "Configuration" tab, select "Security Profile", then from Services properties I have restarted VPXA services which is vCenter account on the ESXi, I lost connection to ESXi host temporary but it recovered after "Reconnect" and the alerts cleared!

-SSH to ESXi

     Command to type:
     /etc/init.d/vpxa restart

You will get the same results as if using the vSphere client less all the errors from the disconnects that the client spits out since you are not connected to the vCenter and performing the task.

Thanks you and Good Luck!

Nigel

Thanks, Nigel VCIX-DTM @vCenterNerd nigelhickey.com
0 Kudos