Virtual Machine UPTime

[gboskin]

looking for a script to get a list of virtual machins in a cluster and each virtual machine UP time....

[LucD · Accepted solution]

The VirtualMachine object keeps track of the last boottime but with a catch.

The VI only considers as boot when a guest changes it's state from powered off to powered on.

That means that when you supend/resume a machine this is not considered as a new boot (which in fact is correct in my opinion).

Luckily the same object also keeps track of the total time a guest was in suspend and when the last resume was performed.

So a script that shows it all could be something like this

$report = @()
Get-Cluster <cluster-name> | Get-VM | Get-View | %{
  $row = "" | select Name, PowerState, LastBoot, LastSuspend, TotalSuspend
  $row.Name = $_.Name
  $row.PowerState = $_.runtime.powerState
  $row.LastBoot = $_.runtime.bootTime
  $row.LastSuspend = $_.runtime.suspendTime
  $row.TotalSuspend = $_.runtime.suspendInterval
  $report += $row
}
$report | ft

Now the suspend related properties don't always seem to get updated correctly in my test environment (VC 2.5u4 and ESX 3.5u4).

Give it a try and let me know if you get any correct suspend related values ?

A better solution, and what I'm currently also using, is the sys.uptime.latest metric.

This returns for a guest the number of seconds since startup.

Get-Cluster <cluster-name> | Get-VM | `
  Get-Stat -Stat sys.uptime.latest -MaxSamples 1 -Realtime | `
  select Entity, Value, Unit

If the number of seconds for a guest is -1, it means the guest was powered off or suspended in the last hour.

If the guest was powered off longer than 1 hour ago, the Get-Stat cmdlet will return an error message stating "The metric counter "sys.uptime.latest" doesn't exist...".

To avoid this you can filter out the powered off guests.

Get-Cluster <cluster-name> | Get-VM | `
  where {$_.PowerState -eq "PoweredOn"} | `
  Get-Stat -Stat sys.uptime.latest -MaxSamples 1 -Realtime | `
  select Entity, 
         @{Name="Boottime";
	     Expression={(Get-Date).AddSeconds(- $_.value).ToString("yy/MM/dd HH:mm:ss")}}

In this script I converted the number of seconds into the last boot time of the guest.

[gboskin]

Yes LuCd works well but as you said the first script reports last suspend time as 01/01/01 ( Which of course is wrong ).. the second scripts perfectly fine but as you clearly explaned does not take the reset or suspend into consideration..

The senario we have is that we had an IPStor live failover test and all the imgeas seems to be up ..justy want to make sure no image actually restarted ..the last Boot time on all the images seems ok though

Also the last boot time could also mean the vCeneter lost network connection with the images and the images might still be running ??? True??

[JerryH]

Hi,

What's your experience with "sys.uptime.latest" Luc?

I've tried using that value in our environment but it reports odd values?

Sometimes it identifies migrations as boottimes for instance, making the value of no use to us.

I'm curious to know if it's just our environment that does this or if it's a common problem?

/Jerry

[LucD]

I suspect this is due to the -Realtime interval you specify on the Get-Stat cmdlet.

That parameter means the metric is obtained from the ESX server itself, and when you migrate it is of course another ESX server.

If you use one of the historical intervals on the VC you should get the correct value.

Only problem there is that you loose the fine granularity (20 seconds) you have on the ESX server (with -Realtime).

And you have to take into account the metrics roll up that happens on the VC.

You won't see the updated value until the roll up has happened.

I don't think you should use this value to retrieve the exact boot time.

For that you should go to the OS in the guest itself.

[JerryH]

Hi,

I got other values when i didn't use the RealTime param, but i'm still off by a month sometimes...

But nm that, i'll try other ways.

The reason i want it is because we want to see what uptime we deliver on the virtual HW, SLA stuff.

Thnx for writing on the forum btw, you write awesome scripts!

/Jerry

[gboskin]

I agree with Jerry I am getting a months off with the uptime... So basically our we saying the VI does not know when a vm has be restarted correctly??

[LucD]

Afaik the uptime has been and still is a problem.

There are numerous mentions of this in other threads in several communities.

To recapture, there are 2 types of "uptime"

1) the uptime of the virtual HW.

This is reported correctly as long as you don't do any vMotions. See for example Jason's blog.

This shows the time since the last power on of the virtual machine.

2) the uptime of the OS running in the virtual HW.

In my opinion this is not something that the VC can or should know (unless a component is added to the VMware Tools).

For OS uptime you will need to run monitoring solutions that poll the OS in the guest (with an agent or agentless).

And that could be a problem for your guests that are isolated from your monitoring solution.

In conclusion, you will need a 3rd party solution to have correct OS uptime figures.

Btw, if someone from VMW can contradict this I would be a happy man

[admin]

That's correct.

I believe there will be a fix for the VMotion resetting the uptime in the future, but as far as I've heard it won't be backported. At this point your best bet is guest-level monitoring.

[dirch201110141]

A bit embarrassing, but googling info here, there and everywhere has not given any clarity so I will bump this 4 year old post anyways: Does anyone know if the uptime reset during vMotion has been solved?

My latest try is with powershell 2.0 vs vSphere 5.1.1235233 and the reset is still present.

I am testing with "Get-Stat -Entity <insert favorite servername> -start (get-date).AddDays(-30) -Finish (Get-Date)-MaxSamples 1000 -stat sys.uptime.latest|Out-GridView" on a server that I have migrated and booted on known dates.

[LucD]

No, the uptime reset is still there.

[CoinGrahamIV]

Just posting to let anyone searching know that yes, it's still a problem in ESXi 5.5 over a decade later.

[esxi1979]

So as Lucd mentioned, "In conclusion, you will need a 3rd party solution to have correct OS uptime figures."

The script does give odd values.

I had one HV died, then when i tried to find the uptime of the vm .. this script shows vm rebooted but in real it do not ..

Anyone is there any fix.solution yet here ?

[esxi1979]

Anyone ?

[ShoHRock]

Hello All,

Can someone please tell me why there is a discrepancy when we fetch the up-time of a VM.

sys.uptime.latest & $_.runtime.bootTime shows same results (36 days) but its quite different from UP-Time mentioned in the task manager (2 days) of the VM (See screenshot).

Please help!

[ShoHRock]

Hello All,

Can someone please tell me why there is a discrepancy when we fetch the up-time of a VM.

sys.uptime.latest & $_.runtime.bootTime shows same results (36 days) but its quite different from UP-Time mentioned in the task manager (2 days) of the VM (See screenshot).

Please help!

[LucD]

I suspect this might be due to the difference in the last reboot of the VM and the last reboot of the guest OS.

The last guest OS reboot should be logged in the guest OS eventlog.

Also see my earlier answer in this thread.