Unexpected Shut Down

[PeteAdmin]

Hi,

I have a hosted solution (they look after the hardware, I look after the VM's and Windows). Three servers DC, Exchnage 2010 and a Remote Desktop. I have very little experience with ESXi

The remote desktop has unexpectedly shut down this morning. It's happened before one or twice I think. Windows doesn't have anything useful logged. The ESXi warning logs for today are:

2018-02-09T10:05:53.835Z cpu29:4740708)WARNING: World: vm 4740708: 3562: VMMWorld group leader = 4740709, members = 6

2018-02-09T10:05:53.835Z cpu11:4740723)WARNING: VSCSI: 1223: scsi0: bad attmpt to kill async processing world.

2018-02-09T10:06:00.244Z cpu1:65649)WARNING: PFrame: vm 4740709: 2486: Deallocating pinned bpn 0x100279000, pinCount 1 throttle 0.

2018-02-09T10:13:43.725Z cpu4:5176533)WARNING: NetDVS: 681: portAlias is NULL

Does this relate to the shutdown? It was reported in Windows at 10.05 so the times about right.

Any other logs I should looking at?

Any help gratefully received.

Thanks Pete

[pwilk]

Is the remote desktop provided through a Terminal Server or is it a Windows 10/7 box with RDP enabled?

What's your version of ESXi and (if it's being used) vSphere?

Cheers,

Paul Wilk

[kishorenelvagal]

Hi Pete ,

When you say the remote desktop shutdown. Do you mean the remote desktop service or the Virtual machine which shutdown.

It is always a good idea to investigate the VMmachine logs as against the ESXi logs. You will be able to find the VM logs in the VM directory.

vmfs/volumes/datastore/VMfolder/  vmware.log.

Feel free to cope snippets from the timestamp and we should be able to find the cause from the logs.

Please refer the following KB which will give you steps to find the cause.

VMware Knowledge Base

[PeteAdmin123]

Hi Paul,

It's a Server 2012 R2 OS, set up with RDS.

ESX info:

Version 1.8.0

Build number 4516221

ESXi Version 6.5.0

ESXi build number 4564106

I'm not sure what vSphere is.

The shutdown hadn't happened since I posted this, but it's just happened at 9.09 this morning.

Thanks

Pete

[PeteAdmin123]

Hi Kishorenelvagalu,

It is the Virtual machine that shuts down. Just the Remote Desktop one, the Domain Controller and Exchange server are fine.

It's just happened again:

Remote on RS25.racksrv.com.racksrv.com in ha-datacenter is powered off

Type Info

Time Friday, 16, Mar., 09:09:30

Virtual machine Remote

Error message on Remote on RS25.racksrv.com.racksrv.com in ha-datacenter: We will respond on the basis of your support entitlement.

Type Info

Time Friday, 16, Mar., 09:09:12

Virtual machine Remote

I'm looking at the link you posted, will reply again soon.

Thanks

Pete

[pwilk]

Hi Pete,

It can be number of reasons for the crash and reboots. Pelase check this KB article​

Can you post the PSOD screenshot and vmkernel logs?

[PeteAdmin123]

Attached is what I see when logged into VMWare portal. I don't have anywhere else to log in, so I'm not able to follow the linked guides first bit. Hopefully I found the log storage anyway.

Attached is the latest log from the Storage\SSD\Remote\VMware-9

There's a lot of Panic's logged which is worrying.

I hope that helps. If not please let me know what other logs I should be looking for.

Thanks a lot

Pete

[PeteAdmin123]

Hi Paul,

Attached is VMKernal.log

What's the PSOD?

Thanks

Pete

[PeteAdmin123]

Ok, Purple Screen Of Death. Not sure where to find it.

Thanks

Pete

[PeteAdmin123]

Is it possible to set the virtual server to restart automatically if it shuts down again before working out the issue?

Thanks

Pete

[pwilk]

Hi Pete,

You can set automatic restart if you follow this KB article VMware Knowledge Base

By PSOD I meant both Purple&Blue Screens of Death - have you spotted either of them when the issue happens? Purple screen of death is caused by ESXi/vSphere crash, while BSOD by Windows. If you spot any of them happening, please take a screenshot or note down the error displayed at the same time and post it here for further investigation.

The vmkernel log you supplied clearly shows errors with the datastore. It seems to me that the datastore which your VM is based on is disconnecting from the host - this is a major issue and this is probably why your VM shuts down. What is the storage type that you use in your environment? Is it a vSAN, iSCSI or local storage? Please double check if you can see any errors on this side of the system. Maybe try migrating the VM onto different storage and see if the issue persist.

Please let me know what you find

[bluefirestorm]

Have a look at this KB https://kb.vmware.com/kb/2149642

The circumstances of the KB looks similar; VM with Terminal Services crashes. The partial log you pasted contains the line

2018-03-15T14:28:55.032Z| vcpu-2| I125: GuestMsg: Too many channels opened.

The KB indicates it is fixed in 6.5U1. You could also try the workaround specified in the KB.

[PeteAdmin123]

Hi Paul,

The storage is local. One SSD array and one SATA array, both RAID'd.

All three VM's are on the same storage. Is it likely that this would affect one VM but not the other two? I think all three servers C drives are SSD and the other partitions are SATA

Is there an error I could look for in the other VM's vmkernal log to see if it is occuring on others?

Thanks

Pete

[PeteAdmin123]

Hi Bluefirestorm,

Thanks for this info. I've had a quick read of the link and it looks a good match. The remote server does have 30+ users logging on at around 9am, it crashed at 9.09am.

I'll try the update or workaround over the weekend and feedback.

Thanks

Pete

[PeteAdmin123]

I've deleted the registry key as per this article https://kb.vmware.com/kb/2149642

I'll let you know if the issue occurs again.

Thanks everyone for advice etc.. much appreciated.

Pete