14 Replies Latest reply on Mar 18, 2018 7:17 AM by PeteAdmin123

    Unexpected Shut Down

    PeteAdmin Lurker

      Hi,

      I have a hosted solution (they look after the hardware, I look after the VM's and Windows). Three servers DC, Exchnage 2010 and a Remote Desktop. I have very little experience with ESXi

       

      The remote desktop has unexpectedly shut down this morning. It's happened before one or twice I think. Windows doesn't have anything useful logged. The ESXi warning logs for today are:

       

      2018-02-09T10:05:53.835Z cpu29:4740708)WARNING: World: vm 4740708: 3562: VMMWorld group leader = 4740709, members = 6

      2018-02-09T10:05:53.835Z cpu11:4740723)WARNING: VSCSI: 1223: scsi0: bad attmpt to kill async processing world.

      2018-02-09T10:06:00.244Z cpu1:65649)WARNING: PFrame: vm 4740709: 2486: Deallocating pinned bpn 0x100279000, pinCount 1 throttle 0.

      2018-02-09T10:13:43.725Z cpu4:5176533)WARNING: NetDVS: 681: portAlias is NULL

       

      Does this relate to the shutdown? It was reported in Windows at 10.05 so the times about right.

       

      Any other logs I should looking at?

       

      Any help gratefully received.

       

      Thanks Pete

        • 1. Re: Unexpected Shut Down
          pwilk Hot Shot
          vExpert

          Is the remote desktop provided through a Terminal Server or is it a Windows 10/7 box with RDP enabled?

           

          What's your version of ESXi and (if it's being used) vSphere?

           

          Cheers,

          Paul Wilk

          • 2. Re: Unexpected Shut Down
            kishorenelvagalu Hot Shot
            Skyline Community 2/2GS Skyline Support VMware Employees

            Hi Pete ,

             

            When you say the remote desktop shutdown. Do you mean the remote desktop service or the Virtual machine which shutdown.

             

            It is always a good idea to investigate the VMmachine logs as against the ESXi logs. You will be able to find the VM logs in the VM directory.

             

            vmfs/volumes/datastore/VMfolder/  vmware.log.

             

            Feel free to cope snippets from the timestamp and we should be able to find the cause from the logs.

             

            Please refer the following KB which will give you steps to find the cause.

             

            VMware Knowledge Base

            • 3. Re: Unexpected Shut Down
              PeteAdmin123 Novice

              Hi Paul,

               

              It's a Server 2012 R2 OS, set up with RDS.

               

              ESX info:

               

              Version 1.8.0

              Build number 4516221

              ESXi Version 6.5.0

              ESXi build number 4564106

               

              I'm not sure what vSphere is.

               

              The shutdown hadn't happened since I posted this, but it's just happened at 9.09 this morning.

               

              Thanks

              Pete

              • 4. Re: Unexpected Shut Down
                PeteAdmin123 Novice

                Hi Kishorenelvagalu,

                 

                It is the Virtual machine that shuts down. Just the Remote Desktop one, the Domain Controller and Exchange server are fine.

                 

                It's just happened again:

                 

                Remote on RS25.racksrv.com.racksrv.com in ha-datacenter is powered off

                Type Info

                Time Friday, 16, Mar., 09:09:30

                Virtual machine Remote

                 

                Error message on Remote on RS25.racksrv.com.racksrv.com in ha-datacenter: We will respond on the basis of your support entitlement.

                Type Info

                Time Friday, 16, Mar., 09:09:12

                Virtual machine Remote

                 

                I'm looking at the link you posted, will reply again soon.

                 

                Thanks

                Pete

                 

                 

                • 5. Re: Unexpected Shut Down
                  pwilk Hot Shot
                  vExpert

                  Hi Pete,

                   

                  It can be number of reasons for the crash and reboots. Pelase check this KB article

                   

                  Can you post the PSOD screenshot and vmkernel logs?

                  • 6. Re: Unexpected Shut Down
                    PeteAdmin123 Novice

                    Attached is what I see when logged into VMWare portal. I don't have anywhere else to log in, so I'm not able to follow the linked guides first bit. Hopefully I found the log storage anyway.

                     

                    Attached is the latest log from the Storage\SSD\Remote\VMware-9

                     

                    There's a lot of Panic's logged which is worrying.

                     

                    I hope that helps. If not please let me know what other logs I should be looking for.

                     

                    Thanks a lot

                     

                    Pete

                    • 7. Re: Unexpected Shut Down
                      PeteAdmin123 Novice

                      Hi Paul,

                       

                      Attached is VMKernal.log

                       

                      What's the PSOD?

                       

                      Thanks

                       

                      Pete

                      • 8. Re: Unexpected Shut Down
                        PeteAdmin123 Novice

                        Ok, Purple Screen Of Death. Not sure where to find it.

                         

                        Thanks

                         

                        Pete

                        • 9. Re: Unexpected Shut Down
                          PeteAdmin123 Novice

                          Is it possible to set the virtual server to restart automatically if it shuts down again before working out the issue?

                           

                          Thanks

                           

                          Pete

                          • 10. Re: Unexpected Shut Down
                            pwilk Hot Shot
                            vExpert

                            Hi Pete,

                             

                            You can set automatic restart if you follow this KB article VMware Knowledge Base

                             

                            By PSOD I meant both Purple&Blue Screens of Death - have you spotted either of them when the issue happens? Purple screen of death is caused by ESXi/vSphere crash, while BSOD by Windows. If you spot any of them happening, please take a screenshot or note down the error displayed at the same time and post it here for further investigation.

                             

                            The vmkernel log you supplied clearly shows errors with the datastore. It seems to me that the datastore which your VM is based on is disconnecting from the host - this is a major issue and this is probably why your VM shuts down. What is the storage type that you use in your environment? Is it a vSAN, iSCSI or local storage? Please double check if you can see any errors on this side of the system. Maybe try migrating the VM onto different storage and see if the issue persist.

                             

                            Please let me know what you find

                            • 11. Re: Unexpected Shut Down
                              bluefirestorm Master

                              Have a look at this KB https://kb.vmware.com/kb/2149642

                               

                              The circumstances of the KB looks similar; VM with Terminal Services crashes. The partial log you pasted contains the line

                               

                              2018-03-15T14:28:55.032Z| vcpu-2| I125: GuestMsg: Too many channels opened.

                               

                              The KB indicates it is fixed in 6.5U1. You could also try the workaround specified in the KB.

                              • 12. Re: Unexpected Shut Down
                                PeteAdmin123 Novice

                                Hi Paul,

                                 

                                The storage is local. One SSD array and one SATA array, both RAID'd.

                                All three VM's are on the same storage. Is it likely that this would affect one VM but not the other two? I think all three servers C drives are SSD and the other partitions are SATA

                                 

                                Is there an error I could look for in the other VM's vmkernal log to see if it is occuring on others?

                                 

                                Thanks

                                 

                                Pete

                                • 13. Re: Unexpected Shut Down
                                  PeteAdmin123 Novice

                                  Hi Bluefirestorm,

                                   

                                  Thanks for this info. I've had a quick read of the link and it looks a good match. The remote server does have 30+ users logging on at around 9am, it crashed at 9.09am.

                                   

                                  I'll try the update or workaround over the weekend and feedback.

                                   

                                  Thanks

                                   

                                  Pete

                                  • 14. Re: Unexpected Shut Down
                                    PeteAdmin123 Novice

                                    I've deleted the registry key as per this article https://kb.vmware.com/kb/2149642

                                     

                                    I'll let you know if the issue occurs again.

                                     

                                    Thanks everyone for advice etc.. much appreciated.

                                     

                                    Pete