1 2 Previous Next 29 Replies Latest reply on Mar 8, 2018 9:45 AM by sarikrizvi Go to original post
      • 15. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
        Rajeev Sharma Enthusiast

        @ThijsW

        As I see you are getting events >>>failed with error No buffer space available (55)<<<< This could be that few applications might have failed to close TCP sockets. Have tried to reboot your host.

        Do check this too VMware KB: ESXi 5.1 host is disconnected in vCenter Server and reports the error:No buffer space available

         


        Rajeev

        Cheers Rajeev
        VCP6-DCV, VTSP-SV, VSP-SV, MCSE, CCNA
        • 16. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
          kellogstri Novice

          I highly believe the application in our case is Veeam One or/and Veeam Backup and Replication

           

          Now to the question why would this causes ESXi to be unstable? Does it mean ESXi 5.1 host is easily subjected to DOS attacks?

          • 17. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
            ThijsW Novice

            Hi,

             

            All issues are gone after using HP Service Pack for ProLiant (SPP).

             

            Regards,

            Thijs

            • 18. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
              kellogstri Novice

              Hi Thijs,

               

              Do you happen to use any Veeam product on this setup?

              • 19. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                ThijsW Novice

                Yes, Veeam One monitor and Backup and Replication

                 


                • 20. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                  kellogstri Novice

                  I have an ongoing case with Veeam and it appears that Veeam One could cause this problem.

                   

                  My support case with them is Veeam Support - Case # 00503976

                   

                  I have uninstalled Veeam One and very fearful to run the Backup and Replication too

                  • 21. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                    jasolution Lurker

                    Did you ever find a resolution to this or determine that the root cause was Veeam B&R for sure? I'm currently running Veeam B&R 7.x (latest Patch #4 build) and having this issue with one of our ESX 5.1 hosts. All hardware and drivers are verified on the HCL in our environment and it only seems to be affecting one of the hosts.

                     

                    Jason

                    • 22. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                      kellogstri Novice

                      Hi Jason,

                       

                      My solution was to stop using Veeam and has stop since then and never had such problems occur.

                       

                      I have been looking for VM replication solution which does not use snapshot on the esxi as well ... do you know any?

                       

                      btw what is the datastore which you are using?

                      • 23. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                        jsc8041 Lurker

                        We are having similar issues and too use Veeam One & VB&R.  I was wondering if anything resulted from your ticket with Veeam?

                         

                        thanks,

                        Claude

                        • 24. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                          kellogstri Novice

                          Hi Claude,

                           

                          My resolution to this was stop using Veeam and it has been great!

                           

                          Right now i am going to use Tintri replicate VM for my DR strategy.

                           

                          May I know what is your storage?

                          • 25. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                            Sharantyr3 Enthusiast

                            Hi there,

                             

                            I'm digging up this thread (sorry for my english) because we had a major issue in our VI infrastructure that seems to be the same problem.

                            We are using veeam b&r and veeam one, and some VMs on a specific host and a specific datastore were completly locked.

                             

                            Unable to poweron VMs, neither migrate them. I tried remove readd inventory VMs and host, no way.

                            vmkfstools -D gives no owner :

                             

                            Lock [type 10c00001 offset 58503168 v 3975, hb offset 4112384

                            gen 521, mode 0, owner 00000000-00000000-0000-000000000000 mtime 5962377

                            num 0 gblnum 0 gblgen 0 gblbrk 0]

                            Addr <4, 101, 78>, gen 3917, links 1, type reg, flags 0, uid 0, gid 0, mode 600

                            len 0, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192

                             

                            I had to remove from inventory VMs, create a new VM, move vmdk into that new VM's directory then add vmdk to the new VM and it works.

                             

                            For guys like me who have the problem, you can get the actual owner of the file using rm (or maybe it displays it too when trying to vmotion).

                            When you try to rm the file, the esxi will timeout :

                             

                            rm: can't remove 'HAProxy-01-ctk.vmdk': Resource temporarily unavailable

                            rm: can't remove 'HAProxy-01.vmx.lck': Resource temporarily unavailable

                            rm: can't remove 'HAProxy-01.vmx~': Resource temporarily unavailable

                            rm: can't remove 'vmware.log': Resource temporarily unavailable

                            BUT in the logs you will get the faulty ESXi :

                            2016-02-23T12:58:53.875Z cpu24:9937646)DLX: 4230: vol 'BJ-ISN-06', lock at 43160576: [Req mode: 1] Not free:

                            2016-02-23T12:58:53.875Z cpu24:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960

                            gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709

                            num 0 gblnum 0 gblgen 0 gblbrk 0]

                            2016-02-23T12:58:53.875Z cpu24:9937646)Res3: 5732: Rank violation threshold reached: cid 0xc1d00002, resType 4, cnum

                            2016-02-23T12:58:56.069Z cpu0:9937646)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:

                            2016-02-23T12:58:56.069Z cpu0:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960

                            gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709

                            num 0 gblnum 0 gblgen 0 gblbrk 0]

                             

                            Ok so it happens to have a faulty lock from time to time, but what was really absolutly WEIRD is the faulty ESXi having 3c4a926c279c as @MAC was in a DIFFERENT cluster off-site !!!

                             

                            I was wondering why would this happen as this particular VM would never be run on this separate cluster, then I noticed that my VEEAM ONE server was running on this specific off site host !!!!

                             

                            I'm using HP blades with latest SPP and latest VMware patches on esxi 5.1

                             

                            So solution n°1 : annoying but recreate the VM and attach the old vmdk

                            solution n°2 : reboot esxi hosting veeam-one server. That may or may not works, I used solution n°1 then found n°2 ...

                             

                            Hope this can help

                            • 26. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                              Sharantyr3 Enthusiast

                              Well for some reason I can't edit my own post, so I add new infos here :

                               

                              After vmotion the veeam-one server to another server, I could rm -rf the old VM directory and here is what I got in the vmkernel logs :

                               

                              2016-02-23T13:09:43.259Z cpu26:5686842)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:

                              2016-02-23T13:09:43.259Z cpu26:5686842)[type 10c00005 offset 43160576 v 78794, hb offset 3624960

                              gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709

                              num 0 gblnum 0 gblgen 0 gblbrk 0]

                              2016-02-23T13:09:43.260Z cpu26:5686842)DLX: 3321: Clearing wrong owner for lock at 43160576 with [HB state abcdef01 offset 3624960 gen 1088 stampUS 1480954430581 uuid 00000000-00000000-0000-000000000000 jrnl <FB 0> drv 14.58]

                              • 27. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                                k0eff Lurker

                                Hi,

                                I'm experiencing exactly same issue but I'm running on 6.0.

                                Here is described the issue and actually I have all the symptoms described here:

                                VMware Knowledge Base

                                but I can't confirm for log files. Currently host disconnected, but alive /on some way/.

                                and Yep  as some of view says, I also running veeam backup and replication with veeam ONE.

                                So is it possible issue still be alive?!

                                • 28. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                                  sarikrizvi Enthusiast

                                  Try below cmds to check/start hostd service

                                   

                                  /etc/init.d/hostd status >>>> /etc/init.d/hostd start  

                                   

                                  Check hostd status , if it's not started then check /var/log/vmkernel.log , /var/log/hostd.log and /var/log/vpxa.log

                                   

                                   

                                  When esxcli cmd not working you can run localcli cmd instead of esxcli .

                                   

                                  # localcli network firewall get

                                  # localcli network firewall set --enable false

                                  # localcli system maintenanceMode set --enable true

                                  # localcli vm process list

                                  # localcli vm process kill -w <World ID> -t soft     (Shutdown VMs)

                                  • 29. Re: ESXi 5.1 - Connect to localhost failed: Connection failure
                                    sarikrizvi Enthusiast

                                    Try below cmds to check/start hostd service

                                     

                                    /etc/init.d/hostd status >>>> /etc/init.d/hostd start  

                                     

                                    Check hostd status , if it's not started then check /var/log/vmkernel.log , /var/log/hostd.log and /var/log/vpxa.log

                                     

                                    When esxcli cmd not working you can run localcli cmd instead of esxcli .

                                     

                                    # localcli network firewall get

                                    # localcli network firewall set --enable false

                                    # localcli system maintenanceMode set --enable true

                                    # localcli vm process list

                                    # localcli vm process kill -w <World ID> -t soft     (Shutdown VMs)

                                    1 2 Previous Next