This is for newly installed VCentere 5.1 and Vsphere5.1 , no VMs created yet .
In Vcenter 5.1 when trying to add that new Vsphere host (5.1 --- VMkernel xxxx.xxx.xxx 5.1.0 #1 SMP Release build-799733 Aug 1 2012 20:03:00 x86_64 GNU/Linux) Vcenter retrieves all the information, I can add the license, push finish and it stays like trying to connect for a few minutes then it get back with :Timed waiting for vpxa to start.
THere are not VMs installed, so nothing about snapshots stored anywhere...
I have connected to that VMware Host using the VMware client and the VPXA is running , I have also restarted everything several times and I went through the internet for days but in the end I have not found what the root cause for this might be?
Also I can :
$ telnet xxx.xxx.xxx.xxx 902
Connected to xxx.xxx.xxx.xxx.
Escape character is '^]'.
220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported
Can you post your hostd.log, vmkwarning.log and vpxa.log files so that we can see if there are any clues?
See log locations;
Tahanks for thee tip in relation with log files.
Quite a few things have changes since previous versions.
Here are some errors I've picked out. can you confirm that there are not DNS issues and no ports blocked between vCenter and your hosts?
2012-12-19T14:33:20.387Z [FFA7A6D0 warning 'commonvpxUtil'] [VpxUtil] Failed to lookup address info for localhost.2020dev.local
2012-12-19T20:21:41.108Z [70440B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2012-12-19T20:26:34.390Z [70840B90 error 'SoapAdapter.HTTPService'] Failed to read request; stream: TCP(<null>), error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2012-12-19T20:37:49.497Z [70903B90 verbose 'Default'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCP(local=127.0.0.1:8307, peer=127.0.0.1:55139)
2012-12-19T20:37:49.497Z [70440B90 error 'HttpSvc.HTTPService'] Failed to read request; stream: TCP(<null>), error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2012-12-12T13:12:45.515Z cpu0:18317)WARNING: Tcpip_Vmk: 717: Failed to set default gateway (51): Network unreachable
2012-12-19T20:04:21.595Z cpu8:16854)WARNING: Team.etherswitch: TeamES_Activate:309:Failed to initialize beaconing on portset 'pps': Not implemented.
2012-12-19T20:04:33.346Z cpu11:17041)WARNING: ScsiScan: 1276: Failed to add path vmhba2:C0:T19:L0 : Not found
2012-12-19T20:04:33.873Z cpu0:16854)WARNING: Uplink: 3075: releasing cap 0x0!
Many thanks for that..
I had to update the /etc/hosts as it wasn't resolving!!!
Seems I have move a bit forward
now the vxpa.log shows:
2012-12-20T14:04:52.087Z [FFDE26D0 info 'Handle checker'] Set system limit to 1024
2012-12-20T14:05:37.609Z [58EC7B90 verbose 'HttpConnectionPool-000000'] [RemoveConnection] Connection removed; cnx: SSL(no stream); pooled: 0
2012-12-20T14:05:37.610Z [58D28B90 verbose 'HttpConnectionPool-000001'] [RemoveConnection] Connection removed; cnx: SSL(no stream); pooled: 0
2012-12-20T14:12:34.781Z [58E21B90 verbose 'SoapAdapter.HTTPService.HttpConnection'] User agent is 'VMware-client/5.0.0'
2012-12-20T14:12:34.880Z [58D6AB90 error 'SoapAdapter'] Unsupported version URI "urn:vpxa3/3.0"
2012-12-20T14:12:34.881Z [58D6AB90 error 'SoapAdapter'] Unsupported version URI "urn:vpxa3/3.0"
--> while parsing SOAP body
--> at line 9, column 0
--> while parsing SOAP envelope
--> at line 2, column 0
--> while parsing HTTP request before method was determined
--> at line 1, column 0
2012-12-20T14:12:34.881Z [58D6AB90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Complete (processed 650 bytes)
2012-12-20T14:12:34.899Z [FFE88B90 verbose 'SoapAdapter.HTTPService.HttpConnection'] User agent is 'VMware-client/5.0.0'
2012-12-20T14:12:35.000Z [58D07B90 error 'SoapAdapter'] Unsupported version URI "urn:vpxa3/2.0"
2012-12-20T14:12:35.001Z [58D07B90 error 'SoapAdapter'] Unsupported version URI "urn:vpxa3/2.0"
(Unsopprted version ?????
VMWare vclient/5.0.0??? I have installed Vcenter5.1 and ESX5.1.. is that correct?)
2012-12-20T14:15:44.722Z [650C2B90 verbose 'Proxy Req 00460'] The client closed the stream, not unexpectedly.
2012-12-20T14:15:50.998Z [FFFA2430 verbose 'Proxy Req 00461'] New proxy client SSL(TCP(local=192.168.60.22:443, peer=192.168.60.91:13411))
2012-12-20T14:15:50.999Z [65040B90 verbose 'Proxy Req 00461'] Connected to localhost:8089
2012-12-20T14:15:51.117Z [652C2B90 verbose 'Proxy Req 00462'] New proxy client SSL(TCP(local=192.168.60.22:443, peer=192.168.60.91:13412))
2012-12-20T14:15:51.118Z [65040B90 verbose 'Proxy Req 00462'] Connected to localhost:8089
2012-12-20T14:15:51.237Z [65281B90 verbose 'Proxy Req 00463'] New proxy client SSL(TCP(local=192.168.60.22:443, peer=192.168.60.91:13413))
2012-12-20T14:15:51.238Z [652C2B90 verbose 'Proxy Req 00463'] Connected to localhost:8089
2012-12-20T14:15:51.720Z [FFFA2430 verbose 'Proxy Req 00461'] The client closed the stream, not unexpectedly.
2012-12-20T14:15:51.727Z [650C2B90 verbose 'Proxy Req 00462'] The client closed the stream, not unexpectedly.
2012-12-20T14:15:51.852Z [65081B90 verbose 'Proxy Req 00463'] The client closed the stream, not unexpectedly.
2012-12-20T14:15:58.117Z [652C2B90 verbose 'Proxy Req 00464'] New proxy client SSL(TCP(local=192.168.60.22:443, peer=192.168.60.91:13416))
2012-12-20T14:15:58.119Z [65081B90 verbose 'Proxy Req 00464'] Connected to localhost:8089
2012-12-20T14:15:58.237Z [FFFA2430 verbose 'Proxy Req 00465'] New proxy client SSL(TCP(local=192.168.60.22:443, peer=192.168.60.91:13417))
2012-12-20T14:15:58.238Z [65081B90 verbose 'Proxy Req 00465'] Connected to localhost:8089
and the hostd.log just showing re-tries all the time ... until it times out
the issue has been sorted.
I can say it was something simple, but it wasn't as it was a compatibility issue between ESXi and Vcenter Server. It took many days until I realised about it.
I re-installed in the end ESXi 5.0.0 (and NOT 5.1.0 as it was originally done) while Vcenter Server was also 5.0.0. The communication between both parties happened without a single issue.
When I got the license for ESXi 5.X.X, I downloaded the latest versions never crossing my mind compatibity issues should happened as anything was about the same Release level.
That was a wrong assumption.
This is a common mistake by many software companies which misslead customers. So priority before installations, compatibility tables (if any!!!), or just test until it works then after you can start migrating and installing VMs (one of the got corrupted during the tests and was fun to put everything right again)
Many thanks to all those who came around with suggestions.
Thanks for sharing your findings. As you have discovered, the vCenter version must be greater than or equal to the version of ESXi host it manages. It would be nice if VMware would implement a compatibility cache whereby a higher version ESXi host could negotiate a compatibility level (similar to EVC for CPUs). This could be called EVV (Enhanced vCenter Versioning) for example.
With the rate of version changes nowadays, vCenter versioning becomes one of the biggest liabilities since this is the toughest component in the vSphere environment to upgrade (i.e. requires downtime, SQL backups, not to mention risky new features/requirements such as SSO, etc..). For organizations that have multiple vCenters, this becomes a risk to your reaction time to bugs/fixes at the ESXi level.
Anyway, nice job identifying and making it through this issue.
I came across the same issue while adding 5.1 host to vcenter 5.0.when i was struggling to decode the error i cam across this blog.
so let me try adding 5.1 or high host version.
and i have attached the error.
however i am going to verify by upgrading vcenter to 5.1 then i will try adding host 5.1 then i will reply the results.i have attached the error.