Morning All,
Hoping someone can introduce some sanity into my life.. We had a power cut last week and it has caused havoc with my Windows 2008 vCenter Server machine. The symptoms (there are many):
o- can't start vSphere Client, returns the error:
"vSphere Client could not connect to "<server name>". An unknown error occurred. (The server could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable.))
o- this led me to look at the state of the various VMWare services on the machine. A number have not started, despite being set for automatic start up:
- VMWare Content Library Service
- VMWare ESX Agent Manager
- VMWare Performance Charts
- VMWare Syslog Collector
- VMWare vCenter workflow manager
- VMWare VirtualCenter Server
- VMWare vService Manager
o- if I try to start VMWare VirtualCenter Server (chose it to try because it sounds important!), it fails. Service control manager returns the error "Error 1067: The process terminated unexpectedly". I get an event log entry with event ID 7031 telling that it failed to start.
o- There's an event in the VMWare vCenter log from last week saying that "vpxd status changed from grey to red". I think that was before the power cut.
o- In my 'vpxs-xxx.log' (which I think are the logs from the VirtualCenter Server service) I have a number of bad looking entries:
2017-08-16T05:21:41.800-05:00 info vpxd[04464] [Originator@6876 sub=Libs] FILE: FileCreateDirectoryRetry: Non-retriable error encountered (C:\ProgramData\VMware): Cannot create a file when that file already exists (183)
2017-08-16T05:21:53.116-05:00 error vpxd[07452] [Originator@6876 sub=vmomi.soapStub[17]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2017-08-16T05:22:18.125-05:00 warning vpxd[08020] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x000000000977ff88, h:2788, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8088'>>, e: system:10061(No connection could be made because the target machine actively refused it)
2017-08-16T05:22:19.144-05:00 error vpxd[08020] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:0000000009cb84a0, TCP:localhost:8088>; cnx: (null), error: class Vmacore::SystemException(No connection could be made because the target machine actively refused it)
I'm really struggling to make sense of this. I thought maybe my vpxd.cfg file was corrupt, so I followed some advice I saw somewhere to rename the file and restart the service, a new one was supposed to be created. It wasn't, and the service failed to start with a "failed to find specified file" error.
There seems to be so much wrong here, I'm struggling to get to the bottom of this 😞 Any suggestions would be very welcome!
Thanks,
Al.
Hi Al,
Firstly, can you stop and start all services running service-control script?
It will give you more details which service fails as a first (vpxd has many dependencies and problem might be cause by other service..)
Open cmd and run:
cd C:\Program Files\VMware\vCenter Server\bin
service-control --stop --all
service-control --start --all
More details about restarting services: How to stop, start, or restart vCenter Server 6.x services (2109881) | VMware KB
Once you know which service fails as a first one, then you might need to go to C:\ProgramData\VMware\vCenterServer\logs and check the logs for this particular service.
Regards,
Beata
Beata,
Thanks very much for your suggestion. Unfortunately, I am not in a position to try it - I decided that 2.5 days wrestling with this thing was quite enough and managed to restore my vCenter Server from a snapshot using the powershell CLI. Annoying, because I would have liked to have got to the bottom of the problem, but other work was piling up 😞
Al.
Please check database connectivity and then try to start services.
Hi Al,
Totally understandable.
If you will encounter similar problem in the future, then you can try above to check what service is causing that vCenter Server is failing to start correctly and hopefully logs will tell us the reason.
Regards,
Beata
Yes, that's a very useful piece of information. I was looking at all the services in the service control manager and wondering what order they were all supposed to start! Although I thought that the SCM knew the dependencies of the services..
Al.