I am running the vsphere 6.5 vcenter appliance. It is the only vcenter server we have. I can connect to it through the webclient, I can log in as administrator@vsphere.local. But it does not show me any inventory (it just shows a spinning wheel).
There is an error message:
Could not connect to one or more vCenter Server systems:
https://vcenterservername@domain.local:443/sdk
This vcenter appliance used to work. But it has had this problem now for a few months. If need be I can just delete the appliance and re-install. There is no special configuration in there - only connection to 2 hosts. I can still use web client on each host to manage each host one at a time, I just can't manage through the vcenter appliance. I really only use the appliance to apply updates to the hosts.
Message was edited by: Bill Leuze (removed false hyperlinks)
bleuz
vpxd.log
2018-07-09T22:30:25.448Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=vpxCommon opID=HostSync-host-28-37dc5435] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
2018-07-09T22:30:25.466Z panic vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435]
-->
--> Panic: Unrecoverable VmRootError. Panic!
Thats the crash .
You need to see the postgres.log during the same time frame for the exact duplicate key ...
Example:-
postgresql.log
UTC 583a9c81.5814 127750 VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"
UTC 583a9c81.5814 127750 VCDB vc DETAIL: Key (id, device_key)=(361, 3002) already exists.
Take a snapshot . Connect to VCSA DB ..
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
Delete the duplicate ID using the below command. Replace with the ID values you get from postgres.log
DELETE FROM vpx_vm_virtual_device where id='xxx' and device_key='xxxx';
I already looked at that KB. I tried solution for vsphere 6.x >> For Appliance.
I assume when it says "Connect to Platform Services Controller using SSH" it means connect to the vcenter appliance as that is the only vcenter controller of any sort we have in our infrastructure. But the only username\password I have that I can use to connect to it is administrator@vphere.local and that logs me in as user "nobody". at the very fist step in the solition it says I have no permissions. Is there a default root password for this appliance that I can use for root access?
I just found KB How to reset the lost or forgotten root password in vCenter Server Appliance 6.5 (2147144)
so resetting password now... then I'll try the solution KB again
During the installation the root password was requested.
Let us know if you fix your problem with root password.
Root password reset successful.
As for the solution in that KB, when I run the command to output the "list of the services registered within the Platform Services Controller". I do not get duplicates like shown in the example. So I don't think I have a duplicate service endpint to unregister. Probably I need a different solution that this KB unless you see somethi9ng here. Following is the output of the "list of the services registered within the Platform Services Controller":
Name: AboutInfo.vpx.name
Description: AboutInfo.vpx.name
Service Product: com.vmware.cis
Service Type: vcenterserver
Service ID: 4e1085a8-2bdf-45b8-9d36-75ca6d624205
Site ID: ccv-vcenter-srv
Node ID: 0f80f893-3f4f-4002-a9b5-36de3bb9d9da
Owner ID: vpxd-a065d362-06f6-4fb6-9504-cab10e2d5a90@vsphere.local
Version: 6.5
Endpoints:
Type: com.vmware.cis.data.provider
Protocol: vmomi
URL: https://ccv-vCenter-srv.colonial.local:443/sdk
SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
Endpoint Attributes:
cis.common.ep.localurl: http://localhost:8085/sdk
Type: com.vmware.cdc.provider
Protocol: vmomi
URL: https://ccv-vCenter-srv.colonial.local:443/sdk
SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
Endpoint Attributes:
cis.common.ep.localurl: http://localhost:8085/sdk
Type: com.vmware.vim
Protocol: vmomi
URL: https://ccv-vCenter-srv.colonial.local:443/sdk
SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
Endpoint Attributes:
cis.common.ep.localurl: http://localhost:8085/sdk
Type: com.vmware.cis.common.resourcebundle
Protocol: http
URL: https://ccv-vCenter-srv.colonial.local:443/catalog/catalog.zip
SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
Endpoint Attributes:
com.vmware.cis.common.resourcebundle.basename: cis.vc.action:cis.vc.alarm:cis.vc.auth:cis.vc.cluster:cis.vc.default:cis.vc.enum:cis.vc.evc:cis.vc.event:cis.vc.eventaux:cis.vc.fault:cis.vc.gos:cis.vc.host:cis.vc.locmsg:cis.vc.option:cis.vc.perf:cis.vc.question:cis.vc.stask:cis.vc.task:cis.vc.vm:cis.vc.profile:cis.vc.hostdiag.enum:cis.vc.hostdiag.eventaux:cis.vc.hostdiag.event:cis.vc.hostdiag.locmsg:cis.vc.VirtualCenter.enum:cis.vc.VirtualCenter.eventaux:cis.vc.VirtualCenter.event:cis.vc.VirtualCenter.extension:cis.vc.VirtualCenter.locmsg
Type: com.vmware.vim.extension
Protocol: vmomi
URL: https://ccv-vCenter-srv.colonial.local:443/sdkTunnel
SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
Endpoint Attributes:
cis.common.ep.localurl: http://localhost:8085/sdkTunnel
Attributes:
com.vmware.cis.cm.GroupInternalId: com.vmware.vim.vcenter
com.vmware.vim.vcenter.instanceName: ccv-vCenter-srv.colonial.local
com.vmware.cis.cm.ControlScript: service-control-default-vmon
com.vmware.cis.cm.HostId: a065d362-06f6-4fb6-9504-cab10e2d5a90
What is the build of vcsa 6.5?
That looks good. Only one registration .
Match instance.cfg from /etc/vmware-vpx/ with Service ID: 4e1085a8-2bdf-45b8-9d36-75ca6d624205 . This should be same.
Also run service-control --status --all and check if vmware-vpxd si up and running . This error can also come if vpxd service is down. If vpxd is dowsn ; please share vpxd.log .
Diego:
the vcsa build is 6.5.0.5100
Ajay:
root@ccv-vCenter-srv [ /etc/vmware-vpx ]# cat instance.cfg
applicationDN=dc\=virtualcenter,dc\=vmware,dc\=int
instanceUuid=4e1085a8-2bdf-45b8-9d36-75ca6d624205
ldapPort=389
ldapInstanceName=VMwareVCMSDS
ldapStoragePath=/etc/vmware-vpx/
looks same as Service ID: 4e1085a8-2bdf-45b8-9d36-75ca6d624205 in the previously dumped psc_services.txt
root@ccv-vCenter-srv [ /etc/vmware-vpx ]# service-control --status --all
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-client vsphere-ui
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health
Hmm, vmware-vpxd is stopped. How do I start it?
Hmm, I started VMware-vpxd but it still shows as stopped:
root@ccv-vCenter-srv [ /etc/rc.d ]# init.d/vmware-vpxd
root@ccv-vCenter-srv [ /etc/rc.d ]# Usage: vmware-vpxd {start|stop|status|restart|clean-embedded-db}
init.d/vmware-vpxd start
<30>Jul 9 22:29:53 vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd
Executing pre-startup scripts...
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/py/vccfg.py", line 493, in <module>
sys.exit(VcvaConfigCmd().run())
File "/usr/lib/vmware-vpx/py/vccfg.py", line 455, in run
opMethod()
File "/usr/lib/vmware-vpx/py/vccfg.py", line 396, in readCmd
result = self.readValues(args)
File "/usr/lib/vmware-vpx/py/vccfg.py", line 156, in readValues
cfgViewsList = self.getKeyCfgViewList(configKey, readOnly=True)
File "/usr/lib/vmware-vpx/py/vccfg.py", line 87, in getKeyCfgViewList
self.cfgFileStores[filename] = cfgViewClass.createFileStore(filename)
File "/usr/lib/vmware-vpx/py/shellvarcfg.py", line 53, in createFileStore
return PlainTextFile(filename)
File "/usr/lib/vmware-vpx/py/shellvarcfg.py", line 21, in __init__
raise CfgException("Cannot open file: " + self.filename)
cfgexception.CfgException: Cannot open file: /usr/lib/vmware-invsvc/lib/server/config/dataservice.properties
<30>Jul 9 22:29:53 vmware-vpxd: Starting vpxd by administrative request.
success
<30>Jul 9 22:29:54 vmware-vpxd: Waiting for vpxd to start listening for requests on 8089
Waiting for vpxd to initialize: .success
<30>Jul 9 22:30:05 vmware-vpxd: vpxd has initialized.
root@ccv-vCenter-srv [ /etc/rc.d ]# service-control --status --all
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-client vsphere-ui
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health
Please, share the file /var/log/vmware/vpx/vpxd.log.
Check the following:
bleuz
vpxd.log
2018-07-09T22:30:25.448Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=vpxCommon opID=HostSync-host-28-37dc5435] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
2018-07-09T22:30:25.466Z panic vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435]
-->
--> Panic: Unrecoverable VmRootError. Panic!
Thats the crash .
You need to see the postgres.log during the same time frame for the exact duplicate key ...
Example:-
postgresql.log
UTC 583a9c81.5814 127750 VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"
UTC 583a9c81.5814 127750 VCDB vc DETAIL: Key (id, device_key)=(361, 3002) already exists.
Take a snapshot . Connect to VCSA DB ..
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
Delete the duplicate ID using the below command. Replace with the ID values you get from postgres.log
DELETE FROM vpx_vm_virtual_device where id='xxx' and device_key='xxxx';
Thanks Ajay
I will look into this tomorrow. I am heading out the door now
Hi
Let us know if you fix your problem.
Remember to take the snapshot vCenter.
Step to step:
Could not connect to one or more vCenter Server systems | DesertPenguin.org | Andrew J. Brehm
SOLVED:
OK, In postgresql-09.log I found:
2018-07-09 22:30:25.447 UTC 5b43e1fa.9ad9 1964284 VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"
2018-07-09 22:30:25.447 UTC 5b43e1fa.9ad9 1964284 VCDB vc DETAIL: Key (id, device_key)=(37, 3002) already exists.
Actually I found 6 more instances of this message all with the same id, device key pair. But this is the one at the exact same timestamp as the Ajay identified in the vpxd.log
So, I made a snapshot, then:
root@ccv-vCenter-srv [ /var/log/vmware/vpostgres ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
psql.bin (9.4.9 (VMware Postgres 9.4.9.1-4536683 release))
Type "help" for help.
VCDB=# DELETE FROM vpx_vm_virtual_device where id='37' and device_key='3002';
DELETE 1
VCDB=# \q
problem still exists, so, rebooting the vcenter appliance...
et viola, that fixed it. All inventory now displays in the webgui. Thanks so much Ajay!!!
Just restart all the servers in the following sequence , 1, passive 2, witness 3,vcsa , then see that all your services are started , then log in again ,