VMware Cloud Community
bleuze
Enthusiast
Enthusiast
Jump to solution

SOLVED: Could not connect to one or more vCenter Server systems:

I am running the vsphere 6.5 vcenter appliance. It is the only vcenter server we have. I can connect to it through the webclient, I can log in as administrator@vsphere.local. But it does not show me any inventory (it just shows a spinning wheel).

There is an error message:

Could not connect to one or more vCenter Server systems:

https://vcenterservername@domain.local:443/sdk

This vcenter appliance used to work. But it has had this problem now for a few months. If need be I can just delete the appliance and re-install. There is no special configuration in there - only connection to 2 hosts. I can still use web client on each host to manage each host one at a time, I just can't manage through the vcenter appliance. I really only use the appliance to apply updates to the hosts.

Message was edited by: Bill Leuze (removed false hyperlinks)

1 Solution

Accepted Solutions
Raj1988
Enthusiast
Enthusiast
Jump to solution

bleuz

vpxd.log

2018-07-09T22:30:25.448Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=vpxCommon opID=HostSync-host-28-37dc5435] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...

2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

2018-07-09T22:30:25.466Z panic vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435]

-->

--> Panic: Unrecoverable VmRootError. Panic!

Thats the crash .

You need to see the postgres.log during the same time frame for the exact duplicate key ...

Example:-

postgresql.log

UTC 583a9c81.5814 127750 VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"
UTC 583a9c81.5814 127750 VCDB vc DETAIL: Key (id, device_key)=(361, 3002) already exists.

Take a snapshot . Connect to VCSA DB ..

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Delete the duplicate ID using the below command. Replace with the ID values you get from postgres.log

DELETE FROM vpx_vm_virtual_device where id='xxx' and device_key='xxxx';

View solution in original post

18 Replies
diegodco31
Leadership
Leadership
Jump to solution

Hi

Check it link:

VMware Knowledge Base

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
bleuze
Enthusiast
Enthusiast
Jump to solution

I already looked at that KB. I tried solution for vsphere 6.x >> For Appliance.

I assume when it says "Connect to Platform Services Controller using SSH" it means connect to the vcenter appliance as that is the only vcenter controller of any sort we have in our infrastructure. But the only username\password I have that I can use to connect to it is administrator@vphere.local and that logs me in as user "nobody". at the very fist step in the solition it says I have no permissions. Is there a default root password for this appliance that I can use for root access?

Reply
0 Kudos
bleuze
Enthusiast
Enthusiast
Jump to solution

I just found KB How to reset the lost or forgotten root password in vCenter Server Appliance 6.5 (2147144)

so resetting password now... then I'll try the solution KB again

Reply
0 Kudos
diegodco31
Leadership
Leadership
Jump to solution

During the installation the root password was requested.

pastedImage_0.png

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
diegodco31
Leadership
Leadership
Jump to solution

Let us know if you fix your problem with root password.

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
bleuze
Enthusiast
Enthusiast
Jump to solution

Root password reset successful.

As for the solution in that KB, when I run the command to output the "list of the services registered within the Platform Services Controller". I do not get duplicates like shown in the example. So I don't think I have a duplicate service endpint to unregister. Probably I need a different solution that this KB unless you see somethi9ng here. Following is the output of the "list of the services registered within the Platform Services Controller":


        Name: AboutInfo.vpx.name
        Description: AboutInfo.vpx.name
        Service Product: com.vmware.cis
        Service Type: vcenterserver
        Service ID: 4e1085a8-2bdf-45b8-9d36-75ca6d624205
        Site ID: ccv-vcenter-srv
        Node ID: 0f80f893-3f4f-4002-a9b5-36de3bb9d9da
        Owner ID: vpxd-a065d362-06f6-4fb6-9504-cab10e2d5a90@vsphere.local
        Version: 6.5
        Endpoints:
                Type: com.vmware.cis.data.provider
                Protocol: vmomi
                URL: https://ccv-vCenter-srv.colonial.local:443/sdk
                SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
                Endpoint Attributes:
                        cis.common.ep.localurl: http://localhost:8085/sdk
                Type: com.vmware.cdc.provider
                Protocol: vmomi
                URL: https://ccv-vCenter-srv.colonial.local:443/sdk
                SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
                Endpoint Attributes:
                        cis.common.ep.localurl: http://localhost:8085/sdk
                Type: com.vmware.vim
                Protocol: vmomi
                URL: https://ccv-vCenter-srv.colonial.local:443/sdk
                SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
                Endpoint Attributes:
                        cis.common.ep.localurl: http://localhost:8085/sdk
                Type: com.vmware.cis.common.resourcebundle
                Protocol: http
                URL: https://ccv-vCenter-srv.colonial.local:443/catalog/catalog.zip
                SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
                Endpoint Attributes:
                        com.vmware.cis.common.resourcebundle.basename: cis.vc.action:cis.vc.alarm:cis.vc.auth:cis.vc.cluster:cis.vc.default:cis.vc.enum:cis.vc.evc:cis.vc.event:cis.vc.eventaux:cis.vc.fault:cis.vc.gos:cis.vc.host:cis.vc.locmsg:cis.vc.option:cis.vc.perf:cis.vc.question:cis.vc.stask:cis.vc.task:cis.vc.vm:cis.vc.profile:cis.vc.hostdiag.enum:cis.vc.hostdiag.eventaux:cis.vc.hostdiag.event:cis.vc.hostdiag.locmsg:cis.vc.VirtualCenter.enum:cis.vc.VirtualCenter.eventaux:cis.vc.VirtualCenter.event:cis.vc.VirtualCenter.extension:cis.vc.VirtualCenter.locmsg
                Type: com.vmware.vim.extension
                Protocol: vmomi
                URL: https://ccv-vCenter-srv.colonial.local:443/sdkTunnel
                SSL trust: MIID2TCCAsGgAwIBAgIJANy7Zpl+SRBfMA0GCSqGSIb3DQEBCwUAMIGlMQswCQYDVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExJzAlBgNVBAoMHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE4MDIyMzAwMDE0MVoXDTI4MDIxODAwMDE0MVowNjEnMCUGA1UEAwweY2N2LXZDZW50ZXItc3J2LmNvbG9uaWFsLmxvY2FsMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAg0+lrBBWzzvAFIob9CP0jYysX3nP09DPbrMDu5+Gz615ZlJgELebqGRup4D2U8C8ub6pmISEjhxKsBt4x0+NM/qXOXgtuLAtGpzU62CRkXp8jOPCcVUuUL0SvSKbI9lmZQZ0g3jACQudQQm4/DHKhytS+S1FG0sjBl9ui3ICydfEjMPk0OVVJmvNNwZe0ceR8iTUIwBh6BPW+8XqPuJDN1Ur8AgdaK3pzrfzaRkLmYpWQlv5LvoiJdrO7Xj96V/O2X/A7mroxi2xaaNzOv2xj3OObNUdqeu2djSp3Hpi45V20P4fUHKLJ+bxmHgAhxyGEHO1jM9D3IKtYGTIkFqMCAwEAAaN6MHgwCwYDVR0PBAQDAgOoMCkGA1UdEQQiMCCCHmNjdi12Q2VudGVyLXNydi5jb2xvbmlhbC5sb2NhbDAdBgNVHQ4EFgQUPbbqIUez7oCCCLYCFF7A0/MY9zkwHwYDVR0jBBgwFoAUNi+Dh0DifI0UAuEK/sf385t/AacwDQYJKoZIhvcNAQELBQADggEBAAxQTE9wzwdmhVwM6DLrAem/D0xubApm8hYq4d3vMw0gmqRsJd3LEyhx87Wjwvof5+RKFipsjI5o0IKm0IkfeJQcPS0sRf5Il2Sd5w3YAR//MUSRnnOJ6CEECAYOcjLG06oMFYxmqurygqSBKeF8ooNB5pt+8fSeLLXw0+CEZKpcQvt4olWmacaxkxe5UpLgAJu4SFacWF9p9bPDQ+xEOqVylv9/SJ8zCd7eDekRkcFBnFUp2mh+i3DPB/YrGtQ9yP6feIWhWALCaEIOOZz8t37lQxQPwoyDiEBPU06/xgoCwxGusuOtshmAdXier4WxMjyLkEVMA2DEaKMaQ/RYAlA=
                Endpoint Attributes:
                        cis.common.ep.localurl: http://localhost:8085/sdkTunnel
        Attributes:
                com.vmware.cis.cm.GroupInternalId: com.vmware.vim.vcenter
                com.vmware.vim.vcenter.instanceName: ccv-vCenter-srv.colonial.local
                com.vmware.cis.cm.ControlScript: service-control-default-vmon
                com.vmware.cis.cm.HostId: a065d362-06f6-4fb6-9504-cab10e2d5a90

Reply
0 Kudos
diegodco31
Leadership
Leadership
Jump to solution

What is the build of vcsa 6.5?

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
Raj1988
Enthusiast
Enthusiast
Jump to solution

That looks good. Only one registration .

Match instance.cfg from /etc/vmware-vpx/ with Service ID: 4e1085a8-2bdf-45b8-9d36-75ca6d624205 . This should be same.

Also run service-control --status --all and check if vmware-vpxd si up and running . This error can also come if vpxd service is down. If vpxd is dowsn ; please share vpxd.log .

Reply
0 Kudos
bleuze
Enthusiast
Enthusiast
Jump to solution

Diego:

the vcsa build is 6.5.0.5100

Ajay:

root@ccv-vCenter-srv [ /etc/vmware-vpx ]# cat instance.cfg

applicationDN=dc\=virtualcenter,dc\=vmware,dc\=int

instanceUuid=4e1085a8-2bdf-45b8-9d36-75ca6d624205

ldapPort=389

ldapInstanceName=VMwareVCMSDS

ldapStoragePath=/etc/vmware-vpx/

looks same as Service ID: 4e1085a8-2bdf-45b8-9d36-75ca6d624205 in the previously dumped psc_services.txt

root@ccv-vCenter-srv [ /etc/vmware-vpx ]# service-control --status --all

Running:

applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-client vsphere-ui

Stopped:

vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health

Hmm, vmware-vpxd is stopped. How do I start it?

Reply
0 Kudos
bleuze
Enthusiast
Enthusiast
Jump to solution

Hmm, I started VMware-vpxd but it still shows as stopped:

root@ccv-vCenter-srv [ /etc/rc.d ]# init.d/vmware-vpxd

root@ccv-vCenter-srv [ /etc/rc.d ]# Usage: vmware-vpxd {start|stop|status|restart|clean-embedded-db}

init.d/vmware-vpxd start

<30>Jul  9 22:29:53 vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

Executing pre-startup scripts...

Traceback (most recent call last):

  File "/usr/lib/vmware-vpx/py/vccfg.py", line 493, in <module>

    sys.exit(VcvaConfigCmd().run())

  File "/usr/lib/vmware-vpx/py/vccfg.py", line 455, in run

    opMethod()

  File "/usr/lib/vmware-vpx/py/vccfg.py", line 396, in readCmd

    result = self.readValues(args)

  File "/usr/lib/vmware-vpx/py/vccfg.py", line 156, in readValues

    cfgViewsList = self.getKeyCfgViewList(configKey, readOnly=True)

  File "/usr/lib/vmware-vpx/py/vccfg.py", line 87, in getKeyCfgViewList

    self.cfgFileStores[filename] = cfgViewClass.createFileStore(filename)

  File "/usr/lib/vmware-vpx/py/shellvarcfg.py", line 53, in createFileStore

    return PlainTextFile(filename)

  File "/usr/lib/vmware-vpx/py/shellvarcfg.py", line 21, in __init__

    raise CfgException("Cannot open file: " + self.filename)

cfgexception.CfgException: Cannot open file: /usr/lib/vmware-invsvc/lib/server/config/dataservice.properties

<30>Jul  9 22:29:53 vmware-vpxd: Starting vpxd by administrative request.

success

<30>Jul  9 22:29:54 vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

Waiting for vpxd to initialize: .success

<30>Jul  9 22:30:05 vmware-vpxd: vpxd has initialized.

root@ccv-vCenter-srv [ /etc/rc.d ]# service-control --status --all

Running:

applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-client vsphere-ui

Stopped:

vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health

Reply
0 Kudos
diegodco31
Leadership
Leadership
Jump to solution

Please, share the file /var/log/vmware/vpx/vpxd.log.

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
bleuze
Enthusiast
Enthusiast
Jump to solution

sorry about the big paste above. lets see if I can do a file attach...

Yes, log attached. So the above long pasted message is now deleted

Reply
0 Kudos
diegodco31
Leadership
Leadership
Jump to solution

Check the following:

VCSA 6.5 Web GUI Error

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
Raj1988
Enthusiast
Enthusiast
Jump to solution

bleuz

vpxd.log

2018-07-09T22:30:25.448Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=vpxCommon opID=HostSync-host-28-37dc5435] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...

2018-07-09T22:30:25.461Z error vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

2018-07-09T22:30:25.466Z panic vpxd[7F4FACAC3700] [Originator@6876 sub=Default opID=HostSync-host-28-37dc5435]

-->

--> Panic: Unrecoverable VmRootError. Panic!

Thats the crash .

You need to see the postgres.log during the same time frame for the exact duplicate key ...

Example:-

postgresql.log

UTC 583a9c81.5814 127750 VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"
UTC 583a9c81.5814 127750 VCDB vc DETAIL: Key (id, device_key)=(361, 3002) already exists.

Take a snapshot . Connect to VCSA DB ..

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Delete the duplicate ID using the below command. Replace with the ID values you get from postgres.log

DELETE FROM vpx_vm_virtual_device where id='xxx' and device_key='xxxx';

bleuze
Enthusiast
Enthusiast
Jump to solution

Thanks Ajay

I will look into this tomorrow. I am heading out the door now

Reply
0 Kudos
diegodco31
Leadership
Leadership
Jump to solution

Hi

Let us know if you fix your problem.

Remember to take the snapshot vCenter.

Step to step:

Could not connect to one or more vCenter Server systems | DesertPenguin.org | Andrew J. Brehm

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
bleuze
Enthusiast
Enthusiast
Jump to solution

SOLVED:

OK, In postgresql-09.log I found:

2018-07-09 22:30:25.447 UTC 5b43e1fa.9ad9 1964284 VCDB vc ERROR:  duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"

2018-07-09 22:30:25.447 UTC 5b43e1fa.9ad9 1964284 VCDB vc DETAIL:  Key (id, device_key)=(37, 3002) already exists.

Actually I found 6 more instances of this message all with the same id, device key pair. But this is the one at the exact same timestamp as the Ajay identified in the vpxd.log

So, I made a snapshot, then:

root@ccv-vCenter-srv [ /var/log/vmware/vpostgres ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
psql.bin (9.4.9 (VMware Postgres 9.4.9.1-4536683 release))
Type "help" for help.

VCDB=# DELETE FROM vpx_vm_virtual_device where id='37' and device_key='3002';
DELETE 1
VCDB=# \q

problem still exists, so, rebooting the vcenter appliance...

et viola, that fixed it. All inventory now displays in the webgui. Thanks so much Ajay!!!

SUMoses
Contributor
Contributor
Jump to solution

Just restart all the servers in the following sequence ,  1,  passive 2, witness 3,vcsa ,     then see that all your services are started , then log in again ,

Reply
0 Kudos