vSAN Encryption - Failed to add more trusted certificate to cluster KMSClusterName. A cluster can configure at most 16 trusted certificates

vHaridas — Wed, 21 Feb 2018 17:36:43 GMT

Hi,

I have enabled vSAN Encryption using HyTrust Key Control for Key Management server. HyTrust KMS had small issue which was forcing me to refresh KMS certificate in vCenter and reestablish trust with KMS each time I reboot KMS server. I worked with HyTrust and they have fixed this issue.

During testing in LAB, multiple times I restarted KMS, refreshed KMS certificate to established trust. Now if I try to refresh certificate am getting below error in vCenter web client.

I checked the vpxd.log on vCenter which says... Failed to add more trusted certificate to cluster vlabKMS01. A cluster can configure at most 16 trusted certificates.

Does anyone knows how i can increase this 16 certificate limits?

where does cluster or vCenter store KMS server certificate?

How can I delete unused certificates of KMS server?

2018-02-21T12:39:41.340Z info vpxd[7F7A8E9D3700] [Originator@6876 sub=vpxLro opID=dam-auto-generated: KmipTrustCertificateDialogMediator:dr-1217:01-2f] [VpxLRO] -- BEGIN lro-1730902 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52cd417f-4036-bf4b-e92e-f47207d6980d(52211441-f4ca-278c-9c58-014cc5c88454)

2018-02-21T12:39:41.340Z info vpxd[7F7A8E9D3700] [Originator@6876 sub=vpxLro opID=dam-auto-generated: KmipTrustCertificateDialogMediator:dr-1217:01-2f] [VpxLRO] -- FINISH lro-1730902

2018-02-21T12:39:41.345Z info vpxd[7F7A8FD7A700] [Originator@6876 sub=vpxLro opID=dam-auto-generated: KmipTrustCertificateDialogMediator:dr-1217:VCenterKmipPropertyProvider:203173:430839-1609690-ngc:70055790-fb] [VpxLRO] -- BEGIN lro-1730903 -- CryptoManager -- vim.encryption.CryptoManagerKmip.retrieveKmipServerCert -- 52cd417f-4036-bf4b-e92e-f47207d6980d(52211441-f4ca-278c-9c58-014cc5c88454)

2018-02-21T12:39:43.271Z error vpxd[7F7A8FD7A700] [Originator@6876 sub=CryptoManagerKmipWrapper opID=dam-auto-generated: KmipTrustCertificateDialogMediator:dr-1217:VCenterKmipPropertyProvider:203173:430839-1609690-ngc:70055790-fb] Failed to connect to key server, QLC_ERR_NEED_AUTH

2018-02-21T12:39:45.833Z info vpxd[7F7A8FD7A700] [Originator@6876 sub=vpxLro opID=dam-auto-generated: KmipTrustCertificateDialogMediator:dr-1217:VCenterKmipPropertyProvider:203173:430839-1609690-ngc:70055790-fb] [VpxLRO] -- FINISH lro-1730903

2018-02-21T12:39:47.656Z info vpxd[7F7AB4ACB700] [Originator@6876 sub=vpxLro opID=KmipServerActionResolver-apply-1609697-ngc:70055791-6d] [VpxLRO] -- BEGIN lro-1730904 -- CryptoManager -- vim.encryption.CryptoManagerKmip.uploadKmipServerCert -- 52cd417f-4036-bf4b-e92e-f47207d6980d(52211441-f4ca-278c-9c58-014cc5c88454)

2018-02-21T12:39:47.668Z error vpxd[7F7AB4ACB700] [Originator@6876 sub=CryptoManager opID=KmipServerActionResolver-apply-1609697-ngc:70055791-6d] Failed to add more trusted certificate to cluster vlabKMS01. A cluster can configure at most 16 trusted certificates.

2018-02-21T12:39:47.669Z info vpxd[7F7AB4ACB700] [Originator@6876 sub=vpxLro opID=KmipServerActionResolver-apply-1609697-ngc:70055791-6d] [VpxLRO] -- FINISH lro-1730904

2018-02-21T12:39:47.669Z info vpxd[7F7AB4ACB700] [Originator@6876 sub=Default opID=KmipServerActionResolver-apply-1609697-ngc:70055791-6d] [VpxLRO] -- ERROR lro-1730904 -- CryptoManager -- vim.encryption.CryptoManagerKmip.uploadKmipServerCert: vim.fault.DatabaseError:

--> Result:

--> (vim.fault.DatabaseError) {

--> faultCause = (vmodl.MethodFault) null,

--> faultMessage = <unset>

--> msg = ""

--> }

--> Args:

-->

--> Arg cluster:

--> (vim.encryption.KeyProviderId) {

--> id = "vlabKMS01"

--> }

--> Arg certificate:

--> "-----BEGIN CERTIFICATE-----

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

--> --> hL+qymRUCAzsiwwr/orCEXoZkgjO0XqBc2SGgdxA3CiXbO5An4N5PQ==

--> -----END CERTIFICATE-----

--> "

2018-02-21T12:39:47.925Z info vpxd[7F7AB4846700] [Originator@6876 sub=vpxLro opID=combined(dam-auto-generated: ObjectPropertyFilter:dr-1231,dam-auto-generated: RecentItemsListener:dr-1219,dam-auto-generated: ObjectPropertyFilter:dr-1229,dam-auto-generated: ObjectPropertyFilter:dr-1225,dam-auto-generated: KmipServersListViewMediator:dr-1] [VpxLRO] -- BEGIN lro-1730905 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52cd417f-4036-bf4b-e92e-f47207d6980d(52211441-f4ca-278c-9c58-014cc5c88454)

2018-02-21T12:39:47.926Z info vpxd[7F7AB4846700] [Originator@6876 sub=vpxLro opID=combined(dam-auto-generated: ObjectPropertyFilter:dr-1231,dam-auto-generated: RecentItemsListener:dr-1219,dam-auto-generated: ObjectPropertyFilter:dr-1229,dam-auto-generated: ObjectPropertyFilter:dr-1225,dam-auto-generated: KmipServersListViewMediator:dr-1] [VpxLRO] -- FINISH lro-1730905

2018-02-21T12:39:47.936Z info vpxd[7F7AB525A700] [Originator@6876 sub=vpxLro opID=dam-auto-generated: ObjectPropertyFilter:dr-1231:VchaPropertyProvider:203173:430847-1609700-ngc:70055793-e2] [VpxLRO] -- BEGIN lro-1730907 -- FailoverClusterConfigurator -- vim.vcha.FailoverClusterConfigurator.getConfig -- 52cd417f-4036-bf4b-e92e-f47207d6980d(52211441-f4ca-278c-9c58-014cc5c88454)

2018-02-21T12:39:47.959Z info vpxd[7F7AB5056700] [Originator@6876 sub=vpxLro opID=combined(dam-auto-generated: GenerationNumbersMonitor:dr-1249,dam-auto-generated: ObjectPropertyFilter:dr-1247):01-40] [VpxLRO] -- BEGIN lro-1730909 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52cd417f-4036-bf4b-e92e-f47207d6980d(52211441-f4ca-278c-9c58-014cc5c88454)

2018-02-21T12:39:47.974Z info vpxd[7F7A8F366700] [Originator@6876 sub=vpxLro opID=dam-auto-generated: ObjectPropertyFilter:dr-1231:VchaPropertyProvider:203173:430847-1609702-ngc:70055793-ff] [VpxLRO] -- BEGIN lro-1730910 -- FailoverClusterManager -- vim.vcha.FailoverClusterManager.getClusterHealth -- 52cd417f-4036-bf4b-e92e-f47207d6980d(52211441-f4ca-278c-9c58-014cc5c88454)

2018-02-21T12:39:47.974Z error vpxd[7F7A8F366700] [Originator@6876 sub=SoapAdapter opID=dam-auto-generated: ObjectPropertyFilter:dr-1231:VchaPropertyProvider:203173:430847-1609702-ngc:70055793-ff] Method vim.vcha.FailoverClusterManager.getClusterHealth threw undeclared fault of type vim.fault.InvalidState

2018-02-21T12:39:47.975Z info vpxd[7F7A8F366700] [Originator@6876 sub=Default opID=dam-auto-generated: ObjectPropertyFilter:dr-1231:VchaPropertyProvider:203173:430847-1609702-ngc:70055793-ff] [VpxLRO] -- ERROR lro-1730910 -- FailoverClusterManager -- vim.vcha.FailoverClusterManager.getClusterHealth: vim.fault.InvalidState:

--> Result:

--> (vim.fault.InvalidState) {

--> faultCause = (vmodl.MethodFault) null,

--> faultMessage = (vmodl.LocalizableMessage) [

--> (vmodl.LocalizableMessage) {

--> key = "com.vmware.vim.vcha.error.clusterNotConfigured",

--> arg = <unset>,

--> message = <unset>

--> }

--> ]

--> msg = ""

--> }

--> Args:

-->

Thanks,

Haridas

Re: vSAN Encryption - Failed to add more trusted certificate to cluster KMSClusterName. A cluster can configure at most 16 trusted certificates

GreatWhiteTec — Wed, 21 Feb 2018 18:13:20 GMT

Hi vHaridas,

Is this test only or prod? If test only, you can reset your KMS server to delete the old certs, which still count against the total number of certs. You can remove all keys by resetting the KMIP server. Go to Settings -> KMIP Server Settings, then click the "Reset KMIP Server" button. This will remove all keys on HyTrust, so DO NOT DO THIS ON PRODUCTION SERVER!. After the reset, Change state to "Enabled", and click the "Apply" button.

In VC, you should be able to see the certs in the UI. Administration>System Configuration>nodes. Select VC>Manage>Certificate Authority.

AFAIK, VMCA uses OpenSSL, so I'm assuming it gets the limits from it.

Re: vSAN Encryption - Failed to add more trusted certificate to cluster KMSClusterName. A cluster ca

AdamKithcart — Tue, 17 Oct 2023 00:21:23 GMT

A bit of an old post, but what if this is a production machine where you can't lose access to the keys?

topic vSAN Encryption - Failed to add more trusted certificate to cluster KMSClusterName. A cluster can configure at most 16 trusted certificates in VMware vSAN Discussions

vSAN Encryption - Failed to add more trusted certificate to cluster KMSClusterName. A cluster can configure at most 16 trusted certificates

Re: vSAN Encryption - Failed to add more trusted certificate to cluster KMSClusterName. A cluster can configure at most 16 trusted certificates

Re: vSAN Encryption - Failed to add more trusted certificate to cluster KMSClusterName. A cluster ca