VMware Cloud Community
flyingrobots_69
Contributor
Contributor

vpxd crash after loading VMCA certificate

Hello,

We have an internal certificate authority (using OpenSSL) and have created a CA intermediate certificate using VCenter's certificate-manager to create the csr.

The certificate is successfully installed using certificate manager (it accepts it without complaining), however, VCenter is not able to start as vpxd is getting a SIGTERM (signal 15) during startup.

I've captured a limited number of vpxd log entries around the signal 15.  It appears to be attempting to call a .setCertficate method and this method seems to fail (it is at the end).

I'm working with VCenter 6.7.0.54000

I feel like I may be missing something in the certificate extensions or something to do with certificate creation.  Here is the certificate in question. 

 

 

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4110 (0x100e)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Texas, O = "MyCompany, Inc", OU = Development, CN = mydomain.net, emailAddress = frank@mydomain.net
        Validity
            Not Before: Feb  5 22:45:39 2023 GMT
            Not After : Feb 15 22:45:39 2024 GMT
        Subject: C = US, ST = Texas, L = Weatherford, O = "MyCompany, Inc", OU = Development, CN = CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:c2:a5:18:82:1f:ba:a9:39:7b:6a:1b:07:90:ff:
                    bb:ac:a3:75:25:02:23:ed:41:30:01:9d:a2:12:94:
                    c7:b3:83:d1:be:1d:9d:d1:e5:87:4e:7a:61:70:16:
                    c3:3f:d1:d1:1f:8f:93:59:a0:01:1f:e1:56:68:ab:
                    78:42:40:03:fe:cb:4f:d5:fb:73:a8:42:9f:fb:74:
                    82:0f:2e:9b:be:83:67:2c:9e:0b:55:31:ee:32:0c:
                    19:ff:16:c4:3b:7a:d0:c3:94:66:a1:29:02:bb:13:
                    58:29:04:27:a9:72:50:7b:a0:a0:6c:8c:a6:79:42:
                    62:ca:db:be:4e:d4:a0:9c:be:89:68:29:bd:87:0e:
                    04:65:7a:1b:36:ce:d4:17:bc:97:c2:1b:ce:d2:18:
                    b2:b3:b2:9a:7a:f1:dd:90:fc:82:4b:ba:30:be:69:
                    4c:16:90:85:86:1b:b7:a6:ba:92:4b:88:af:ec:f2:
                    76:0f:6d:d3:0e:8f:93:83:1e:03:52:03:33:94:17:
                    03:7b:88:b0:9d:ae:5a:5a:c5:d8:ea:b7:72:86:4b:
                    14:f2:8b:3b:4f:8a:59:d1:8f:82:ab:8b:8a:40:28:
                    11:ea:34:90:2b:c7:c8:f7:d1:61:d4:a8:ae:6d:a7:
                    e8:ed:58:a3:d1:52:d4:8b:22:1b:51:ce:05:95:92:
                    1c:e9
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                6E:81:0D:26:2C:A6:D9:A5:11:B8:01:7C:EE:D3:5A:AB:85:C4:2D:C1
            X509v3 Authority Key Identifier: 
                keyid:F3:9C:8D:DB:28:A1:8E:CB:2D:30:58:7F:DF:9F:FB:98:64:5A:1B:A6

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         1a:d7:4d:ca:a6:46:cc:91:11:da:10:6c:93:c3:db:0c:11:7d:
         6c:6e:94:d6:56:87:68:b8:4b:6b:4d:01:e0:a7:49:1c:e7:64:
         49:d1:4e:65:6d:13:1e:74:72:c1:c4:6b:59:5d:8f:dc:35:33:
         1b:bb:93:5f:6b:a4:ea:9a:05:9b:95:49:cf:39:e4:f4:c1:33:
         d1:6e:13:a6:6f:7e:c9:d6:cb:db:1f:48:5a:05:a7:e0:4e:85:
         87:7e:05:29:fe:49:58:58:e7:f5:a1:14:35:9c:88:6c:06:00:
         ae:64:ae:24:75:95:17:9f:0c:77:bd:0d:a7:0a:63:e0:4d:13:
         91:47:9a:2c:e7:b8:54:5c:91:72:b2:a4:95:c6:e9:48:4f:db:
         02:f2:c5:a7:2d:68:f2:dc:88:52:4e:f4:71:9d:5c:06:10:50:
         a3:ed:a5:9f:06:07:89:f0:bb:69:c4:e2:2d:23:d2:9f:34:bc:
         af:36:b9:28:62:1c:2a:a3:f7:ad:cd:36:c2:15:54:a7:87:d6:
         58:6b:d2:93:67:20:f5:d5:35:06:bf:c3:89:e8:1b:06:4e:d2:
         1e:99:ce:5f:8e:b6:fa:54:6d:bd:f6:de:01:cc:2e:81:82:da:
         6d:d7:5b:fd:03:92:c6:b1:60:aa:32:3c:c3:c8:43:c0:6c:86:
         7b:03:b7:fe:99:91:b7:fb:25:2a:a3:54:f1:51:dd:46:cf:57:
         3a:c9:46:64:0f:ad:83:08:be:e7:66:51:63:f4:90:f2:ac:65:
         05:c7:d6:72:87:fc:3f:f4:1a:86:5a:68:e9:9c:68:dc:0d:4e:
         e1:57:df:6c:00:41:0b:68:62:95:85:c0:ff:e9:05:81:67:2c:
         8e:a1:88:7b:3b:88:ca:25:bc:2e:b6:8f:49:0c:fa:d9:e0:47:
         d6:8a:e8:8f:85:ed:bb:e6:df:43:15:37:a8:60:6d:dc:43:48:
         ee:42:b4:9a:56:cb:35:98:9c:70:99:24:49:dd:dc:1b:41:70:
         f9:aa:27:bc:6d:fe:9b:2b:08:e2:f7:e2:ac:d3:df:aa:43:8c:
         00:de:a9:32:c4:02:bf:0d:f9:0e:c4:69:5b:0a:a3:38:1e:1a:
         14:ba:8e:6f:cc:37:e9:ac:5b:9e:54:6f:9b:64:1e:17:fb:ed:
         28:d6:60:76:f5:f3:c5:11:f6:2b:11:72:1d:af:36:4c:aa:02:
         e8:31:4f:50:21:ff:86:f1:a4:6f:16:80:ae:1f:3e:11:ec:80:
         95:61:f2:96:3c:b9:e2:21:a2:d7:53:57:0e:8c:f2:d5:56:fa:
         74:23:3c:a9:52:f8:d0:d1:9a:db:d3:99:95:11:02:f1:77:97:
         03:82:6e:54:46:da:f5:48
2023-02-05T23:26:41.517Z info vpxd[31590] [Originator@6876 sub=AuthorizeManager opID=31155621] [Auth]: User VSPHERE.LOCAL\Administrator
2023-02-05T23:26:41.518Z info vpxd[31590] [Originator@6876 sub=vpxLro opID=31155621] [VpxLRO] -- FINISH lro-8488
2023-02-05T23:26:41.533Z info vpxd[31564] [Originator@6876 sub=vpxLro opID=361c547] [VpxLRO] -- BEGIN lro-8490 -- ExtensionManager -- vim.ExtensionManager.setCertificate -- 52d54268-3287-0396-bf36-4f316291e435(52fbdd5e-5e93-4043-9dd0-ba942a6f5623)
2023-02-05T23:26:41.534Z info vpxd[31564] [Originator@6876 sub=vpxLro opID=361c547] [VpxLRO] -- FINISH lro-8490
2023-02-05T23:26:41.534Z info vpxd[31564] [Originator@6876 sub=Default opID=361c547] [VpxLRO] -- ERROR lro-8490 -- ExtensionManager -- vim.ExtensionManager.setCertificate: vim.fault.NotFound:
--> Result:
--> (vim.fault.NotFound) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = <unset>
-->    msg = ""
--> }
--> Args:
--> 
--> Arg extensionKey:
--> "com.vmware.imagebuilder"
--> Arg certificatePem:
--> "-----BEGIN CERTIFICATE-----
--> MIIEGDCCAwCgAwIBAgIJAOFnQF8bcuFzMA0GCSqGSIb3DQEBCwUAMHUxCzAJBgNV
--> BAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLV2VhdGhlcmZvcmQxHTAb
--> BgNVBAoMFEFlcmlhbCBSb2JvdGljcywgSW5jMRQwEgYDVQQLDAtEZXZlbG9wbWVu
--> dDELMAkGA1UEAwwCQ0EwHhcNMjMwMjA1MjMxNjM4WhcNMjQwMjE1MjI0NTM5WjCB
--> iTEXMBUGA1UEAwwOdnB4ZC1leHRlbnNpb24xFzAVBgoJkiaJk/IsZAEZFgd2c3Bo
--> ZXJlMRUwEwYKCZImiZPyLGQBGRYFbG9jYWwxCzAJBgNVBAYTAlVTMTEwLwYDVQQL
--> DChtSUQtMTU1ZTg4ZWMtNjcxOC00ZjYyLWE4NzEtMTM4MTI4OTU0Njc2MIIBIjAN
--> BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtt8IQhXhfuVlb8g8xu8yRvsYkrwn
--> AhJMCyViM74QeQ47K0TioguDV8wm/zDN5kma97AQFKZ/bGNcisHUV14qoX2MUmnr
--> 5ntv9BGztV9te7NacW0GqcnDxEDnS3+Lobetl9eQnSXMeiz+mvZYSJ/opHlVL/q1
--> BKy5a9By4Q9tdPS7pOEvr+K6W97UX1Xje1G7UK1mfhl9EGKcj3o/GJvXwFPEBdtx
--> DVnJvgc+ldsEclpdkT4xiTIiBBuVJu3g4Sx7eHpssu6fZSlvWS9tZIF6n5je/Mng
--> L6DHLc7RWlg4kmBN9btxJ1FNd34lnXAzjbDjLPWRGIioDyS4zFhXwNBjmwIDAQAB
--> o4GVMIGSMAsGA1UdDwQEAwIF4DAdBgNVHQ4EFgQUh4jODNakYb6VkfnDFZxwH6+K
--> xYgwHwYDVR0jBBgwFoAUboENJiym2aURuAF87tNaq4XELcEwQwYIKwYBBQUHAQEE
--> NzA1MDMGCCsGAQUFBzAChidodHRwczovL3ZjZW50ZXIuYXJpbGFicy5uZXQvYWZk
--> L3ZlY3MvY2EwDQYJKoZIhvcNAQELBQADggEBAJ7DQbWj7aE4uTqmB8UbG4rAQLQZ
--> 4JUWdq3F4ztloWEwOaZygcXzVdCg7+G1CeCl+bX/oewqW8h5HB9npBeKjBiVTi+K
--> 1fcCDfuABna8UaQOCNrKWxO3dpk8+jRtVQ8ykuwPVytr7vryww9G45Aa57q5ee0B
--> w7pKvLT7Plxax1EKEF2pdgTWc7MgX4xdIa5+5p91vMD3JDXiwGa2XTnV7gU0wg7p
--> 3S4Ph7kU3BffxrvymTTi4OPKNBobdKlCYZd31Ap0P1ql7mV4d7nWi4nwYOcn9rvb
--> STbDNf8BsSdq2+FAw5/jAnHqG9QrOUkH3jIcjo9/NBTnKKEy3wR3w+HOsDc=
--> -----END CERTIFICATE-----
--> -----BEGIN CERTIFICATE-----
--> MIIE3zCCAsegAwIBAgICEA4wDQYJKoZIhvcNAQELBQAwgYoxCzAJBgNVBAYTAlVT
--> MQ4wDAYDVQQIDAVUZXhhczEdMBsGA1UECgwUQWVyaWFsIFJvYm90aWNzLCBJbmMx
--> FDASBgNVBAsMC0RldmVsb3BtZW50MRQwEgYDVQQDDAthcmlsYWJzLm5ldDEgMB4G
--> CSqGSIb3DQEJARYRa2V2aW5AYXJpbGFicy5uZXQwHhcNMjMwMjA1MjI0NTM5WhcN
--> MjQwMjE1MjI0NTM5WjB1MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxFDAS
--> BgNVBAcMC1dlYXRoZXJmb3JkMR0wGwYDVQQKDBRBZXJpYWwgUm9ib3RpY3MsIElu
--> YzEUMBIGA1UECwwLRGV2ZWxvcG1lbnQxCzAJBgNVBAMMAkNBMIIBIjANBgkqhkiG
--> 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwqUYgh+6qTl7ahsHkP+7rKN1JQIj7UEwAZ2i
--> EpTHs4PRvh2d0eWHTnphcBbDP9HRH4+TWaABH+FWaKt4QkAD/stP1ftzqEKf+3SC
--> Dy6bvoNnLJ4LVTHuMgwZ/xbEO3rQw5RmoSkCuxNYKQQnqXJQe6CgbIymeUJiytu+
--> TtSgnL6JaCm9hw4EZXobNs7UF7yXwhvO0hiys7KaevHdkPyCS7owvmlMFpCFhhu3
--> prqSS4iv7PJ2D23TDo+Tgx4DUgMzlBcDe4iwna5aWsXY6rdyhksU8os7T4pZ0Y+C
--> q4uKQCgR6jSQK8fI99Fh1Kiubafo7Vij0VLUiyIbUc4FlZIc6QIDAQABo2MwYTAd
--> BgNVHQ4EFgQUboENJiym2aURuAF87tNaq4XELcEwHwYDVR0jBBgwFoAU85yN2yih
--> jsstMFh/35/7mGRaG6YwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAcYw
--> DQYJKoZIhvcNAQELBQADggIBABrXTcqmRsyREdoQbJPD2wwRfWxulNZWh2i4S2tN
--> AeCnSRznZEnRTmVtEx50csHEa1ldj9w1Mxu7k19rpOqaBZuVSc855PTBM9FuE6Zv
--> fsnWy9sfSFoFp+BOhYd+BSn+SVhY5/WhFDWciGwGAK5kriR1lRefDHe9DacKY+BN
--> E5FHmiznuFRckXKypJXG6UhP2wLyxactaPLciFJO9HGdXAYQUKPtpZ8GB4nwu2nE
--> 4i0j0p80vK82uShiHCqj963NNsIVVKeH1lhr0pNnIPXVNQa/w4noGwZO0h6Zzl+O
--> tvpUbb323gHMLoGC2m3XW/0DksaxYKoyPMPIQ8BshnsDt/6Zkbf7JSqjVPFR3UbP
--> VzrJRmQPrYMIvudmUWP0kPKsZQXH1nKH/D/0GoZaaOmcaNwNTuFX32wAQQtoYpWF
--> wP/pBYFnLI6hiHs7iMolvC62j0kM+tngR9aK6I+F7bvm30MVN6hgbdxDSO5CtJpW
--> yzWYnHCZJEnd3BtBcPmqJ7xt/psrCOL34qzT36pDjADeqTLEAr8N+Q7EaVsKozge
--> GhS6jm/MN+msW55Ub5tkHhf77SjWYHb188UR9isRch2vNkyqAugxT1Ah/4bxpG8W
--> gK4fPhHsgJVh8pY8ueIhotdTVw6M8tVW+nQjPKlS+NDRmtvTmZURAvF3lwOCblRG
--> 2vVI
--> -----END CERTIFICATE-----
--> -----BEGIN CERTIFICATE-----
--> MIIGDDCCA/SgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgZ4xCzAJBgNVBAYTAlVT
--> MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLV2VhdGhlcmZvcmQxHTAbBgNVBAoM
--> FEFlcmlhbCBSb2JvdGljcywgSW5jMRIwEAYDVQQLDAlEZXZlbG9wZXIxFDASBgNV
--> BAMMC2FyaWxhYnMubmV0MSAwHgYJKoZIhvcNAQkBFhFrZXZpbkBhcmlsYWJzLm5l
--> dDAeFw0yMzAyMDUwNDQ5MzRaFw0zMzAyMDIwNDQ5MzRaMIGKMQswCQYDVQQGEwJV
--> UzEOMAwGA1UECAwFVGV4YXMxHTAbBgNVBAoMFEFlcmlhbCBSb2JvdGljcywgSW5j
--> MRQwEgYDVQQLDAtEZXZlbG9wbWVudDEUMBIGA1UEAwwLYXJpbGFicy5uZXQxIDAe
--> BgkqhkiG9w0BCQEWEWtldmluQGFyaWxhYnMubmV0MIICIjANBgkqhkiG9w0BAQEF
--> AAOCAg8AMIICCgKCAgEAxfbhYycv83WbfkaQOwS3kVyTzanX3bbQvqPgydgi2zag
--> 8Af3IxQsO3raSu+9hQL5IluKE2/8jL7ZDiv+wSe7H+9PvckZFCufkaN+Zp6lP703
--> nX7e6hPnUnwXLbqMPtfaLDWQ9UF1uKkrDvfXWwtDSQytTYR1w2Cmb5IFDnPhx27/
--> 82WTAfwIZZI4tNL9JDJZ3RZUsL/SLwH20mjkBkyKELv6ScQb58FquxONabAOOQWs
--> 2DcMUi6zuiR7ruoWUpi3Xga+SMmPwbP847e6PvX+r43koIWELZfyeMf+MkmpMNOV
--> P5sS09xa9zWr3L6e9KCGZkOXrhw/ktIIhR1FpYv4Dmj3gWFTCPHLVCfJ4AaQHbt7
--> XKkJVIP6o9b0RnpSsxubPWNlXUrIkd+RRgkZCWRcxrJBA+SBBCXwTWDZLa2qGHUp
--> 0iTF8pWv1z19pBq6NTqnVXzn+WOSRoplzzu0a+I01ShPzDjUwW2ERX6WmYv0cpI2
--> uMsWCcyY/hOuGYsUtBP2RTPzomNkdzjcU9KwQwPwwaQEjpvQ2n+f3R71xSmTbW6F
--> /YmIyClazBMz4gyYS9f8Gl1wf9xrWV7wZmbuJro2BnEVr9QbCzq1QyY0mAMICcGo
--> 7Y2uaZHWCpPDsoTrprKy3WfBjKS9ecoTjPg9MuKLcmGS6RgmVJ9IbQzMCowd+u0C
--> AwEAAaNmMGQwHQYDVR0OBBYEFPOcjdsooY7LLTBYf9+f+5hkWhumMB8GA1UdIwQY
--> MBaAFD0BZs0nTOmKikVxfHYYzOo4NcUCMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYD
--> VR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBU8QJfWOMKrsGk71RMOXU2
--> LjsBIPpybLO4KaxIdl0wiDgJ/T2oCEUPAsejyUJKPOrr/bt84cqfvGx/ou0yhBq8
--> u0WsnjQI1zwG6Mcfcrmz1AFO8ewQBZWLoavkthMwEhrZ2T7s4+hFXf6iIyhlLdqY
--> zjC/3Jk+ivm33CfPC5WOsgy37iEjonXXCvZVPBHqHb3gX3DPW4YENiielBdjwEaG
--> Yyasbue64Z4ZsOxmkixhNSDsWCHoG4wGEglbYsyxhDWZ3kMttxGT4F1yjfxi+Fiq
--> B1puP5K56A3Iq6axhC8VsltdyBMSVpr3kvEKKKsMNvpbzb0INLcQ7zxs7AZd/kl6
--> 2tU9sx0+f3wqfijmc3FFH8wCKoZMRfBZuLAtdA8aIjh7vZN3p76xJ+7ea1wrRmdQ
--> oRcPSnFCWK8xB4FyZF9WBG57ct0kwDzqLwuBCEUXLp9RRbC1cz8EdvOmXdr1FeUB
--> wNDgJ6zxKQLalinnuqJunjSYMBP/7AX/rH/XAOODoQnGtiQzJEwLB95HZoUxJYey
--> XhETO84H9DZVy6rsYPPV14NvHDyTlQ9ejlZ8yxyRt1cQQNgBvH3dftBxem1VM6bf
--> zv4PzbFS2jdS+GDhYTtZNoppAxZr+isukehRBW102j/Ij7M8vZb281ZKtM5MIIag
--> HJXeakWpFb+2AATP2TkKrw==
--> -----END CERTIFICATE-----
--> 
--> "
2023-02-05T23:26:41.537Z info vpxd[31605] [Originator@6876 sub=vpxLro opID=7deba0d] [VpxLRO] -- BEGIN lro-8491 -- SessionManager -- vim.SessionManager.logout -- 52d54268-3287-0396-bf36-4f316291e435(52fbdd5e-5e93-4043-9dd0-ba942a6f5623)
2023-02-05T23:26:41.537Z info vpxd[31605] [Originator@6876 sub=vpxLro opID=7deba0d] [VpxLRO] -- FINISH lro-8491
2023-02-05T23:26:43.006Z info vpxd[31565] [Originator@6876 sub=vpxdvpxdSignal] Signal 15 received, exiting
2023-02-05T23:26:43.006Z info vpxd[31565] [Originator@6876 sub=Default] Initiating VMware VirtualCenter shutdown
2023-02-05T23:26:43.006Z info vpxd[31487] [Originator@6876 sub=Default] Shutting down VMware VirtualCenter

 

 

 

Any help is appreciated.

Thank you,

Kevin

Labels (3)
Reply
0 Kudos
4 Replies
maksym007
Expert
Expert

It means that you have uploaded the wrong certificate. 

 

Are you using 3rd party Certificate provider or Microsoft Certificate Authority? 

From where you have requested CSR? via GUI or via SSH? 

Reply
0 Kudos
flyingrobots_69
Contributor
Contributor

I used Certificate Manager (via ssh) to get the CSR.  It creates a CSR and a private key.

I copied the CSR to our internal certificate authority (it uses openssl) and create the signed CA certificate (CA:TRUE, signed by an intermediate key).

I concatenate the new certificate, intermediate cert and the root cert into a single file (in that order) and copy it back to the vcenter server.

I then use certifiate-manager (option 2) to import the certificate, combining it with the private key.

It accepts it (as long as I wait long enough, there is that Start Time error if I do it too soon) and gets to about 85% (that is when vpxd decides to give up).

This is the command used to create the certificate:

 

openssl ca -config intermediate/openssl.cnf -extensions v3_ca -days 375 -notext -md sha256 -in intermediate/csr/vmca_issued_csr.csr -out intermediate/certs/vmca.certs.pem

 

 

v3_ca extensions contain the following:

 

[ v3_ca ]
# Extensions for a CA
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign, nonRepudiation

 

 

I'm pretty sure I'm importing the correct certificate (well, at least the certificate I intend to import, whether it is correct or not is another matter...)

Thanks for taking a look at this.

Kevin

 

Reply
0 Kudos
maksym007
Expert
Expert

Check that article

https://virtualblog.nl/2020/10/26/vmware-vcenter-replace-machine-certificate-with-custom-ca/

here is step-by-step guide. Maybe you missed smth. 

Fingers crossed

Reply
0 Kudos
flyingrobots_69
Contributor
Contributor

Great article!  Thank you.  I've scoured the internet looking for other perspectives.  Found some things that did help, but this by far the most complete I've seen.  Thanks again

The only thing is that I really need to figure out how to replace the CA certificate (the Certificate Authority part of VCenter , Option 2 of Configuration Manager).  This is what is failing.  I've been able to replace the MachineSSL certificate, but not the built in CA Certificate.

Thanks,

Kevin

Reply
0 Kudos