vSphere 6.5 custom cert upgrade from SHA1 to SHA256

[vSphereRider]

Hi All

I installed custom certs in our vSphere 6.5 environment. There are 2 sites in enhanced link mode vCenter and PSC in each both with a custom cert from our local CA of SHA1 and connected SRM.

I need to upgrade these certs to SHA256. I cant find much info on the internet about it, does anyone have any experience with this and steps taken?

Thank you so much for any info

[vSphereRider]

So nothing special to note here, just go through the same process as SHA1.

Was just looking for some advice where someone said, mate is this really worth it you got the CA SHA1's on their in a distributed HA environment quit while you ahead. I mean why in the hell are you replacing all the certs anyway. Just so some security team can convince the money man they should be there?

I have replaced a few vSphere environments with MS CA certs and I have come to the conclusion I am done with it.

Half the time it works out half the time it does not. I had an HA environment with external VCSA's and PSC's ver 6.0. first site updated fine all working. Updated second site same process, vCenter turns to a ball of flames. VMware support spent 4 or 5 days trying to fix the issue before they told me I had to rebuild the whole environment along with all the other vSphere products connected into my vCenter's.

With regards to this case I built 6.5 environment for a customer and added the SHA1. Was smooth sailing Angels smiling on me, this time go replace the certs with new SH256 same process right.

No, of course not, luck had ran out. Machines SSL fails on the first PSC, after digging around I had to remove the storage plugins and delete them from all vCenters MOD database in the environment. Next go to vCenter, Machines SSL fails, of course I had made the mistake of having hope I was near the end. This is a known issue with no resolution take said work around to manually change folder permissions in the cli to older versions and TRY again. I will pick this up on Monday and post the rest of the issues I'm sure will come up if I don't throw myself off the roof.

PS: If you have a vCenter with embedded PSC just snapshot that thing and give it ago, you got a 50/50 and you can always just roll back. If you have a highly available distributed setup ask yourself why, then take a moment and ask yourself that question again, with a smile on your face you get up from your desk and go treat yourself to a pint, why because you love yourself and self harm is not your cup of tea! There are easier ways to get the browser cert to be trusted. Using WINSCP I check all config files and settings before pressing the Y(Button of death) I would suggest you get well versed in the VMCA mechanisms before you have a tinkle and hit that button.

Friends don't let friends change vSphere certs.

[daphnissov]

Some feedback based on my having installed custom SSL certificates for many a customer over the years.

1. Always snapshot. Snapshot whatever you're replacing certs on. If you don't do this you're just stupid. Seriously.
2. Read the documentation completely on certificates. If you don't understand certificates well and you don't understand how vSphere uses/needs them, do not proceed. You cannot hope for success if you don't have basic information understood.
3. Before agreeing to replace certificates, speak to your security people to understand what *type* of certificates they can generate. Collect all details. SHA-1 should have told you to stop right there because it shouldn't be used any more. You need to collect intel on certs before you even begin the process.
4. Generate the CSR from within vCenter and double and triple check the attributes are correct. And once you've checked that, check it one more time.
5. Finally, did I say snapshot?

[vSphereRider]

Get off your high horse mate, I used SHA1 3 years ago as it is what they were dishing out at the time.

If you do it all the time across the versions you know its riddle with issues. There is a whole world off google to back me up.

If you read my post again I mention "snapshot" but don't be fooled snapshot is not always going to get you out a certificate pickle.

From VMware

"Can I use snapshots against my PSC 6.0? How about image-based backups?

You can snapshot a single Platform Services Controller so long as it does not exist in a multi-site or highly available configuration within a vSphere domain. This is due to the use of Update Sequence Number (USN) for replication, and when restoring a PSC via snapshot or image-based backup, the sibling nodes are out of sync"

[daphnissov]

SHA-1 even 3 years ago wasn't considered very secure, and almost 2 years ago now it was so bad that Google, Mozilla, and others decided to warn about and block by default presented SHA-1 certificates. Doesn't mean it was good to use even back then. In fact, they were recommended to not be used at that point in various sources (here, here, here), even as far back as 2005. But regardless, the cert replacement process would (and still does) work with SHA-1, so that's not the reason itself for failures.

If you read my post again I mention "snapshot" but don't be fooled snapshot is not always going to get you out a certificate pickle.

From VMware

"Can I use snapshots against my PSC 6.0? How about image-based backups?

You can snapshot a single Platform Services Controller so long as it does not exist in a multi-site or highly available configuration within a vSphere domain. This is due to the use of Update Sequence Number (USN) for replication, and when restoring a PSC via snapshot or image-based backup, the sibling nodes are out of sync"

If you know how to take them in a replicated PSC environment it will. USNs don't change unless there are changes to the PSC inventory like SSO config, users, licenses, tags, etc. This can easily be tracked with vmafd-cli. Using PowerCLI or API you can snapshot all PSCs that are replica partners and ensure they remain consistent. If a roll-back is needed, yes that's not a fun process because you'd have to roll them all back.

[vSphereRider]

Mate, what the hell is your problem. All you saying has no relevance to my post, complaining about SHA1, we know you got out of the wrong side of the bed.

You say yes snapshot, you are AGREEING with me saying its filled with problems as snapshots are so crucial to you, you revert cause it went Pete Tong the snapshot didn't make it work.

In this case my cert install failed because of this issue with VCSA https://kb.vmware.com/s/article/2150895  this is not in any of the documentation process for certs its not something I could have prevented for a smooth upgrade its a fault. And then this issue which made the upgrade process fail at another stage VMware Knowledge Base  ...............................

[daphnissov]

Ok, so what's your point? You ran into some known issues. Tough. That happens with software. Sounds like you worked with support and got it taken care of, though. Also, my point about snapshots was they should be used in the certificate replacement process. My second point was they can be used to safely revert linked PSCs if that replacement process goes haywire. So what's your problem with any of these points?

[vSphereRider]

Haha you such a waste man, came to troll a post thinking you biggy big blocks, now you admit you agreeing with me.

Well I am glad you think I am at the level of VMware support because I googled the errors from the log file and found KB's relating to them. You should try it you wont need support as much.

Anyway I started this post to ask any one if they ran into issues. I ran in to 2 and have provided the KB's so someone migh be preparing to update and if they are on VMCA 6.5 know they will run into a problem with update manager servie, or if they have 3rd party plugins will know they going to run into the error. they wont have to troll through logs like me.

This is a place for information not trolls go to twitter

[daphnissov]

It's good you posted this information so others can see what problems you ran into. Yes, I have done this replacement from SHA-1 just as you're doing. Yes, I've done it with vCSA 6.5. And, yes, I've done it with third-party plug-ins. Never had any issues. That said, it doesn't mean someone else won't (clearly you did). There was no disagreement from the start, only providing more information. And, BTW, the only reason you found those KBs is because someone at VMware took the time to write them because a customer before you (probably multiple) had the same issues for which they *did* open support requests. Good for you that you did some research on your own problem and found those articles.

[Mr_G_Grant]

... So basically don't bother trying to upgrade the certs from sha1 to sha256 because VMware have made the process overly complex? ...... Cool ..... self signed certs it is