Greetings friends, I told you a few weeks ago about the mess Let’s Encrypt has made with the expiring CA, and I also told you how to put a new SSL with ZeroSSL. Unfortunately, in the process, I think I broke something, as I was messing around with the certificates trying to remove the expiring CA, etc.
The problem that all this generated for me in my lab was:
- Not being able to see anything in the Update Manager
- Not being able to see, edit, or do anything in the Storage Policies
- Skyline Center wouldn’t load, wouldn’t display anything
- Skyline Center wouldn’t load, wouldn’t display anything
I had really messed it up. But I’m not the only one, because it seems that when we change the SSL certificate, if something goes wrong, or many rollbacks are made, VCSA is unstable because some services are left with old certificates, the error (one of them), you will see is as follows:
How to solve this complex problem with SSL certificates
First of all, please make a snapshot and a backup of your VCSA (veeam+native backup). If you already have everything, it’s time to turn to lsdoctor.
A brief look at lsdoctor
Lookup Service Doctor (lsdoctor) is a tool used to troubleshoot problems with data stored in the PSC database as well as local data in a vCenter (regardless of whether the PSC is external or embedded). The tool can be used to detect and correct problems that could cause failures in topology changes (convergence, repointing, etc.), upgrades, or failures that occur as a result of maintenance (e.g., incorrect application of new SSL certificates).
Considerations.
Currently, lsdoctor supports vCenter 6.5 and higher (both Windows and VCSA). When new versions of vCenter are released, lsdoctor must be updated asynchronously. This means that lsdoctor support for the latest version of vCenter may be updated sometime after a new build is released.
Download link and more information.
You can find this fantastic tool in the official KB, from there you can also download it:
Taking the leap to fixing our SSL certificate problems
Ok, I’ve already downloaded the package from the official KB, and put it inside my VCSA, in my case I’ve used SCP to move the copy from an ESXi to vCenter, but there are many ways.
The command we will launch is trustfix, this option fixes SSL trust mismatch issues in the lookup service. The lookup service records may have an SSL trust value that does not match the MACHINE_SSL_CERT on port 443 on the node. This can be caused by a failure during certificate replacement, among other failures.
You can look at the rest of the commands by clicking on each one:
Here we go,
python lsdoctor-master/lsdoctor.py --trustfix
WARNING: This script makes permanent changes. Before running, please take *OFFLINE* snapshots of all VC's and PSC's
of all VC's and PSC's at the SAME TIME. Failure to do so can result in PSC or VC inconsistencies.
Logs can be found here: /var/log/vmware/lsdoctor
2021-11-02T14:59:55 INFO main: You are checking for and fixing SSL trust mismatches in the local SSO site. NOTE: Please run this script one PSC or VC per SSO site.
We will now be asked if we have made a backup, or snapshot of our VCSA, and PSC, we will say yes:
Have you taken offline (PSCs and VCs powered down at the same time) snapshots of all nodes in the SSO domain or supported backups?[y/n]y
It will now ask us for our administrator user password, we will enter it and continue:
Provide password for [email protected]: 2021-11-02T15:00:11 INFO __init__: Retrieved services from SSO site: default-site 2021-11-02T15:00:11 INFO findAndFix: Checking services for trust mismatches... 2021-11-02T15:00:11 INFO findAndFix: Attempting to reregister 27024872-0790-4804-8baf-52ea6cedcf8b for vcsa.jorgedelacruz.es 2021-11-02T15:00:12 INFO findAndFix: Attempting to reregister 9ae1be99-aabd-47a5-bd9a-f97f74eaf78f for vcsa.jorgedelacruz.es 2021-11-02T15:00:12 INFO findAndFix: Attempting to reregister 090b3bc5-7b47-4523-9bcf-ece9dd3bd5f2 for vcsa.jorgedelacruz.es 2021-11-02T15:00:12 INFO findAndFix: Attempting to reregister c1816628-73b9-41d3-bd69-bcfc22ce7132 for vcsa.jorgedelacruz.es 2021-11-02T15:00:13 INFO findAndFix: Attempting to reregister a29abfca-809f-41b5-8d92-703d80644274 for vcsa.jorgedelacruz.es 2021-11-02T15:00:13 INFO findAndFix: Attempting to reregister 707048f1-bd68-4e9e-8553-7cecd69a7e0e for vcsa.jorgedelacruz.es 2021-11-02T15:00:13 INFO findAndFix: Attempting to reregister fcd320b3-f305-446a-8381-d539782f182a for vcsa.jorgedelacruz.es 2021-11-02T15:00:13 INFO findAndFix: Attempting to reregister 6a272105-0c01-4588-806e-4f767ca5d33b for vcsa.jorgedelacruz.es 2021-11-02T15:00:13 INFO findAndFix: Attempting to reregister f1168519-37ca-4b1f-8b1a-0dc50232b67c for vcsa.jorgedelacruz.es 2021-11-02T15:00:14 INFO findAndFix: Attempting to reregister 3e24f979-d726-4dc2-a346-c7d7263661e7 for vcsa.jorgedelacruz.es 2021-11-02T15:00:14 INFO findAndFix: Attempting to reregister a8280ba6-61dd-4048-83f2-b569f2f04237 for vcsa.jorgedelacruz.es 2021-11-02T15:00:14 INFO findAndFix: Attempting to reregister 836b502b-a07b-4afd-b55c-06d4960c0851 for vcsa.jorgedelacruz.es 2021-11-02T15:00:14 INFO findAndFix: Attempting to reregister bb837576-b08c-4fa9-b198-283040e7b0e2 for vcsa.jorgedelacruz.es 2021-11-02T15:00:15 INFO findAndFix: Attempting to reregister 1dd6f0bf-bfe3-42b6-a287-c55a7d53fde5 for vcsa.jorgedelacruz.es 2021-11-02T15:00:15 INFO findAndFix: Attempting to reregister 816c6c0b-f6cf-4c21-a75c-3a0d0fa77c55 for vcsa.jorgedelacruz.es 2021-11-02T15:00:15 INFO findAndFix: Attempting to reregister default-site:c528e353-4680-4885-9e07-6d1d5b5b632d for vcsa.jorgedelacruz.es 2021-11-02T15:00:15 INFO findAndFix: Attempting to reregister 88a4a758-c9b1-4e37-bc93-f45469dcbd74 for vcsa.jorgedelacruz.es 2021-11-02T15:00:16 INFO findAndFix: Attempting to reregister 6a8f8c0e-f55e-4bcb-93e8-f125d4b984eb for vcsa.jorgedelacruz.es 2021-11-02T15:00:16 INFO findAndFix: Attempting to reregister 078dd39a-bea4-4f2f-9b9e-9a4210cfec92 for vcsa.jorgedelacruz.es 2021-11-02T15:00:16 INFO findAndFix: Attempting to reregister default-site:612774a5-5093-4eaa-892c-d5735d3af0fe for vcsa.jorgedelacruz.es 2021-11-02T15:00:16 INFO findAndFix: Attempting to reregister 4b1d6bac-c4d1-4c31-b018-879ed775f2eb for vcsa.jorgedelacruz.es 2021-11-02T15:00:16 INFO findAndFix: Attempting to reregister 6be89e7e-732e-4322-a632-9ebdf195a4a9 for vcsa.jorgedelacruz.es 2021-11-02T15:00:16 INFO findAndFix: Attempting to reregister 9a434046-4a51-47f7-a681-6d664ed69446 for vcsa.jorgedelacruz.es 2021-11-02T15:00:17 INFO findAndFix: Attempting to reregister 816c6c0b-f6cf-4c21-a75c-3a0d0fa77c55_kv for vcsa.jorgedelacruz.es 2021-11-02T15:00:17 INFO findAndFix: Attempting to reregister 816c6c0b-f6cf-4c21-a75c-3a0d0fa77c55_authz for vcsa.jorgedelacruz.es 2021-11-02T15:00:17 INFO findAndFix: Attempting to reregister 245bbfd7-dfcf-4dc1-88f9-229248f5ee95 for vcsa.jorgedelacruz.es 2021-11-02T15:00:17 INFO findAndFix: Attempting to reregister 8e97e667-7b0d-4f0c-8f4b-b2cc8daf289b for vcsa.jorgedelacruz.es 2021-11-02T15:00:17 INFO findAndFix: Attempting to reregister 5a544e36-016e-4c19-bfcb-232c58deefa7 for vcsa.jorgedelacruz.es 2021-11-02T15:00:17 INFO findAndFix: Attempting to reregister e5f7b046-3ca4-400a-8ede-fac7a264b43a for vcsa.jorgedelacruz.es 2021-11-02T15:00:18 INFO findAndFix: Attempting to reregister default-site:55c555a8-5e19-4ff6-8d92-e4154152e2bd for vcsa.jorgedelacruz.es 2021-11-02T15:00:18 INFO findAndFix: Attempting to reregister 8dc912b2-d198-4d4b-a19d-a4b2555912b1 for vcsa.jorgedelacruz.es 2021-11-02T15:00:18 INFO findAndFix: Attempting to reregister 28103393-3a79-4098-a842-a7ea842b9ac5 for vcsa.jorgedelacruz.es 2021-11-02T15:00:18 INFO findAndFix: Attempting to reregister 9b08f7a8-3d50-47a4-b484-354b71c41b6e for vcsa.jorgedelacruz.es 2021-11-02T15:00:18 INFO findAndFix: Attempting to reregister 4bfcf56e-4c7a-4664-903e-3b6727def2f4 for vcsa.jorgedelacruz.es 2021-11-02T15:00:18 INFO findAndFix: Attempting to reregister cbf91919-1b8d-480f-8289-cb47c2060281 for vcsa.jorgedelacruz.es 2021-11-02T15:00:19 INFO findAndFix: Attempting to reregister 677fba44-aa0d-4b37-ad52-0c86f5ef8932 for vcsa.jorgedelacruz.es 2021-11-02T15:00:19 INFO findAndFix: Attempting to reregister 33d045f6-76ba-41a6-9078-1263e8a093ea for vcsa.jorgedelacruz.es 2021-11-02T15:00:19 INFO findAndFix: We found 37 mismatch(s) and fixed them :) 2021-11-02T15:00:19 INFO main: Please restart services on all PSC's and VC's when you're done.
Not bad, 37 errors in my certificates, normal that I did not start, or did not work well the services that needed VCSA, let’s restart now the services:
root@vcsa [ /tmp/lsdoctor ]# service-control --stop --all
We shall see how the services stop one by one and start again:
Operation not cancellable. Please wait for it to finish... Performing stop operation on service observability... Successfully stopped service observability Performing stop operation on service vmware-pod... Successfully stopped service vmware-pod Performing stop operation on service vmware-vdtc... Successfully stopped service vmware-vdtc Performing stop operation on profile: ALL... Successfully stopped service vmware-vmon Successfully stopped profile: ALL. Performing stop operation on service vmcad... Successfully stopped service vmcad Performing stop operation on service vmdird... Successfully stopped service vmdird Performing stop operation on service vmafdd... Successfully stopped service vmafdd Performing stop operation on service lwsmd... Successfully stopped service lwsmd root@vcsa [ /tmp/lsdoctor ]# service-control --start --all Operation not cancellable. Please wait for it to finish... Performing start operation on service lwsmd... Successfully started service lwsmd Performing start operation on service vmafdd... Successfully started service vmafdd Performing start operation on service vmdird... Successfully started service vmdird Performing start operation on service vmcad... Successfully started service vmcad Performing start operation on profile: ALL... Successfully started service vmware-vmon Successfully started profile: ALL. Performing start operation on service observability... Successfully started service observability Performing start operation on service vmware-vdtc... Successfully started service vmware-vdtc Performing start operation on service vmware-pod... Successfully started service vmware-pod
We return to our VCSA, we load some of the views that previously had problems, such as Storage Policies for example:
Nothing more friends, I hope you find it as useful as I do.
thanks bro, you saved my freelance project!
regards,
Haikal
Big Thankyou!
Yes, thank you – works also flawless with Vcenter 8!