» NSX 4.1 – Alarms indicating Corfu certificates are expired or about to expire #

Recently, after we have updated 3.2 to 4.1 NSX-T appliances, as part of the VCF 4.5.2 to 5.1.1 change, many Corfu certificates seemed to be expired. We were not aware of Corfu certificates. However, a quick Google query let us to the KB94898 , which showed us exactly what to do: run a script and everything will be resolved. Sounded too good to be true, but it worked out.

After logging into the NSX-T GUI, you can find the symptom in NSX-T > System > Certificates
The screenshot is filtered on expired certificates

Background

NSX Managers have many certificates for internal services.
In version NSX 3.2.1, Cluster Boot Manager (CBM) service certificates were incorrectly given a validity period of 825 days instead of 100 years. This was corrected to 100 years in NSX 3.2.3. However any environment originally installed on NSX 3.2.1 will have the internal CBM Corfu certs expire after 825 regardless of upgrade to the fixed version or not.

On NSX-T 3.2.x internal server certificates could expire and no alarm would trigger. There was no functional impact.
Starting from NSX 4.1.0.2, NSX alarms now monitor validity of internal certificates and will trigger for expired or soon to expire certificates. Note on NSX 4.1.x, there is currently no functional impact when an internal certificate expires however alarms will continue to trigger. This is expected behaviour.

Workaround

Scripted Resolution

VMware have developed a script that will replace all internal self signed certs with new certs of validity period 100 years.
The script is compatible with NSX version 4.1.0 and above. The script does not replace API and cluster certificates.
An NSX backup must be taken before running the script. Also ensure the passphrase is known.

This is a python version 3 script which should be run from a client machine which has paramiko and cryptography python packages installed.
Depending on the system this may be installed with a command such as #sudo pip3 install cryptography
The script cannot be run directly on the NSX Manager as it does not have the required python module. It is not supported to install it on the NSX Manager.

1) Download the attached script replace_certs.py through KB94898
2) To execute the script run the following command and follow the prompts

#python3 replace_certs.py

3) You will need to input the NSX Manager cluster IP and admin credentials at the relevant prompts.

4) In some environments it may be necessary to increase the timeout value used by the script to allow the script to complete successfully.
long_wait_time defaults to a value of 60 but can be increased to 180 (or higher) and then re-run the script.

If the script does not work in your environment, there is the option to follow the manual procedure.
Alternatively open an SR to report the failure scenario

The steps involve

Use a system that is capable of running python code
Make sure pip is upgraded, and cryptography and paramiko are installed as well
- python -m pip install –upgrade pip
- sudo pip3 install cryptography
- sudo pip3 install paramiko
  See screenshots below
- Upgrade pip and install paramiko
- Run python3 replace_certs.py, as can be downloaded through KB94898
- Please enter the details for:
  - Virtual IP address (VIP) of the NSX-T Cluster
    - For this, navigate to NSX-T > System > Appliances
  - The cluster’s admin password (twice)
- The output of the script should be something like
  - You will find that the script runs through each NSX-T Manager node, one by one, and updates the certificates sequentially.
- Note that in the meantime the script has been updated from KB94898, and output will be different, with more information.
After the script has finished, you will be pleased to find the expired certificates are missing.
Great, job well done!