A new blog post: our VCF environments have faced disconnected accounts in vRealize Suite Lifecycle Manager, aka Aria Suite Lifecycle Manager, for quite a while now. Actually, ever since we upgraded vRLI and vROps to a new release:
- vRealize Operations Manager: 8.10.x to 8.14.x
- vRealize LogInsight 8.10.x to 8.14.x
- Aria Suite Lifecycle 8.14 to 8.16
At first, KB90798 is the first place to check out. This KB advises to update to VCF 4.5.2.0. That’s what we already did. Therefore, we contacted VMware GSS.
Apparently, this is due to that Aria components have started introducing new key pairs (ECDSA algorithm, ecdsa-sha2-nistp256), which are not automatically imported in the hosts files on the Aria Suite Lifecycle Manager.
Symptom: when you want to remediate the password within SDDC Manager >> Password Management, the following error appears on screen.
The operationsmanager.log file in /var/log/vmware/vcf/operationsmanager/ contains the following text, confirming you are looking in the right direction:
- Reject HostKey
- Reference Token, as found in SDDC Manager Task Error”
- “Unable to create jsch CLI session: com.jcraft.jsch.JSchException: reject HostKey: vrlcm01.dmware.local”
- “Could not connect to the SSH server @ vrlcm01.dmware.local for configuration. com.jcraft.jsch.JSchException: reject HostKey: vrlcm01.dmware.local”
- “Key AAAAE2VjZHNhLXNo…..jKPxIvB0= of type ecdsa-sha2-nistp256 for host vrlcm01.dmware.local not found in”
Procedure
SSH to each vRLI/vROps/vRLCM node
- Get the keys using ssh-keyscan -p 22 localhost
-
ssh-keyscan -p 22 localhost
- Run this on an SSH session to each disconnected node to get the ecdsa-sha2-nistp256 key
-
- Look for the same key in the operationsmanager.log file in /var/log/vmware/vcf/operationsmanager/
- For each node copy the line “localhost ecdsa-sha2-nistp256 AAAAdjkorgn8y3ifnwlkfnmlwkdlmklm……..seD/=”
- Repeat the above for each vRLI (or VROPS, VRSLCM) node
Take a snapshot of the SDDC Manager appliance through the vSphere Client.
Edit each known_hosts file (see list below) and add the ecdsa key for both the FQDN and IP of the (vRLI) nodes e.g.
<vRLI Node FQDN> ecdsa-sha2-nistp256 AAAAdjkorgn8y3ifnwlkfnmlwkdlmklm……..seD/=
<vRLI Node IP> ecdsa-sha2-nistp256 AAAAdjkorgn8y3ifnwlkfnmlwkdlmklm……..seD/=
The known_hosts files are located as below:
vi /root/.ssh/known_hosts vi /etc/vmware/vcf/commonsvcs/known_hosts vi /home/vcf/.ssh/known_hosts cat /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts
- If the fourth known_hosts file is empty, you can leave if empty. Otherwise, append the lines in the text
Note: from text editor vi you can easily find text using / or ?, switch to the last line by G, i voor insert, esc to back out… Try it first before changing files unexpected.
Once complete with all nodes for the component/Aria app, restart the services on the SDDC Manager:
sh /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
Remediate the disconnected accounts and wait 24 hours to ensure they do not enter a disconnected state again
This issue comes with SDDC Manager, VCF, 4.5.0.0, VCF 4.5.2.0 and VCF 5.x
This post is specific for Aria components, such as Aria Operations, Aria Operations for Logs and Aria Suite Lifecycle Manager. Note that from within VCF 4.5.x, these products are still named VROPS, VRLI and VRSLCM.
In case of this issue for NSX-T, please check the following: NSX-T Edge Nodes Disconnected in Password Manager on SDDC (90512)