To resolve a 502 or 503 error on an SDDC Manager and free up disk space, you can troubleshoot log rotation issues and clean up unnecessary files. First, verify that log rotation is configured correctly by checking the /etc/logrotate.conf file and the /etc/logrotate.d directory. You can also check for large files and delete them if they are not needed. If journal logs are causing the issue, cleaning them up can free up space. If the root partition is still full, you may need to increase its size. Note however, that this process may not be applicable for SDDC Manager 4.5 and 5.0. I’m running 5.2.x currently.
How the process above works exactly, will be shown in this blog post.
Firstly, you log in to SDDC Manager, and you find something like this. That’s a showstopper, since you were just in the process to extend one of your cluster, do some certificate management or … Anyway, you need SDDC Manager… NOW!
502 Bad Gateway - nginx
Now. Well, first use SSH into SDDC Manager, login as the vcf user, login to root with the command su root.
Check if some or several partitions have run into disk space problems… In the case below, you will see that /dev/mapper/vg_system_lv_root is full, 100% usage. No wonder, but why?
There will be some particularly large files, and to check, run the following command:
du -ah /var/log | sort -rh | head -n 20
This shows a list of 20 largest file in the /var/log folder structure
As you can (clearly) see, the messages.1 file is quite large and has possibly filled up the complete root partition. When checking the /var/log folder itself, you will find that the previous files have been rotated (compressed into .gz files). For some reason, log rotation has failed. If this is the symptom, VMware Broadcom has the KB article for you! And I will guide you 🙂
Depending on your situation, you will find one or more large files: the messages file, the audit.log or the auth.log file is not rotated. In each case, run the following three commands.
Note: prior to this, if you want to, you can copy/downoad the large file through WinSCP for further analysis:
echo > /var/log/messages
echo > /var/log/audit/audit.log
echo > /var/log/auth.log
Now the files have been reset, you need to confirm logrotation configuration is correct, and restart the logrotation service, by using the following command:
logrotate -f /etc/logrotate.conf
You will notice, a (new) messages.1.gz has been created and a messages file re-appears.
Restart the SDDC Manager VM and you will find the magic re-appears
