We have had rolling issues across our datacenters due to NSX-T management nodes and edge nodes getting to 100% in var+log. We have resolved the issue by setting the root/admin/audit passwords to never expire, and by clearing our syslog and auth.log files.
However, we have one NSX Edge Node that is being problematic. The passwords we had set for root and admin are not working. We attempted to use the password reset feature through GRUB, but that password isn't sticking after a reboot -f. The theory is that because var log is full, the changes aren't being applied. SO...
logged in as root@(none) I am trying to clear out the syslog and auth.log files to get some free space in var+log. I have been able to start the LVM2 services, and see the var log LV with lvscan as inactive. However, when I use lvchange -a y on that volume, it just sits there forever. the cursor blinks, and I can type but I can't break out of the command unless I reboot the node. Because the LV is inactive, I can't mount it.
Is there a way to clear out var log using this (or any other) method without having NSX root/admin rights, and just the root@(none) user?
