Kicking off a post to see if the community has shell scripters who can help out all the vCenter admins out there to draft a script to mitigate the log4j RCE vulnerability in vCSA.
The mitigation steps for each vCSA version are detailed in webpage:
https://kb.vmware.com/s/article/87081
Although the KB clearly mentions that the instructions are "... a temporary solution only", I automated the instructions for my lab environment.
DISCLAIMER: The following code has only been tested in a limited environment. Use at your own risk!!!
Some caveats:
Again: Use at your own risk!!!
#Requires -Modules @{ ModuleName="Posh-Ssh"; ModuleVersion="3.0.0" }
$vcsaFQDN = 'your.vcsa' # FQDN of the VCSA
$user = 'root' # User that can SSH into VCSA
# Provide PSCredential
# This sample uses a VICredentialStoreItem
$viCred = Get-VICredentialStoreItem -Host $vcsaFQDN -User $user
$cred = New-Object -TypeName PSCredential -ArgumentList $user, (ConvertTo-SecureString -String $viCRed.Password -Force -AsPlainText)
# Required for the stop/start of All Services
$vcsaTimeout = 300
#region Code snippets
$codeVersion = @'
vpxd -v
'@
$codeVMON0and1 = @'
sed -i.bak 's/exec $java_start_bin $jvm_dynargs "$@"/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"\nexec $java_start_bin $jvm_dynargs $log4j_arg "$@" /usr/lib/vmware-vmon/java-wrapper-vmon
'@
$codeVMON2and3 = @'
sed -i.bak 's/exec $java_start_bin $jvm_dynargs $security_dynargs $original_args/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"\nexec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args/g' /usr/lib/vmware-vmon/java-wrapper-vmon
'@
$codeVMON7Restart = @'
chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon;
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon;
service-control --stop --all;
service-control --start --all;
'@
$codeUpdMgr = @'
sed -i.bak -e '$a\-Dlog4j2.formatMsgNoLookups=true' /usr/lib/vmware-updatemgr/bin/jetty/start.ini;
service-control --restart vmware-updatemgr;
'@
$codeAnaSrv = @'
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak;
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class;
service-control --restart vmware-analytics;
'@
$codeDBCC = @'
cp /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak;
zip -q -d /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
'@
$codeVerVMON = @'
ps auxww
'@
$codeVerUpdMgr = @'
cd /usr/lib/vmware-updatemgr/bin/jetty/;
java -jar start.jar --list-config
'@
$codeVerAnaSrv = @'
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
'@
$codeVerDBCC = @'
grep -i jndilookup /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar | wc -l
'@
#endregion
$session = New-SSHSession -ComputerName $vcsaFQDN -Credential $cred -AcceptKey
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVersion -Verbose
$vcsaVersion = [Version]($result.Output.Split()[2])
switch ($vcsaVersion.Major) {
# VCSA 6
'6' {
# To be completed
}
# VCSA 7
'7' {
#region vMON Service
if ('2', '3' -contains $vcsaVersion.Build ) {
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON2and3
} else {
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON0and1
}
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON7Restart -TimeOut $vcsaTimeout
#endregion
#region Update Manager Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeUpdMgr
#endregion
#region Analytics Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeAnaSrv
#endregion
#region DBCC Utility
$result = Invoke-SSHCommand -SSHSession $session -Command $codeDBCC
#endregion
#region Verify
#region vMON Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerVMON
Write-Host 'vMON Service ' -NoNewline
if ($result.Output | Where-Object { $_ -match '/usr/java/jre-vmware/bin' -and $_ -match '-Dlog4j2.formatMsgNoLookups=true' }){
Write-Host -ForegroundColor green 'ok'
}
else{
Write-Host -ForegroundColor red 'nok'
}
#endregion
#region Update Manager
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerUpdMgr
Write-Host 'Update Manager Service ' -NoNewline
if ($result.Output | Where-Object { $_ -match [RegEx]::Escape('log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)')}) {
Write-Host -ForegroundColor green 'ok'
} else {
Write-Host -ForegroundColor red 'nok'
}
#endregion
#region Analytics Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerAnaSrv
Write-Host 'Analytics Service ' -NoNewline
if ($result.Output -eq 0) {
Write-Host -ForegroundColor green 'ok'
} else {
Write-Host -ForegroundColor red 'nok'
}
#endregion
#region DBCC Utility
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerDBCC
Write-Host 'DBCC Utility ' -NoNewline
if ($result.Output -eq 0) {
Write-Host -ForegroundColor green 'ok'
} else {
Write-Host -ForegroundColor red 'nok'
}
#endregion
#endregion
}
Default {
Throw "Unhandled VCSA version $vcsaVersion"
}
}
Remove-SSHSession -SSHSession $session | Out-Null
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I suspect I don't have this issue since I set the default shell for root to /bin/bash.
That is probably also the reason the KB with the Python script asks you to run it inside the VCSA.
You can change the default shell with the command (while in the bash shell)
chsh -s /bin/bash root
this will permanently change that for root.
To go back to the appliance, use (again inside the bash shell).
chsh -s /bin/appliancesh root
I'm not sure if there is an easier solution.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hey Luc, I think I figured out a solution. I implemented a prep script to ensure that bash was configured as the default vCSA shell before I ran your script. If the prep script prints out the vCSA software version, your script will run without issue.
#Requires -Modules @{ ModuleName="Posh-Ssh"; ModuleVersion="3.0.0" }
$vcList = Import-Csv .\Log4j_vCSA_Inventory.csv | Select DNSName | Sort-Object DNSName
$cred = Get-Credential -Username "root" -Message "Please Enter Root Password:"
Foreach($vcsa in $vcList) {
# Create SSH Session
$session = New-SSHSession -ComputerName $vcsa.DNSName -Credential $cred –AcceptKey
$session
# Create SSH Shell Stream
$stream = New-SSHShellStream -SSHSession $session -TerminalName tty
# Switch to bash shell
$stream.WriteLine("shell")
sleep 3
while ($stream.Length -ne 0){
$stream.Read()
}
# Configure Bash as default shell
$stream.WriteLine("chsh -s /bin/bash root")
sleep 3
while ($stream.Length -ne 0){
$stream.Read()
}
# Execute Bash shell command as a test
$stream.WriteLine("vpxd -v")
sleep 3
while ($stream.Length -ne 0){
$stream.Read()
}
# Disconnect stream
$stream.Close()
Remove-SSHSession -SSHSession $session | Out-Null
Write-Host "Bash enabled as default shell..."
}
Although the KB clearly mentions that the instructions are "... a temporary solution only", I automated the instructions for my lab environment.
DISCLAIMER: The following code has only been tested in a limited environment. Use at your own risk!!!
Some caveats:
Again: Use at your own risk!!!
#Requires -Modules @{ ModuleName="Posh-Ssh"; ModuleVersion="3.0.0" }
$vcsaFQDN = 'your.vcsa' # FQDN of the VCSA
$user = 'root' # User that can SSH into VCSA
# Provide PSCredential
# This sample uses a VICredentialStoreItem
$viCred = Get-VICredentialStoreItem -Host $vcsaFQDN -User $user
$cred = New-Object -TypeName PSCredential -ArgumentList $user, (ConvertTo-SecureString -String $viCRed.Password -Force -AsPlainText)
# Required for the stop/start of All Services
$vcsaTimeout = 300
#region Code snippets
$codeVersion = @'
vpxd -v
'@
$codeVMON0and1 = @'
sed -i.bak 's/exec $java_start_bin $jvm_dynargs "$@"/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"\nexec $java_start_bin $jvm_dynargs $log4j_arg "$@" /usr/lib/vmware-vmon/java-wrapper-vmon
'@
$codeVMON2and3 = @'
sed -i.bak 's/exec $java_start_bin $jvm_dynargs $security_dynargs $original_args/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"\nexec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args/g' /usr/lib/vmware-vmon/java-wrapper-vmon
'@
$codeVMON7Restart = @'
chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon;
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon;
service-control --stop --all;
service-control --start --all;
'@
$codeUpdMgr = @'
sed -i.bak -e '$a\-Dlog4j2.formatMsgNoLookups=true' /usr/lib/vmware-updatemgr/bin/jetty/start.ini;
service-control --restart vmware-updatemgr;
'@
$codeAnaSrv = @'
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak;
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class;
service-control --restart vmware-analytics;
'@
$codeDBCC = @'
cp /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak;
zip -q -d /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
'@
$codeVerVMON = @'
ps auxww
'@
$codeVerUpdMgr = @'
cd /usr/lib/vmware-updatemgr/bin/jetty/;
java -jar start.jar --list-config
'@
$codeVerAnaSrv = @'
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
'@
$codeVerDBCC = @'
grep -i jndilookup /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar | wc -l
'@
#endregion
$session = New-SSHSession -ComputerName $vcsaFQDN -Credential $cred -AcceptKey
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVersion -Verbose
$vcsaVersion = [Version]($result.Output.Split()[2])
switch ($vcsaVersion.Major) {
# VCSA 6
'6' {
# To be completed
}
# VCSA 7
'7' {
#region vMON Service
if ('2', '3' -contains $vcsaVersion.Build ) {
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON2and3
} else {
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON0and1
}
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON7Restart -TimeOut $vcsaTimeout
#endregion
#region Update Manager Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeUpdMgr
#endregion
#region Analytics Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeAnaSrv
#endregion
#region DBCC Utility
$result = Invoke-SSHCommand -SSHSession $session -Command $codeDBCC
#endregion
#region Verify
#region vMON Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerVMON
Write-Host 'vMON Service ' -NoNewline
if ($result.Output | Where-Object { $_ -match '/usr/java/jre-vmware/bin' -and $_ -match '-Dlog4j2.formatMsgNoLookups=true' }){
Write-Host -ForegroundColor green 'ok'
}
else{
Write-Host -ForegroundColor red 'nok'
}
#endregion
#region Update Manager
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerUpdMgr
Write-Host 'Update Manager Service ' -NoNewline
if ($result.Output | Where-Object { $_ -match [RegEx]::Escape('log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)')}) {
Write-Host -ForegroundColor green 'ok'
} else {
Write-Host -ForegroundColor red 'nok'
}
#endregion
#region Analytics Service
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerAnaSrv
Write-Host 'Analytics Service ' -NoNewline
if ($result.Output -eq 0) {
Write-Host -ForegroundColor green 'ok'
} else {
Write-Host -ForegroundColor red 'nok'
}
#endregion
#region DBCC Utility
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVerDBCC
Write-Host 'DBCC Utility ' -NoNewline
if ($result.Output -eq 0) {
Write-Host -ForegroundColor green 'ok'
} else {
Write-Host -ForegroundColor red 'nok'
}
#endregion
#endregion
}
Default {
Throw "Unhandled VCSA version $vcsaVersion"
}
}
Remove-SSHSession -SSHSession $session | Out-Null
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Thanks Luc!
I will be testing (in DEV) this evening and will give feedback on how it goes.
There is an official Python script in KB87088
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi Luc,
I'm encountering errors on the following line:
$result = Invoke-SSHCommand -SSHSession $session -Command $codeVersion -Verbose
Error: Unknown command: `vpxd'
I've encountered issues like this before when using Posh-Ssh to execute command against vCSA, but we never got to the bottom of it.
New-SSHSession executes successfully, but Invoke-SSHCommand fails. I'm unsure if the root cause is related to timing issues, or the default shell that the Invoke-SSHCommand commands execute against.
I am able to successfully execute "service-control --status" with Invoke-SSHCommand.
Which VCSA version are you running this against?
Did you check that the Posh-Ssh module is at least version 3.0.0?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
PoshSsh Version: 3.0.0
PowerShell Version: 5.1.19041.1320
PowerCLI Version: 12.4.1
I'm running my tests against a single, standalone vCSA 7.0.2 appliance.
When you establish an intercative SSH session with the VCSA, does the vpxd command work?
Or do you get the same error?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
No, the vpxd command fails with:
Host : vcsa-01.lab.local
Output : {Unknown command: `vpxd'}
ExitStatus : 0
I get the same error when I SSH to the vCSA via Putty and execute "vpxd -v" against the default appliance shell:
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"
Command> vpxd -v
Unknown command: `vpxd'
Command>
In Putty, once I type "shell" and access the Bash shell, i can successfully execute "vpxd -v".
It looks to me that the Invoke-SSHCommand command is executing against the appliance shell instead of the Bash shell.
I tried the following code:
__________________________________________________________________
$shellChange = @'
shell
'@
Invoke-SSHCommand -SSHSession $session -Command $shellChange -Verbose
__________________________________________________________________
But the command times out.
_______________________________________________________________________
Exception calling "EndExecute" with "1" argument(s): "Command 'shell' has timed out."
At C:\Program Files\WindowsPowerShell\Modules\Posh-SSH\3.0.0\Posh-SSH.psm1:266 char:25
+ $Output = $_.cmd.EndExecute($_.Async)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : SshOperationTimeoutException
I suspect I don't have this issue since I set the default shell for root to /bin/bash.
That is probably also the reason the KB with the Python script asks you to run it inside the VCSA.
You can change the default shell with the command (while in the bash shell)
chsh -s /bin/bash root
this will permanently change that for root.
To go back to the appliance, use (again inside the bash shell).
chsh -s /bin/appliancesh root
I'm not sure if there is an easier solution.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Seems I forgot my own solution for this VCSA - shell issue, see Solved: Re: Script to Create vCenter Accounts - VMware Technology Network VMTN
By using a Stream you can enter the 'shell' command.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi Luc,
For further feedback:
Once I manually SSH to the vCSA, enter the shell (by typing 'shell'), then reconfigure the default shell to bash (by typing 'chsh -s /bin/bash root'), the script runs like a dream. It took approximately 10 mins to execute against an empty vCSA 7.0.2:
vMON Service ok
Update Manager Service ok
Analytics Service ok
DBCC Utility ok
Hey Luc, I think I figured out a solution. I implemented a prep script to ensure that bash was configured as the default vCSA shell before I ran your script. If the prep script prints out the vCSA software version, your script will run without issue.
#Requires -Modules @{ ModuleName="Posh-Ssh"; ModuleVersion="3.0.0" }
$vcList = Import-Csv .\Log4j_vCSA_Inventory.csv | Select DNSName | Sort-Object DNSName
$cred = Get-Credential -Username "root" -Message "Please Enter Root Password:"
Foreach($vcsa in $vcList) {
# Create SSH Session
$session = New-SSHSession -ComputerName $vcsa.DNSName -Credential $cred –AcceptKey
$session
# Create SSH Shell Stream
$stream = New-SSHShellStream -SSHSession $session -TerminalName tty
# Switch to bash shell
$stream.WriteLine("shell")
sleep 3
while ($stream.Length -ne 0){
$stream.Read()
}
# Configure Bash as default shell
$stream.WriteLine("chsh -s /bin/bash root")
sleep 3
while ($stream.Length -ne 0){
$stream.Read()
}
# Execute Bash shell command as a test
$stream.WriteLine("vpxd -v")
sleep 3
while ($stream.Length -ne 0){
$stream.Read()
}
# Disconnect stream
$stream.Close()
Remove-SSHSession -SSHSession $session | Out-Null
Write-Host "Bash enabled as default shell..."
}
Great!
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Thanks again Luc. I learnt so many new PowerShell skills in your solution. How to implement different types of "sed" commands (substitute and append), how to implement Posh-Ssh SSH shell stream and shell commands, and how to do a PowerShell regex! You're an absolute legend when it comes to tackling the unknowns in automated VMware PowerShell scripted solutions.