VMware Cloud Community
NickDaGeekUK
Enthusiast
Enthusiast

HelloKitty Ransomware Mitigation

Hi All 

in a recent Veeam newsletter the following note was posted.

"VMware vSphere users > be aware that the Linux version of HelloKitty ransomware is now directly targeting ESXi hosts (and apparently has been doing so since March). According to this article, it is easy to protect your ESXi hosts against this attack vector by prohibiting the execution of custom code inside ESXi using the VMkernel.Boot.execInstalledOnly parameter. Note that this parameter is officially supported and recommended starting from ESXi 7.0 only, although apparently it works fine for ESXi 6.5 and 6.7 too."

The question I have is does it work in 6.0 and if so how do I do this please.

Kind regards,
Nick.
Reply
0 Kudos
7 Replies
pkvmw
VMware Employee
VMware Employee

Hi,

for vSphere 6.0 the "General Support" ended on 12nd of March of 2020 and does not receive any security patches or bug fixes. So concerning the lack of security updates and therefore unpatched security vulnerabilities, ransomware mitigation (with this specific option) for your ESXi 6.0 hosts are most likely your least problems. The "End of Technical Guidance" is already in March 2022.

You might want to update your hosts as soon as possible to be fully supported and receive security updates.

See more here:
https://lifecycle.vmware.com/
https://www.vmware.com/support/policies/lifecycle.html

And as the Veeam newsletter notes, the option is not supported for vSphere 6.x. So you can try it on your ESXi and check if it works. If so, then you're lucky.

Regards,
Patrik

NickDaGeekUK
Enthusiast
Enthusiast

 

Hi Patrik

I TOTALLY agree with you and would if I could.

Sadly my aging servers are not listed as compatible with even the current version we have installed. It was a shot in the dark when my predecessor installed this version 6 years ago.

Hence why I am currently in mitigation mode without a support contract on an unsupported system.

At the time it was a risk he could afford to take as they were replacing another server which was running production and there were no down time risks. I don't have that luxury: any upgrades would be in place on the only server we have. 

I am praying that we get a budget for a replacement server before we either get hit or the hardware fails. Project was in hand before Covid-19 killed the economy just as effectively as it did people. Like many other companies we got financially sick and are still trying to recover.

 

Kind regards,
Nick.
Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso

Hello.
Technically the 6 licenses are valid for ESXi hosts running versions 6.0, 6.5 and 6.7.
If your server (Hardware) is running version 6 it is very likely that it can also run version 6.5 or 6.7. If your server model is not in the compatibility list for the versions indicated, you can validate the CPU model, the model of the disk controller and the models of the network cards, fiber and SAS. If you have external storage you must also validate it.
Attached is a link to validate hardware compatibility.
http://partnerweb.vmware.com/comp_guide2/search.php

 

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
NickDaGeekUK
Enthusiast
Enthusiast

Hi Enrique,

Really appreciate the link.

Having selected only the CPU (Intel X5550) I find that nothing past 6.5 U3 seems to be supported. I am currently looking at all the other pieces of hardware to check if I can go up to 6.5 U3.

I am really nervous about trying an in place upgrade of our only production environment housing all our servers. Do you have any advice or guides on doing this and how to fall back if it fails?

I am thinking that would be a complete Disaster Recovery scenario and while its a great test I think the managements view of the downtime is going to be rather dim. 

I am considering that as our DL380 G6 servers are ancient it might be cheap enough to purchase a test rig of identical configuration and try out the install of 6.5U3 or lower depending on other hardware compatibility.

Any idea on minimum RAM and HDD space needed for a test install?

Kind regards,
Nick.
Reply
0 Kudos
e_espinel
Virtuoso
Virtuoso

Hello.
If the CPU is compatible with version 6.5 this should be the version to choose. You must validate the internal disk controller, the network cards and if you have connection to an external storage you must validate the HBAs (FC, SAS ..etc) and the storage.


According to the compatibility matrix the vcenter server 6.5 Update 3 can manage ESXi Host in versions 6.5 (all levels) and 6.0 (from Update 2).


The general plan would be:
1. install the vcener server (VCSA) in a VM using the OVA version 6.5 Update 3 on one of the ESXi hosts that has  enough space and resources. Configure it with another IP and name
2. Verify the new vCenter server (web client, ping to the ESXi hosts).
3. Disconnect the ESXi hosts from the current vCenter server.
The VMs continue to run smoothly on the ESXi hosts.
4. Shutdown the current vcener server
5. Connect the ESXi hosts (requires root user and password) to the new vcenter sever version 6.5 Update 3.
6. Verify that everything is working normally and that there are no errors.
At this point you can take a break to familiarize yourself with the web client (new method).

In case of problems, the return path would be:
A. disconnect the ESXi hosts from the new vcenter server.
B. shut down the new vcenter server
C. power on the vcenter server 6.0
D. connect the ESXi hosts to the vcenter server 6.0
E. Verify that everything is working normally.

Upgrade ESXi host
1. Obtain the ISO of version 6.5 Update 3, create a bootable DVD or USB Key.
2. If possible move all VMs from one ESXi host to the other ESXi hosts according to the available resources,
3. Shutdown the chosen ESXi host
4. Boot the server with the previously created DVD or USB key.
5. Perform the ESXi upgrade by choosing the option that includes preserving the VMFS (be careful with this, it is very important).
When the upgrade is finished the ESXi should reboot and start up without any problems connecting to the new vcenter server.
7 Verify that everything is normal with the ESXi host updated with version 6.5 Update 3.
8. Move the VMs from another ESXi host (version 6) to the ESXi host recently upgraded with version 6.5, verify that the VMs are working without problems.
9. Perform steps 3 to 7 with this other ESXi host.
10. Repeat the steps with the remaining ESXi hosts with version 6.0.

It is recommended to wait a reasonable time after finishing the upgrade of the first ESXi host and verify that all VMs are working normally.
If there are any problems with the ESXi host upgraded to aversin 6.5 Update 3 you can try to solve it if it is not critical or run the rollback to version 6.0.

Rollback to version 6.0
A. Move all the VMs from the ESXi host to the other ESXi hosts.
B. Shutdown the ESXi host
C. Boot the server with the previously created DVD or USB key
D. Install ESXi by choosing the option that includes preserving VMFS, a clean installation will be performed, so it must be reconfigured with IP, name and more parameters.
E. At the end of the installation of version 6.0 the server must reconnect to the vcenter server.

You must obtain the ISO of version 6.0 and create a bootable DVD or USB key.

If you can get a server with identical characteristics for testing purposes it would be a very good option, it could even be used as a replacement server in case of hardware failure.
Because of the age of the hardware the price of the server should be very affordable.

 

 

Enrique Espinel
Senior Technical Support on IBM, Lenovo, Veeam Backup and VMware vSphere.
VSP-SV, VTSP-SV, VTSP-HCI, VTSP
Please mark my comment as Correct Answer or assign Kudos if my answer was helpful to you, Thank you.
Пожалуйста, отметьте мой комментарий как Правильный ответ или поставьте Кудо, если мой ответ был вам полезен, Спасибо.
NickDaGeekUK
Enthusiast
Enthusiast

Absolutely brilliant!😀

Thanks for your very detailed advice.

I am working on validating every part and looking for a duplicate server to use as a test and backup server. Thanks very much. Will update this with a progress report when I get to the testing stage.

Kind regards,
Nick.
Reply
0 Kudos
NickDaGeekUK
Enthusiast
Enthusiast

Hi Enrique,

Thought you might like an update, the dreaded SD card corruption issue forced my hand during a recent patch session. I had to replace card and reinstall ESXi from scratch. Ironic as one of the reasons for the reboot was to enable UserVars.ToolsRamdisk in response to this VMware KB about SD card durability. Thought this would be the best time to try 6.5. 

Sadly, even after checking all the hardware was compatible as you so kindly showed me, it doesn't work. It installs and boots but is unreliable. I had to revert to the HPE 6.0 U3 image and repatch.

What is really annoying though is what started all this was wanting to mitigate HelloKitty via setting the VMkernel.Boot.execInstalledOnly option. It is a waste of time because of an unsigned VIB used by VMWare in its Baseline update manager.

See here for VMWare KB confirming this and Secure your VMware ESXi hosts against ransomware in three simple steps! - Truesec which lead me to it. (important text from the article below my signature)

Sadly again the workaround of moving from Baseline to Image is not an option below 7.0 so even I had succeeded in moving to 6.5 it would not help.

Extremely disappointing I am sorry to say, especially considering the reason we can't mitigate is an unsigned VIB from the vendor, but nevertheless thanks for your help and sorry for wasting your time.

Off to CLI in to my hosts to undo the damage I did trying to follow best practice. 😞

all the best

Nick

 

Step 3: execInstalledOnly

---
[UPDATE 2022-04-28]: IMPORTANT: If you are updating/patching ESXi hosts using vSphere Lifecycle Manager (formerly known as Update Manager) using the old fashioned Baseline method rather than the newer Image method (link to article describing the difference) you will bump into problems when having execInstalledOnly set to TRUE.

The recommended workaround is to switch to the Image method, since it will also bring other benefits. If you can't switch, you will unfortunately need to wait until vSphere 8.0 before being able to enable execInstalledOnly.

When using the baseline method and enabling execInstalledOnly, the error message you will get when scanning an ESXi host for patch compliance is:

Cannot deploy host upgrade agent. Ensure that vSphere Lifecycle Manager is officially signed. Check the network connectivity and logs of host agent and vpxa for details.

 

If you are getting this error message, either switch to the vLCM Image method or follow the instructions in https://kb.vmware.com/s/articl... to revert the execInstalledOnly setting and the enforcement of the setting.

If you set execInstalledOnly back to FALSE but keep the enforcement at TRUE, you will get a purple screen when rebooting the ESXi host. The purple screen is by design, and is described at the end of this blog post.

# Revert the enforcement of the setting
esxcli system settings encryption set --require-exec-installed-only=FALSE 

# Revert the setting itself
esxcli system settings kernel set -s execInstalledOnly -v FALSE 

The reason for the Lifecycle Manager problem is that when using Baselines, VMware is apparently using an unsigned VIB, which is the update agent that Lifecycle Manager pushes out to the ESXi hosts when scanning or updating them. This is a very unfortunate mistake, which I hope they will fix before vSphere 8.0. I recommend you open a support case and tell VMware they need to fix this in 7.0 as well.

Kind regards,
Nick.
Reply
0 Kudos