VMware Cloud Community
Monolithic
Enthusiast
Enthusiast

ESXi File Integrity Monitoring

Hey everyone,

Not sure if this is the right place to put this, but i'm wondering what people are using for file integrity monitoring.  In the April Security Hardening Document, it suggests "establishing and maintaining configuration file integrity".  There are certain intrusion detection software that can do this, such as OSSEC using the  ssh_generic_diff example, which you can specify which files to monitor.  I was wondering if anyone is using this or what other methods are there to verify file integrity on an esxi host.  Since there is no service console there are no native md5 or other checksum applications that are apart of or built into ESXI.

Are there any power cli commands that can remotely pull this information down or do a remote file integrity check?

If not, I was thinking about creating a script that will either grab the file size and date modified of the specific files and then run a comparison of the current and previous values at a certain interval.

Anyone do anything remotely like this? I'm trying to get away with not having to build another server (OSSEC) for monitoring.

Thanks in advanced!

-Robert

Reply
0 Kudos
15 Replies
vmroyale
Immortal
Immortal

Hello.

Note: This discussion was moved from the VMware ESXi 4 community to the Security and Compliance community.

Good Luck!

Brian Atkinson | vExpert | VMTN Moderator | Author of "VCP5-DCV VMware Certified Professional-Data Center Virtualization on vSphere 5.5 Study Guide: VCP-550" | @vmroyale | http://vmroyale.com
Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

You can do something like this using the vifs -get command from vMA but pulling data down and running md5 would be very difficult to do. There are better ways, but unfortunately I have not seen anything developed yet.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
Monolithic
Enthusiast
Enthusiast

Thanks for the reply, i'm not actively using the vMA, I'm having all logs be sent to a remote logging server for more dissection and control on a customized syslog box.  I find it irritating that VMware best practices specify something that you should do but have no easy way of achieving it. I spent the last two days researching how this should be done and since I dont' want to build another box just to monitor file integrity, i'm going to implement a script that will create a listing of the file name, byte size and date modified for each of the config and other files listed in the hardening guide, then create a secondary file of the same information on a schedule.  I'm then going to run a file comparison between the two using our zabbix monitoring server.  This is the best solution that i've seen for my environment.  OSSEC looks easier as it has the polling and comparison all worked out, you just need to feed it credentials for logging in and which files to watch.

Thank you for the reply and please let me know if there are any other ways around this if you can think of?

-Robert

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

Actually Byte Size is not going to be good enough for this. You will need to use a hash. Your tool could use vCLI which can be installed on any system and then get access to vifs --get that way.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
Monolithic
Enthusiast
Enthusiast

Thanks for your fast reply, is there an easy way to get the hash, or would you recommend pulling down the file itself and then running an independent hash on it?

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

There are a few ways to do this....

1) download the file, run the hash, store the hash. This is by far the simpler of the two but would take up ALOT of networking

2) find an sha256 hash tool that works within Busybox, then inject this code into busybox, and execute the command remotely looking at the hash values. I have not tried #2 yet but has its own security implications as well. This would also require SSH or something like that to be open so you can run the command.

So at the moment #1 is the easiest to do, but you need to understand what you are downloading first. What files are the most important to you.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
Monolithic
Enthusiast
Enthusiast

Again thank you so much for your quick response and reply!!!

I'm now researching #1 now, it seems very doable to me, just some scripting and secure storage. #2 intrigues as well, I have SSH opened for remote management, sha256 injection into busybox scares me, but i'm going to look into it further.  I will keep you informed on what I end up doing! I love this stuff, but it's frustrating because there is minimal documentation and literally no examples of anyone doing this on the interwebs.

Again thanks you much for your insight, you've definiately pointed me in the right direction!!

For anyone else that needs help, i've located the list of files that are in the security hardening document April 2011 version, that need to be  "monitored" for integrity - some of the filenames were different on the esxi host than the document:

The following files are listed in /host and do not change unless modified by a change to the system:

  1. /etc/vmware/esx.conf
  2. /etc/vmware/snmp.xml
  3. /etc/vmware/vmware.lic
  4. /etc/vmware/hostd/proxy.xml
  5. /etc/hosts
  6. /etc/motd
  7. /etc/openwsman/openwsman.conf
  8. /etc/sfcb/sfcb.cfg
  9. /etc/syslog.conf
  10. /etc/vmware/hostd/config.xml - listed as hostAgentConfig.xml
  11. /etc/vmware/ssl/rui.crt - listed as ssl_cert
  12. /etc/vmware/ssl/rui.key - listed as ssl_key
  13. /etc/vmware/config   - listed as vmware_config
  14. /etc/vmware/configrules  - listed as vmware_configrules

The following files are listed in /host and their DATE MODIFIED changes, however the size seems to remain in tact

  1. /etc/vmware/license.cfg
  2. /etc/opt/vmware/vpxa/vpxa.cfg
  3. /var/log/ipmi/0/sdr_content.raw
  4. /var/log/ipmi/0/sdr_header.raw
  5. /var/log/ipmi/0/sensor_readings.raw

The following files are listed in /host and they DO CHANGE:

  1. /var/log/messages
  2. /var/log/vmware/hostd.log
  3. /var/log/vmware/vpx/vpxa.log
    Reply
    0 Kudos
    Monolithic
    Enthusiast
    Enthusiast

    Hey again!

    I actually just ran the help file for busybox directly on my esxi host and there is a sha1sum function already built in! The syntax is sha1sum <filename>

    So I think I can now poll the sha1sum directly from the esxi host using an ssh check through my zabbix host!!!!!! This is so exciting! Should I try and figure out a sha256 hash tool or would sha1 be sufficient? THANKYOU!!!!

    Reply
    0 Kudos
    Texiwill
    Leadership
    Leadership

    Hello,

    sha1sum may be sufficient for your needs. But you should note that sha1 is no longer considered cryptographically safe. However as an early warning device it is sufficient for smaller files.

    If this is a US Federal set of computers you will be required to use sha256... Good read o http://www.schneier.com/blog/archives/2005/02/sha1_broken.html, read the comments if interested. Yes it is an old hack

    sha1sum is better than nothing so yes I would use it. As I do not run ESXi yet (other issues) I still use the opensource 'tripwire'.

    Best regards,

    Edward L. Haletky

    Communities Moderator, VMware vExpert,

    Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

    Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

    --
    Edward L. Haletky
    vExpert XIV: 2009-2023,
    VMTN Community Moderator
    vSphere Upgrade Saga: https://www.astroarch.com/blogs
    GitHub Repo: https://github.com/Texiwill
    Reply
    0 Kudos
    Monolithic
    Enthusiast
    Enthusiast

    Thanks again, I really appreciate your reply. You have been very helpful. I'm going to research a little more about injecting an sha256 bit program into busybox. Hopefully VMware will include that iin their next update! I'll let you know if I end up finding a solution that works.

    -Robert

    Reply
    0 Kudos
    Texiwill
    Leadership
    Leadership

    Hello,

    Also thanks for compiling the list of files! None of those are so big that you cannot use vifs --get to download them for those who want to do so.

    Best regards,

    Edward L. Haletky

    Communities Moderator, VMware vExpert,

    Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

    Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

    --
    Edward L. Haletky
    vExpert XIV: 2009-2023,
    VMTN Community Moderator
    vSphere Upgrade Saga: https://www.astroarch.com/blogs
    GitHub Repo: https://github.com/Texiwill
    Reply
    0 Kudos
    MikeMoskalick
    Contributor
    Contributor

    Sounds like you have this covered, but my company (a VMware partner) also offers integrity monitoring for VMware ESX and ESXi servers.  If interested, you can find more information at www.cimcor.com/cimtrak/products/vmwaremonitoring

    Thanks and have a great weekend,

    Mike Moskalick

    Cimcor, Inc.

    Reply
    0 Kudos
    Monolithic
    Enthusiast
    Enthusiast

    Thanks Mike,

    I like to do most stuff in house, but will take a look at your company's solution.

    For anyone else who needs help running a script or pulling just the sha1 hash directly from ESXi the command is:

    #sha1sum <filename> | awk '{print $1}'

    This will return the hash only and not the filename!

    Reply
    0 Kudos
    mweigand
    Contributor
    Contributor

    I am having the same concerns with regards to how to handle this requirement HMT03 in the hardening guide.  Would you be willing to share your script and how you have implemented it?   I have a concern that it is not very practical to monitor multiple files from multiple servers in a manual fashion and detect unauthorized changes.

    Thanks

    Mike

    Reply
    0 Kudos
    Bezgodov
    Contributor
    Contributor

    I try to configure OSSEC server to monitor file integrity of ESXi 5.0.0 server by SSH and have some problem. I register the ESXi server with /var/ossec/agentless/register_host.sh on OSSEC server and add it to <agentless> in ossec.conf, but the check is not work. Then I try to run the /var/ossec/agentless/ssh_generic_diff and /var/ossec/agentless/ssh_integrity_check_linux manually to test it. The result is:

    root@bt:/var/ossec# sudo -u ossec ./agentless/ssh_generic_diff root@192.168.1.6 ls

    spawn ssh root@192.168.1.6

    Password: (pause and flashing cursor)

    ERROR: Timeout while connecting to host: root@192.168.1.6 .

    root@bt:/var/ossec#

    If enter the password then the result is:

    root@bt:/var/ossec# sudo -u ossec ./agentless/ssh_generic_diff root@192.168.1.6 ls

    spawn ssh root@192.168.1.6

    Password: mypassword

    ERROR: Timeout while connecting to host: root@192.168.1.6 .

    root@bt:/var/ossec# mypassword

    mypassword: command not found

    root@bt:/var/ossec#

    Please tell me how can I fix it? Thank you.

    Evgeniy.

    Reply
    0 Kudos