Help! With vmware.log files

[Jujo]

Guys,

I had several hundred of the vmware-X.log files on a couple of my VMs, both VMs were on the same host. Oddly, there were 4 VMs on the host, but only 2 had this problem. It seems the problem happened at a specific time as both VMs started generating logs within 30 seconds of each other. I have checked vpxa, messages, vmkernel, and the hostd logs. Don't really see anything that jumps out at me around that time.

As you can see, it looks as if the VMs started generating new log files because they couldn't backup a particular log file. Maybe there was a file lock on the current log files?

I have seen that others have posted about how to clean these up (which I've done) but I'm wondering if anybody has an idea of what would cause this behavior? I have pasted the contents of one of the log files below.

Also, if anyone could shed some light on the stuff about "power on failed" I'd appreciate it. These VMs were running the entire time these logs were being written so I don't get that either..

Jun 15 16:49:51.194: vmx| LogBackupByRenumber: failed to remove /vmfs/volumes/diskX/vm-name/vmware-7.log failed: Device or resource busy

Jun 15 16:49:51.470: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Jun 15 16:49:51.541: vmx| VMXVmdbCbVmVmxExecState: Exec state change requested to state poweredOn without reset

Jun 15 16:49:51.541: vmx| TOOLS delaying state change request to state 3

Jun 15 16:49:51.541: vmx| PowerOn

Jun 15 16:49:51.548: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Jun 15 16:49:51.619: vmx| VMMon_CreateVM: vmmon.numVCPUs=1

Jun 15 16:49:55.714: vmx| Msg_Post: Error

Jun 15 16:49:55.714: vmx| http://msg.vmmonVMK.creatVMFailed Could not power on VM : No swap file.

Jun 15 16:49:55.714: vmx| http://msg.monitorLoop.createVMFailed Failed to power on VM----

---

Jun 15 16:49:55.731: vmx| Module MonitorLoop power on failed.

Jun 15 16:49:55.731: vmx| VMX_PowerOn: ModuleTable_PowerOn = 0

Jun 15 16:49:56.285: vmx| vmdbPipe_Streams Couldn't read: OVL_STATUS_EOF

Jun 15 16:49:56.286: vmx| VMX idle exit

Jun 15 16:49:56.297: vmx| Flushing VMX VMDB connections

Jun 15 16:49:56.332: vmx| IPC_exit: disconnecting all threads

Jun 15 16:49:56.332: vmx| VMX exit.

[AndreTheGiant]

Could be a simple problem (VMFS full) or also a bad problem (VMFS corrupted).

How much free space do you have?

Andre

[dconvery]

After you have fixed this issue, you can add log.rotateSize=500000 and log.keepOld=6 entries in the advanced settings or directly in the .vmx to avoid the problem in the future. The VM will need to be powered off to add the settings. They should be added to any templates as a security practice.

Dave Convery

VMware vExpert 2009

Careful. We don't want to learn from this.

Bill Watterson, "Calvin and Hobbes"

[dconvery]

If you are OK on disk space, you may want to try to clone the VMs to see if the problem follows the clones.

Dave Convery

VMware vExpert 2009

Careful. We don't want to learn from this.

Bill Watterson, "Calvin and Hobbes"

[Jujo]

The VMFS had plenty of free space. I've since learned about managing the logs via the Security Hardening doc. That seems like a good idea.

I moved the vmware.log file out of the vm-name directory and then vmotioned the VMs to another host and it stopped creating the logs. I rebooted the bad host andmoved the VMs back to it today and so far so good. I still wish I knew what happened though. I got it fixed ok and I know what to do if it ever happens again.

[Texiwill]

Hello,

There is a limit of 31000 or so files on a VMFS, you may have also hit that limit. There is also a filesize limit based on your blocksize. You could also have hit that limit.

Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]