I'm seeing the same issue with ESXi 4.1 on two brand-new servers. I can configure things for a while after boot, and other things appear fine for a while after boot, then I stop being able to make any config changes, same as reported in this thread. Another reboot and I'm back in.
The whole comment about the disk (in my case a local drives, brand new, and having run perfectly on tests run with another OS) are still writeable. Logging into maintenance access (via SSH) I can certainly create files without issue. (a simple "touch foo" does not fail, for example). I also see the system successfully writing to /var/log/messages, something that would not happen if the file system had switched to read-only.
Other things stop working right too... IMPI status of host sensors stop being reported too. So it's not just updating configs that doesn't work, it's a lot of interaction with the system. Is it possible the hardware has issues? Sure. Two identical systems with the same issue make it unlikely (and a third, that I'd used showed the same issues too).
Something running on a periodic basis appears to cause problems, and rebooting appears to solve them. This sure sounds like a software problem and not hardware. Coming up with a way to diagnose this would surely be helpful, and it sounds like might help a lot of folks.
