Hi GregRoberts201110141,
Do you also have access to the ESXi hosts / vCenter (guessing not)? Perhaps you can request a read only account if not, then we can guide you through some additional things you can check. You should ideally have PowerCLI access (or other preferred API access into the virtual infrastructure) so you can determine if the issues you are observing are at layers lower than the Guest Operating System.
For example, you would want to check if the vDC is oversubscribed (i.e. limited by Resource Pool style constraints) or if the ESXi hosts are oversubscribed (i.e. high %rdy time) which could affect CPU scheduling, but more than likely if this is happening at 3AM I would also check into when the snapshots are happening. In line with that, confirm that the time sync on the ESXi hosts is healthy as that can cause some odd behavior as noted in my post here on VMTN yesterday.
While still on the snapshot topic, you can also review the time it takes for the final snapshot to commit which can be observed via an ssh session to the ESXi host. Simply cd to the directory where the VM is registered and perform a 'grep Checkpoint_Unstun ./vmware.log'. This will return a list in microseconds. Copy / Paste that into Excel and use text to columns then convert microseconds, to seconds, to minutes. Review any times where the VM is stunned for an extended period of time and that may explain the discrepancy at 3AM.
Other factors could be Qos or Storage Contention / activities at that time. Other things you can consider is trying to get a VM outside of vCloud or otherwise removed from normal maintenance activities. Of course, ensure that VMware Tools is healthy, scrutinize virtual hardware choices / revisions, and all that good stuff.
Anyway, sounds like a cool project. I would be happy to help out if you need anything. Ping me via IM here at VMTN, or catch me on the twitters. All the best.
