Storage I/O during night, what is happening?

[Frans_P]

Hi folks,

We have some SAN issues and i'm busy troubleshooting what's going on. I need some extra brainpower so that's why i'm here

During the night i see heavy I/O to our HP EVA controllers, especially the value 'read requests'. This is beeing probably being caused by our backup (vStorage) but when i shut down the server and no backup is running, the value's keep on beeing high, like 4 times higher than during normal daytime when production runs.

Anybody any idea what is causing this load?

Could it be too many snapshots? Something else?

Best Regards,

Joris Kemperman

[AWo]

Could it be too many snapshots? Something else?

Are you doing snapshots at night? You should know that....

Or do you think that the snapshots just sitting there cause the load?

AWo

VCP 3 & 4

\[:o]===\[o:]

=Would you like to have this posting as a ringtone on your cell phone?=

=Send "Posting" to 911 for only $999999,99!=

[AntonVZhbankov]

Defragmentation?

---

MCSA, MCTS, VCP, VMware vExpert '2009

http://blog.vadmin.ru

[Frans_P]

Well, our backup is basted on vStorage/VCB which uses snapshot technology to perform it's backup.

I DID however removed some other snapshots aswell, and two servers had 6 snapshots on them running. I removed them one by one. At one stage VCenter reported it was done removing the snapshots and i stopped our backup, so in my opinion the utilization on our SAN should be low. Could it be that VCenter reports the snapshot has been removed but in the background the ESX server is still busy removing?

[PaulSvirin]

Well, what are the VMs that keep on running on ESXi ?

Maybe you can determine the top I/O process by using iostat?

---

Paul Svirin

StarWind Software developer ( )

[AWo]

But you shut down the backup server and the load still seems to be there? Did I got that right?

When you shut down the backup server, no process should be started regarding VCB which creates snapshots.

AWo

VCP 3 & 4

\[:o]===\[o:]

=Would you like to have this posting as a ringtone on your cell phone?=

=Send "Posting" to 911 for only $999999,99!=

[vmroyale]

Hello.

I like Anton's way of thinking. Defrags, AV Scans, bulk copies from scheduled tasks or any number of "normal" server activities could be causing this spike. This can be especially true, if you have a common template that you deploy from that may have scheduled tasks that will run concurrently across multiple virtual machines. You could verify if it is the virtual machines by running esxtop on each host, press "d" and then "v" and look at the CMDS/s value to see if any individual VM is generating a lot of IO. If you find anything of interest, then it's just a matter of determining what the VM(s) are actually doing.

Good Luck!

[Frans_P]

Thanks guys, i'll take a look at this tomorrow!