VMware Cloud Community
kcucadmin
Enthusiast
Enthusiast

ESX 4 update 1 slow storage vmotion and snapshots

was wondering if anyone could shed some light on somthing i'm seeing.

When i vmotion VM's from one host to another using vCenter, even running hosts, they vmotion very quickly. most in less than 30 secs.

when i delete all snapshots or storage vmotion to different data stores, it takes much longer. I understand it's movign more data and watch the folders. the files all seam to be moved to the new target, or the snapshot files seam to be built back into the root vmdk, but the jobs just hang anywhere from 70-90% for several minuets with no real disk activity that i can see.

any idea what clean up tasks may be going on here that are taking so long?

some background info.

all hosts are brand new ESX4 installs with update 1 applied.

all hosts have dedicated nics for storage and vmotion.

almost ALL VM's are built out on NFS Datastores on a Celerra NX4. however i do see these pauses on iscsi data stores as well.

VCenter Server is latest build and running in a VM on a host.

I have 8 hosts, in two HA Clusters. one with drs and one without. alll hosts have new Nahleam Processors with 36gig ram

I've been monitoring Network Performance, and I'm not maxing any of the 1gige nics, at the esx host side or the nx4 side.

any suggestions on areas to look at? or is this just Normal? My gut tells me somthing is timing out here, then the job completes, but i'm not sure, and ESX logs still are a complete mystry to me.

Please forgive me, this is litteraly only Month 2 of my exposure to ESX, and week 3 of production =D.

any help is greatly appreciated, and i will award points for suggestions.

thanks.

Reply
0 Kudos
5 Replies
weinstein5
Immortal
Immortal

How large are the virtual disks you are moving?

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
Reply
0 Kudos
kcucadmin
Enthusiast
Enthusiast

honestly not very large, very new environment most at this point are just the OS and application files. anywhere from 5GB-20GB right now.

all my citrix servers are basiclly 10GB used right now. That's what's got me puzzled. litterly when i look at the target data store it looks like the file has been moved. it's not growing in size any longer, but i dont know if VMWare allocates the space first because of the NFS(thin provisioning) then back fills the data? and if so, 10GB should still go pretty quick with 15k sas disks on a NX4. some of my vmotion jobs take over an HOUR to complete and that's 9-10gb data.

guess what i'm trying to determine if that's normal, or do i have a kink i need to chase down.

i should note that the source and target are on different File Systems (LUNS) using different storage pools. so i dont think it's a source and target on same spindel kind of thing.

again, i'm seeing this behavor with Storage VMotion and SNAPS, also with the Data Recovery backups, extremly slow.

EMC swears the NX4 is fine, and that the storage is not the bottle neck. but nobody there seams to really know why vmware is taking it's time moving the files or removing snapshots.

Reply
0 Kudos
jfelinski
Enthusiast
Enthusiast

I'll start by looking into Host performance statistics and Disk/Command latency + aborted commands. Check out following chart during sVmotion or backup

Performance->advanced->Disk->Real Time, Disk Write Latency, Disk Read Latency, Disk Command Aborts, Disk Queue Command Latency






---

MCSA+S, VCP 3, VCP 4

http://wirtualizacja.wordpress.com[/url]

--- MCSA+S, VCP 3, VCP 4, vExpert [url=http://wirtualizacja.wordpress.com]http://wirtualizacja.wordpress.com[/url]
Reply
0 Kudos
kcucadmin
Enthusiast
Enthusiast

ok this is going to sound stupid, but when i got to setup those performance charts, i only see iSCSI VMFS Data Stores, not any of my NFS Data Stores, does esx see those as DISKS?

Reply
0 Kudos
kcucadmin
Enthusiast
Enthusiast

Relocate virtual machine

NGTS01

Completed

KCUC\rsamples

vs-vcenter02.kcuc.local

12/16/2009 8:28:23 AM

12/16/2009 8:28:23 AM

12/16/2009 8:47:58 AM

That was Migrating from one data Store to another only 8gb of data. 20mins. is that normal? seams like it shouldn't take that long.

not that 20mins is bad, i'm just concerned when these servers get to be 50-60gb that they will take HOURS to move.

Reply
0 Kudos