Storage

Hi All,

We have two ESX servers 1 and 2 which has a common shared storage on which VMs reside(Storage is overland snap box). ESX 1 has two windows servers, exchange and terminal services server. ESX 2 has file server. Present issue is access to shares on file server is very slow.

Observations:

++ event logs on all 3 severs are filled with event id 11 and 15 related to disk

++ windows servers get rebooted by themselves atleast once in a week not sure if it can be related to access speed of file server

++ access to shares on file server is very slow for users

++ QB software is also installed on file server and users feel very slow access to this one too

Steps taken to improve performance of file server:

++ memory was upgraded to 3 GB

++ separate disk was added for swap

++ disk defragmentation was done on all the drives using Defragler. The performance had improved a bit after this step however, looks like it’s back to same thing again

No relevant logs found on ESX servers 1 and 2 and also on storage device. Any help is much appreciated.

Before changing too much on your config, try to pinpoint the bottle neck. Windows eventide 11 and 15 are pointing towards storage from a vm point, but there are many layers between de guest os disk and the physical disks in your storage. Important tools to work with are esxtop, vscsistats and vmkernel logs.

I'm surprised you see no strange activity in your vmkernel logs because they should normally show when there are disk problems. What type of storage are you using and how is it connected? Do the errors in windows show in certain time frames or throughout the day?

http://www.GabesVirtualWorld.com

Hi Gabrie,

Below are the answers:

What type of storage are you using and how is it connected - Storage is a product from Overland, model - 520. the unit uses SATA HDDs. Its connected to network switch and mounted on ESX as nfs

Do the errors in windows show in certain time frames or throughout the day? - No, disk errors show only during high I/O activities like when backups going on, during exchange store maintenace window, etc. However, access to file server is always slow.

I suspect you have some performance problems with your storage backend. Please check with perfmon if there are any constant quetes Localdisk/AvgDiskQueueLength, and please also check on your storage systems performance tools if you can find any relevant info about the performance.

Lars

Hi Larstr,

I will check things mentioned by you and get back with results. Thank you.

Hi Larstr,

Perfmon results show that resources are really used to full extent. Below are the results:

Object: Memory

Counter: Pages / sec

Result: frequently varies between 25 to 100 %

Object: Physical disk

Counter: Ag. disk queue length

Result: Constantly at 100%, occasionally comes down to 90 %

Object: Processor

Counter: % processor time

Result: keeps on varying between 4 to 100 %

I don't think you're interpreting those graphs correctly as they are not give in %.

However, on a system with good performance Pages/sec should be 0 and AvgDiskQueueLength should be close to 0.

If you're having lots of pages/sec it indicates that the guest VM doesn't have enough ram allocated.

If you're AvgDiskQueueLength is high (more than 1 or 2) it's an indication that your storage backend can't keep up with the IO generated from your storage connected systems. You may want to move the VM in question to a different LUN or a different RAID level if possible.

Lars

Log in to esx console and check /var/log/vmkernel. Can't believe there

are no errors in it

Sent from my iPhone

On May 5, 2010, at 12:37, vgovani <communities-emailer@vmware.com

Hi Gabrie,

A VM got rebooted again today on its own. I found below logs which match the time of reboot in /var/log/vmkernel. Please help me understand this and if these logs make any sense:

Jun 9 00:56:10 vmhead1 vmkernel: 274:21:56:35.799 cpu0:1152)VSCSI: 1897: Reset request on handle 8217 (2 outstanding commands)

Jun 9 00:56:10 vmhead1 vmkernel: 274:21:56:35.799 cpu1:1043)VSCSI: 2103: Resetting handle 8217

Jun 9 00:56:10 vmhead1 vmkernel: 274:21:56:35.998 cpu0:1043)VSCSI: 1946: Completing reset on handle 8217 (0 outstanding commands)

Jun 9 00:56:10 vmhead1 vmkernel: 274:21:56:36.124 cpu1:1152)VSCSI: 1897: Reset request on handle 8217 (0 outstanding commands)

Jun 9 00:56:10 vmhead1 vmkernel: 274:21:56:36.125 cpu0:1043)VSCSI: 2103: Resetting handle 8217

Jun 9 00:56:10 vmhead1 vmkernel: 274:21:56:36.125 cpu0:1043)VSCSI: 1946: Completing reset on handle 8217 (0 outstanding commands)

Jun 9 00:56:16 vmhead1 vmkernel: 274:21:56:42.228 cpu1:1065)NFSLock: 514: Stop accessing fd 0x5a1953c 4

Jun 9 00:56:16 vmhead1 vmkernel: 274:21:56:42.228 cpu1:1065)NFSLock: 514: Stop accessing fd 0x5a1c1dc 4