3 Replies Latest reply on Nov 23, 2012 12:31 PM by sparrowangelstechnology

    Disk I/O Latency increase

    Roberba Lurker

      2-5 times a day I get a warning in the events log that looks like this...

       

      Device naa.600508b1001c2be40929c423819e3359 performance has
      deteriorated. I/O latency increased from average value of 5627
      microseconds to 235837 microseconds.
      warning

      It's always the same device. This is running on an HP ML350 G6 and a RAID 1 configuration. 20GB RAM. Three VM guests.

      I am using the ESXi 5.1 bits downloaded from HP.

      We have replaced the System board, Drive Backplane and the connecting cable. The next option is to start replacing the hard drives.

      Is there a configuration change I should be making.

      I'm very new to this technology, so be kind

       

      Thanks

        • 1. Re: Disk I/O Latency increase
          depping Champion
          VMware EmployeesUser Moderators

          Other then this message are you experiencing issues?

           

          235ms latency seems extremely high, are there any jobs running in the guest VMs that could cause this?

          • 2. Re: Disk I/O Latency increase
            Roberba Lurker

            There does not seem to be any other processes running at that time with the exception of a AV definition update, but the update seems to occure a couple seconds after the i/o issue shows up. They could be related and I have contacted the vendor about tuning the update.  I made the change per thier request, but as I check the events in the vShpere client I see 7 i/o latency warning entries spanning from 1:12:34 AM - 1:13:05 AM.

            • 3. Re: Disk I/O Latency increase
              sparrowangelstechnology Master

              Roberba wrote:

               

              There does not seem to be any other processes running at that time with the exception of a AV definition update, but the update seems to occure a couple seconds after the i/o issue shows up. They could be related and I have contacted the vendor about tuning the update.  I made the change per thier request, but as I check the events in the vShpere client I see 7 i/o latency warning entries spanning from 1:12:34 AM - 1:13:05 AM.

              change the av update timr from 1:12am to something like 4am and see if your latency moves,

              but if its happening 2-5 times a day its something else...

              dont think its a hardwaare issue probably disk activity.

               

              run esxtop, go to the storage view and add your reads and writes to get your iops.