Warning: NMP: nmp_DeviceRequestFastDeviceProbe

[_rbn_]

Hi guys.

I am fairly new to vmware, but I have been really happy with it running my server for the last year.

Now I dont know if this error has been there very since the start, or if it is new. But I am getting loads of warnings in my vmkwarning.log file.

My setup:

HP Dl320e G8
Adaptec 6445

Intel Expander RES2SV240

4 SAS enclosures with 4 WD Red 4TB drives per enclosure (raid 6 over 8 disks, 2 in every enclosure, so 8 slots are still empty).

Nothing fancy, but is great for the purpose.

Now to the not so great part, this is how the log looks.

naa.5001e677b9e6cffd is the Intel Expander.

mpx.vmhba2:C0:T1:L0 is the Raid 6

mpx.vmhba2:C0:T0:L0 are two SSDs in the HP slots with RAID 1

mpx.vmhba2:C0:T2:L0 are the other two slots in the HP with two 1TB drives in Raid1

Ok, So a part or the log... and this goes on and on:

2015-01-22T17:22:34.113Z cpu1:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T1:L0" has bad pagecode: 0x0

2015-01-22T17:22:34.126Z cpu1:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T0:L0" has bad pagecode: 0x0

2015-01-22T17:22:34.129Z cpu1:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T2:L0" has bad pagecode: 0x0

2015-01-22T17:22:34.239Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:22:56.989Z cpu3:32788)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:27:56.990Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:32:56.990Z cpu0:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:37:56.991Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:42:56.990Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:47:56.990Z cpu0:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:52:34.800Z cpu1:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T1:L0" has bad pagecode: 0x8

2015-01-22T17:52:34.828Z cpu1:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T0:L0" has bad pagecode: 0xd

2015-01-22T17:52:34.832Z cpu1:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T2:L0" has bad pagecode: 0x5

2015-01-22T17:52:34.943Z cpu2:32787)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:52:56.990Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T17:57:56.990Z cpu3:32788)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:02:56.990Z cpu0:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:07:56.990Z cpu2:32787)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:12:56.991Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:17:56.990Z cpu0:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:22:35.506Z cpu2:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T1:L0" has bad pagecode: 0x10

2015-01-22T18:22:35.530Z cpu2:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T0:L0" has bad pagecode: 0x0

2015-01-22T18:22:35.534Z cpu2:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T2:L0" has bad pagecode: 0x2b

2015-01-22T18:22:35.645Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:22:35.980Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:22:56.990Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:27:56.992Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:32:56.990Z cpu2:32787)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:37:56.990Z cpu3:32788)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:42:56.990Z cpu2:32787)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:47:56.990Z cpu2:32787)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:52:36.207Z cpu2:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T1:L0" has bad pagecode: 0x11

2015-01-22T18:52:36.232Z cpu2:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T0:L0" has bad pagecode: 0x0

2015-01-22T18:52:36.237Z cpu2:34168)WARNING: ScsiDeviceIO: 7005: IEC page to device "mpx.vmhba2:C0:T2:L0" has bad pagecode: 0x0

2015-01-22T18:52:36.348Z cpu0:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:52:36.908Z cpu2:32787)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:52:56.991Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T18:57:56.991Z cpu1:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T19:02:56.990Z cpu2:32787)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T19:07:56.991Z cpu3:32788)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

2015-01-22T19:12:56.991Z cpu3:32788)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5001e677b9e6cffd" state in doubt; requested fast path state update...

[_rbn_]

Maybe I should add that this unit is used for backup. Backing up about 50 clients right now.

I am telling you this because of this line on vmwares info page about the error seen in the vobd.log:

Also check if these messages are generated when there were any scheduled tasks, such as backups, replications, etc., as these can also cause intermittent performance hits.

But I tried to turn the backup off, the entries in the /var/log/vmkwarning.log did continue (the aprox 10 min cycle), so I don´t have to much trust in this being just the backup running 😕 But I wanted to tell you everything I have been testing my self!

The vobd.log as well

2015-01-20T00:00:43.950Z: [scsiCorrelator] 1765367276456us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T2:L0 performance has improved. I/O latency reduced from 5301 microseconds to 4421 microseconds.

2015-01-20T00:00:43.950Z: [scsiCorrelator] 1765367277003us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T2:L0 performance has improved. I/O latency reduced from 5301 microseconds to 4421 microseconds.

2015-01-20T00:40:51.159Z: [scsiCorrelator] 1767774485074us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12197 microseconds to 393127 microseconds.

2015-01-20T00:40:51.159Z: [scsiCorrelator] 1767774485640us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12197 microseconds to 393127 microseconds.

2015-01-20T00:46:12.361Z: [scsiCorrelator] 1768095687062us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 393127 microseconds to 77498 microseconds.

2015-01-20T00:46:12.361Z: [scsiCorrelator] 1768095687609us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 393127 microseconds to 77498 microseconds.

2015-01-20T01:26:44.097Z: [scsiCorrelator] 1770527423001us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12199 microseconds to 368277 microseconds.

2015-01-20T01:26:44.097Z: [scsiCorrelator] 1770527423596us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12199 microseconds to 368277 microseconds.

2015-01-20T01:30:06.129Z: [scsiCorrelator] 1770729454694us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368277 microseconds to 73338 microseconds.

2015-01-20T01:30:06.129Z: [scsiCorrelator] 1770729455288us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368277 microseconds to 73338 microseconds.

2015-01-20T01:51:28.673Z: [scsiCorrelator] 1772011998774us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73338 microseconds to 23880 microseconds.

2015-01-20T01:51:28.673Z: [scsiCorrelator] 1772011999383us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73338 microseconds to 23880 microseconds.

2015-01-20T03:25:02.215Z: [scsiCorrelator] 1777625540512us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12216 microseconds to 371844 microseconds.

2015-01-20T03:25:02.215Z: [scsiCorrelator] 1777625541103us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12216 microseconds to 371844 microseconds.

2015-01-20T03:26:46.280Z: [scsiCorrelator] 1777729605765us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 371844 microseconds to 73316 microseconds.

2015-01-20T03:26:46.280Z: [scsiCorrelator] 1777729606326us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 371844 microseconds to 73316 microseconds.

2015-01-20T03:58:40.271Z: [scsiCorrelator] 1779643597414us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73316 microseconds to 24096 microseconds.

2015-01-20T03:58:40.272Z: [scsiCorrelator] 1779643598052us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73316 microseconds to 24096 microseconds.

2015-01-21T02:43:49.038Z: [scsiCorrelator] 1861552364053us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11838 microseconds to 357325 microseconds.

2015-01-21T02:43:49.038Z: [scsiCorrelator] 1861552364694us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11838 microseconds to 357325 microseconds.

2015-01-21T02:44:38.848Z: [scsiCorrelator] 1861602173474us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357325 microseconds to 71302 microseconds.

2015-01-21T02:44:38.848Z: [scsiCorrelator] 1861602174108us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357325 microseconds to 71302 microseconds.

2015-01-21T03:03:03.195Z: [scsiCorrelator] 1862706521047us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11839 microseconds to 389600 microseconds.

2015-01-21T03:03:03.195Z: [scsiCorrelator] 1862706521624us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11839 microseconds to 389600 microseconds.

2015-01-21T03:04:49.393Z: [scsiCorrelator] 1862812718887us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 389600 microseconds to 76579 microseconds.

2015-01-21T03:04:49.393Z: [scsiCorrelator] 1862812719508us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 389600 microseconds to 76579 microseconds.

2015-01-21T03:17:44.791Z: [scsiCorrelator] 1863588117001us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 76579 microseconds to 23479 microseconds.

2015-01-21T03:17:44.791Z: [scsiCorrelator] 1863588117577us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 76579 microseconds to 23479 microseconds.

2015-01-21T03:57:11.318Z: [scsiCorrelator] 1865954643900us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11851 microseconds to 368958 microseconds.

2015-01-21T03:57:11.318Z: [scsiCorrelator] 1865954644492us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11851 microseconds to 368958 microseconds.

2015-01-21T03:58:47.330Z: [scsiCorrelator] 1866050655796us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368958 microseconds to 72969 microseconds.

2015-01-21T03:58:47.330Z: [scsiCorrelator] 1866050656403us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368958 microseconds to 72969 microseconds.

2015-01-21T04:03:36.648Z: [scsiCorrelator] 1866339974393us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 72969 microseconds to 23507 microseconds.

2015-01-21T04:03:36.648Z: [scsiCorrelator] 1866339974994us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 72969 microseconds to 23507 microseconds.

2015-01-21T18:59:14.845Z: [scsiCorrelator] 1920078170629us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11510 microseconds to 379288 microseconds.

2015-01-21T18:59:14.845Z: [scsiCorrelator] 1920078171219us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11510 microseconds to 379288 microseconds.

2015-01-21T19:01:16.836Z: [scsiCorrelator] 1920200162422us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 379288 microseconds to 75549 microseconds.

2015-01-21T19:01:16.837Z: [scsiCorrelator] 1920200163039us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 379288 microseconds to 75549 microseconds.

2015-01-21T19:07:19.007Z: [scsiCorrelator] 1920562333444us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 75549 microseconds to 22669 microseconds.

2015-01-21T19:07:19.008Z: [scsiCorrelator] 1920562334057us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 75549 microseconds to 22669 microseconds.

2015-01-21T20:31:34.433Z: [scsiCorrelator] 1925617758608us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 365880 microseconds.

2015-01-21T20:31:34.433Z: [scsiCorrelator] 1925617759236us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 365880 microseconds.

2015-01-21T20:32:09.254Z: [scsiCorrelator] 1925652579533us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 745015 microseconds.

2015-01-21T20:32:09.254Z: [scsiCorrelator] 1925652580154us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 745015 microseconds.

2015-01-21T20:32:32.628Z: [scsiCorrelator] 1925675954357us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 745015 microseconds to 146457 microseconds.

2015-01-21T20:32:32.628Z: [scsiCorrelator] 1925675954939us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 745015 microseconds to 146457 microseconds.

2015-01-21T20:32:44.389Z: [scsiCorrelator] 1925687714790us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 358113 microseconds.

2015-01-21T20:32:44.389Z: [scsiCorrelator] 1925687715354us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 358113 microseconds.

2015-01-21T20:33:10.953Z: [scsiCorrelator] 1925714279444us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 358113 microseconds to 69812 microseconds.

2015-01-21T20:33:10.954Z: [scsiCorrelator] 1925714280043us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 358113 microseconds to 69812 microseconds.

2015-01-21T20:35:31.577Z: [scsiCorrelator] 1925854902790us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 373986 microseconds.

2015-01-21T20:35:31.577Z: [scsiCorrelator] 1925854903386us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 373986 microseconds.

2015-01-21T20:37:05.872Z: [scsiCorrelator] 1925949198202us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 373986 microseconds to 74350 microseconds.

2015-01-21T20:37:05.872Z: [scsiCorrelator] 1925949198790us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 373986 microseconds to 74350 microseconds.

2015-01-21T20:45:22.808Z: [scsiCorrelator] 1926446134219us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74350 microseconds to 22416 microseconds.

2015-01-21T20:45:22.808Z: [scsiCorrelator] 1926446134824us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74350 microseconds to 22416 microseconds.

2015-01-21T22:26:11.311Z: [scsiCorrelator] 1932494637316us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 360319 microseconds.

2015-01-21T22:26:11.311Z: [scsiCorrelator] 1932494637929us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 360319 microseconds.

2015-01-21T22:34:26.519Z: [scsiCorrelator] 1932989844615us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 360319 microseconds to 71603 microseconds.

2015-01-21T22:34:26.519Z: [scsiCorrelator] 1932989845238us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 360319 microseconds to 71603 microseconds.

2015-01-21T22:43:04.185Z: [scsiCorrelator] 1933507511075us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 357249 microseconds.

2015-01-21T22:43:04.185Z: [scsiCorrelator] 1933507511706us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 357249 microseconds.

2015-01-21T22:49:38.692Z: [scsiCorrelator] 1933902018022us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357249 microseconds to 70490 microseconds.

2015-01-21T22:49:38.692Z: [scsiCorrelator] 1933902018628us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357249 microseconds to 70490 microseconds.

2015-01-21T23:26:05.258Z: [scsiCorrelator] 1936088583913us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11528 microseconds to 355514 microseconds.

2015-01-21T23:26:05.258Z: [scsiCorrelator] 1936088584498us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11528 microseconds to 355514 microseconds.

2015-01-21T23:30:13.597Z: [scsiCorrelator] 1936336922530us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 355514 microseconds to 70830 microseconds.

2015-01-21T23:30:13.597Z: [scsiCorrelator] 1936336923129us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 355514 microseconds to 70830 microseconds.

2015-01-22T00:10:44.107Z: [scsiCorrelator] 1938767433400us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 70830 microseconds to 22576 microseconds.

2015-01-22T00:10:44.108Z: [scsiCorrelator] 1938767434023us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 70830 microseconds to 22576 microseconds.

2015-01-22T03:17:28.580Z: [scsiCorrelator] 1949971906415us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 348863 microseconds.

2015-01-22T03:17:28.581Z: [scsiCorrelator] 1949971907047us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 348863 microseconds.

2015-01-22T03:23:52.833Z: [scsiCorrelator] 1950356158615us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348863 microseconds to 68635 microseconds.

2015-01-22T03:23:52.833Z: [scsiCorrelator] 1950356159211us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348863 microseconds to 68635 microseconds.

2015-01-22T03:23:54.282Z: [scsiCorrelator] 1950357608044us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68635 microseconds to 22995 microseconds.

2015-01-22T03:23:54.282Z: [scsiCorrelator] 1950357608675us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68635 microseconds to 22995 microseconds.

2015-01-22T03:48:47.486Z: [scsiCorrelator] 1951850812258us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 376890 microseconds.

2015-01-22T03:48:47.486Z: [scsiCorrelator] 1951850812929us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 376890 microseconds.

2015-01-22T04:00:17.871Z: [scsiCorrelator] 1952541197205us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 376890 microseconds to 74290 microseconds.

2015-01-22T04:00:17.871Z: [scsiCorrelator] 1952541197837us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 376890 microseconds to 74290 microseconds.

2015-01-22T04:38:01.784Z: [scsiCorrelator] 1954805109594us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74290 microseconds to 22432 microseconds.

2015-01-22T04:38:01.784Z: [scsiCorrelator] 1954805110193us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74290 microseconds to 22432 microseconds.

2015-01-22T05:26:05.282Z: [scsiCorrelator] 1957688608008us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11536 microseconds to 348785 microseconds.

2015-01-22T05:26:05.282Z: [scsiCorrelator] 1957688608587us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11536 microseconds to 348785 microseconds.

2015-01-22T05:28:03.138Z: [scsiCorrelator] 1957806463877us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348785 microseconds to 68859 microseconds.

2015-01-22T05:28:03.138Z: [scsiCorrelator] 1957806464437us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348785 microseconds to 68859 microseconds.

2015-01-22T06:24:29.841Z: [scsiCorrelator] 1961193166618us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68859 microseconds to 22911 microseconds.

2015-01-22T06:24:29.841Z: [scsiCorrelator] 1961193167218us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68859 microseconds to 22911 microseconds.

[_rbn_]

No one any thoughts?

I also noticed that the storage behind the expander, 4 enclosures with 2 drives in each a total of 8 drives in a raid 6, is really slow!!!

I know Red drives are not that awesome in speed, but I get a average of 140 millisecons write on that array. Is that possible... can it be that slow, or does this sound like a hardware problem to you guys. The drives that are internally in the HP get average of below 1 read and just about under 3 write, so it does not sound like there is a problem with the raid card (but of course on thing to have in mind is that those two arrays are Raid1)

/rbn

[_rbn_]

Has no one any thought on this!? Please, anyone? I really need to know what these error posts are saying.

//rbn