vob.iscsi.connection.stopped

[MTrachy]

We recently had an issue where a database server started to have difficulties in response time as well as database corruption.  We moved the server to a different host and all was well.  Below is a sample of the vobd.log.  Suggestions would be appreciated.

2013-09-04T14:50:48.050Z: [iscsiCorrelator] 3077466054022us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T17

2013-09-04T14:51:15.100Z: [iscsiCorrelator] 3077493103305us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:51:19.116Z: [iscsiCorrelator] 3077497119011us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:51:32.945Z: [iscsiCorrelator] 3077510948268us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:51:37.083Z: [iscsiCorrelator] 3077515085705us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:51:52.098Z: [iscsiCorrelator] 3077530100977us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:51:57.890Z: [iscsiCorrelator] 3077535893267us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:52:02.218Z: [scsiCorrelator] 3077540221050us: [vob.scsi.device.io.latency.high] Device naa.68b7b2ccd680d40cafff141e04c16361 performance has deteriorated. I/O latency increased from average value of 6021 microseconds to 168061 microseconds.

2013-09-04T14:52:02.218Z: [scsiCorrelator] 3077566046440us: [esx.problem.scsi.device.io.latency.high] Device naa.68b7b2ccd680d40cafff141e04c16361 performance has deteriorated. I/O latency increased from average value of 6021 microseconds to 168061 microseconds.

2013-09-04T14:52:48.504Z: [scsiCorrelator] 3077586506612us: [vob.scsi.device.io.latency.improved] Device naa.68b7b2ccd680d40cafff141e04c16361 performance has improved. I/O latency reduced from 168061 microseconds to 32566 microseconds.

2013-09-04T14:53:10.501Z: [iscsiCorrelator] 3077608503212us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T13

2013-09-04T14:53:16.652Z: [iscsiCorrelator] 3077614653954us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T13

2013-09-04T14:54:16.619Z: [iscsiCorrelator] 3077674620576us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T17

2013-09-04T14:54:25.450Z: [iscsiCorrelator] 3077683451629us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T17

2013-09-04T14:54:26.619Z: [iscsiCorrelator] 3077684620524us: [vob.iscsi.session.recovery.timeout] iScsi session recovery timeout for vmhba37:C1:T17:

2013-09-04T14:54:26.619Z: [scsiCorrelator] 3077684620643us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba37:C1:T17:L0 changed state from on

2013-09-04T14:54:26.621Z: [scsiCorrelator] 3077710448754us: [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device naa.68b7b2ccd680b423afffd41e04c10395. Path vmhba37:C1:T17:L0 is down. Affected datastores: "SAN-XXXXX1-TempDB".

2013-09-04T14:54:35.055Z: [scsiCorrelator] 3077693057065us: [vob.scsi.device.io.latency.improved] Device naa.68b7b2ccd680d40cafff141e04c16361 performance has improved. I/O latency reduced from 32566 microseconds to 11964 microseconds.

2013-09-04T14:54:39.280Z: [iscsiCorrelator] 3077697281692us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T17

2013-09-04T14:54:40.182Z: [iscsiCorrelator] 3077698183352us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C2:T17

2013-09-04T14:54:41.750Z: [scsiCorrelator] 3077699751657us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba37:C1:T17:L0 changed state from dead

2013-09-04T14:54:41.752Z: [scsiCorrelator] 3077725579801us: [esx.clear.storage.redundancy.restored] Path redundancy to storage device naa.68b7b2ccd680b423afffd41e04c10395 (Datastores: "SAN-XXXXX1-TempDB") restored. Path vmhba37:C1:T17:L0 is active again.

2013-09-04T14:54:41.754Z: [scsiCorrelator] 3077699755555us: [vob.scsi.scsipath.add] Add path: vmhba37:C2:T17:L0

2013-09-04T14:54:41.755Z: [scsiCorrelator] 3077699756832us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba37:C2:T17:L0 changed state from dead

2013-09-04T14:55:08.867Z: [iscsiCorrelator] 3077726868145us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C2:T17

2013-09-04T14:55:09.146Z: [iscsiCorrelator] 3077727147636us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T17

2013-09-04T14:55:09.458Z: [scsiCorrelator] 3077727459260us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba37:C1:T17:L0 changed state from on

2013-09-04T14:55:09.460Z: [scsiCorrelator] 3077753287667us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.68b7b2ccd680b423afffd41e04c10395 degraded. Path vmhba37:C1:T17:L0 is down. Affected datastores: "SAN-XXXXX1-TempDB".

2013-09-04T14:55:12.341Z: [iscsiCorrelator] 3077730342491us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C2:T17

2013-09-04T14:55:16.986Z: [iscsiCorrelator] 3077734987369us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:55:17.017Z: [scsiCorrelator] 3077735018576us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba37:C1:T15:L0 changed state from on

2013-09-04T14:55:17.019Z: [scsiCorrelator] 3077760847010us: [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device naa.68b7b2ccd680a419afff741e04c1e37d. Path vmhba37:C1:T15:L0 is down. Affected datastores: "SAN-XXXXX1-DB".

2013-09-04T14:55:17.021Z: [scsiCorrelator] 3077735022657us: [vob.scsi.scsipath.remove] Remove path: vmhba37:C1:T15:L0

2013-09-04T14:55:17.537Z: [iscsiCorrelator] 3077735538334us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:55:19.145Z: [scsiCorrelator] 3077737146265us: [vob.scsi.scsipath.add] Add path: vmhba37:C1:T15:L0

2013-09-04T14:55:19.146Z: [scsiCorrelator] 3077737147554us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba37:C1:T15:L0 changed state from dead

2013-09-04T14:55:19.148Z: [scsiCorrelator] 3077762976137us: [esx.clear.storage.redundancy.restored] Path redundancy to storage device naa.68b7b2ccd680a419afff741e04c1e37d (Datastores: "SAN-XXXXX1-DB") restored. Path vmhba37:C1:T15:L0 is active again.

2013-09-04T14:55:19.530Z: [iscsiCorrelator] 3077737531012us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C2:T17

2013-09-04T14:55:19.530Z: [scsiCorrelator] 3077737531266us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba37:C2:T17:L0 changed state from on

2013-09-04T14:55:19.531Z: [scsiCorrelator] 3077763359284us: [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device naa.68b7b2ccd680b423afffd41e04c10395. Path vmhba37:C2:T17:L0 is down. Affected datastores: "SAN-XXXXX1-TempDB".

2013-09-04T14:55:20.277Z: [scsiCorrelator] 3077738278040us: [vob.scsi.scsipath.remove] Remove path: vmhba37:C2:T17:L0

2013-09-04T14:55:20.277Z: [scsiCorrelator] 3077738278454us: [vob.scsi.scsipath.remove] Remove path: vmhba37:C1:T17:L0

2013-09-04T14:55:27.690Z: [iscsiCorrelator] 3077745691660us: [vob.iscsi.target.connect.error] vmhba37 @ vmk3 failed to login to iqn.2001-05.com.equallogic:8-cb2b76-23b480d6c-9503c1041ed4ffaf-alln-XXXXX1-tempdb because of a network connection failure.

2013-09-04T14:55:27.691Z: [iscsiCorrelator] 3077771518656us: [esx.problem.storage.iscsi.target.connect.error] Login to iSCSI target iqn.2001-05.com.equallogic:8-cb2b76-23b480d6c-9503c1041ed4ffaf-alln-XXXXX1-tempdb on vmhba37 @ vmk3 failed. The iSCSI initiator could not establish a network connection to the target.

2013-09-04T14:56:00.961Z: [iscsiCorrelator] 3077778962332us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:56:04.540Z: [iscsiCorrelator] 3077782540548us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T17

2013-09-04T14:56:10.303Z: [iscsiCorrelator] 3077788304214us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:56:10.962Z: [iscsiCorrelator] 3077788962865us: [vob.iscsi.session.recovery.timeout] iScsi session recovery timeout for vmhba37:C1:T15:

2013-09-04T14:56:10.968Z: [scsiCorrelator] 3077788969087us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba37:C1:T15:L0 changed state from on

2013-09-04T14:56:10.970Z: [scsiCorrelator] 3077814798129us: [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device naa.68b7b2ccd680a419afff741e04c1e37d. Path vmhba37:C1:T15:L0 is down. Affected datastores: "SAN-XXXXX1-DB".

2013-09-04T14:56:12.059Z: [scsiCorrelator] 3077790060209us: [vob.scsi.scsipath.add] Add path: vmhba37:C1:T17:L0

2013-09-04T14:56:12.061Z: [scsiCorrelator] 3077790061596us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba37:C1:T17:L0 changed state from dead

2013-09-04T14:56:12.062Z: [scsiCorrelator] 3077815890033us: [esx.clear.storage.redundancy.restored] Path redundancy to storage device naa.68b7b2ccd680b423afffd41e04c10395 (Datastores: "SAN-XXXXX1-TempDB") restored. Path vmhba37:C1:T17:L0 is active again.

2013-09-04T14:56:14.069Z: [scsiCorrelator] 3077792069774us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba37:C1:T15:L0 changed state from dead

2013-09-04T14:56:14.070Z: [scsiCorrelator] 3077817898430us: [esx.clear.storage.redundancy.restored] Path redundancy to storage device naa.68b7b2ccd680a419afff741e04c1e37d (Datastores: "SAN-XXXXX1-DB") restored. Path vmhba37:C1:T15:L0 is active again.

2013-09-04T14:56:14.242Z: No correlator for vob.vmfs.resource.corruptondisk

2013-09-04T14:56:14.242Z: [vmfsCorrelator] 3077818070547us: [esx.problem.vmfs.resource.corruptondisk] 4ffaf2e3-7f7a77d7-8c72-441ea160a3ec SAN-XXXXX1-TempDB

2013-09-04T14:56:14.260Z: [vmfsCorrelator] 3077818088470us: [esx.problem.vmfs.resource.corruptondisk] 4ffaf2e3-7f7a77d7-8c72-441ea160a3ec SAN-XXXXX1-TempDB

2013-09-04T14:56:14.260Z: No correlator for vob.vmfs.resource.corruptondisk

2013-09-04T14:56:28.365Z: [iscsiCorrelator] 3077806365450us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:56:28.365Z: [scsiCorrelator] 3077806365719us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba37:C1:T15:L0 changed state from on

2013-09-04T14:56:28.366Z: [scsiCorrelator] 3077832194452us: [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device naa.68b7b2ccd680a419afff741e04c1e37d. Path vmhba37:C1:T15:L0 is down. Affected datastores: "SAN-XXXXX1-DB".

2013-09-04T14:56:28.378Z: [scsiCorrelator] 3077806378393us: [vob.scsi.scsipath.remove] Remove path: vmhba37:C1:T15:L0

2013-09-04T14:56:34.170Z: [iscsiCorrelator] 3077812171209us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C1:T15

2013-09-04T14:56:39.923Z: [iscsiCorrelator] 3077817923398us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C0:T7

2013-09-04T14:56:39.923Z: [scsiCorrelator] 3077817923629us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba37:C0:T7:L0 changed state from on

2013-09-04T14:56:39.924Z: [scsiCorrelator] 3077843752256us: [esx.problem.storage.redundancy.lost] Lost path redundancy to storage device naa.68b7b2cc36b1a08e42fa6406000020d6. Path vmhba37:C0:T7:L0 is down. Affected datastores: "SAN-XXXXX2".

2013-09-04T14:56:40.675Z: [scsiCorrelator] 3077818675840us: [vob.scsi.scsipath.remove] Remove path: vmhba37:C0:T7:L0

2013-09-04T14:56:42.315Z: [iscsiCorrelator] 3077820315474us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba37:C0:T7

2013-09-04T14:56:42.811Z: [scsiCorrelator] 3077820811361us: [vob.scsi.scsipath.add] Add path: vmhba37:C0:T7:L0

2013-09-04T14:56:42.812Z: [scsiCorrelator] 3077820812566us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba37:C0:T7:L0 changed state from dead

2013-09-04T14:56:42.813Z: [scsiCorrelator] 3077846641586us: [esx.clear.storage.redundancy.restored] Path redundancy to storage device naa.68b7b2cc36b1a08e42fa6406000020d6 (Datastores: "SAN-XXXXX2") restored. Path vmhba37:C0:T7:L0 is active again.

[OscarDavey]

Best thing to start with is to check the stability and performance of the storage in the first host , you are having an hdd failure . i suggest that u check them up first .

hope that helped .

Best regards

Your Oscar