VMware Cloud Community
ziftech
Contributor
Contributor
‎09-30-2013 12:06 AM

Datastore missing

Please help!

I 've started copying some files in datastore to another location with datastore browser. Everything was fine, but then some issues arised and datastore become inaccessible, also all VM...

Here some logs:

hostd

2013-09-29T18:38:48.712Z [3D3E2B90 info 'Libs' opID=A5004DDC-000000ED] FileIOErrno2Result: Unexpected errno=22, Invalid argument

2013-09-29T18:38:48.712Z [3D3E2B90 info 'Libs' opID=A5004DDC-000000ED] OBJLIB-FILEBE : FileBEQueue: could not queue the operation on '/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-flat.vmdk' : Invalid argument (1441794)

2013-09-29T18:38:48.712Z [3D3E2B90 info 'DiskLib' opID=A5004DDC-000000ED] DISKLIB-PARALLELSSPARSE: ObjLib_Queue at offset=0, size=64 failed with error 1441801.

2013-09-29T18:38:48.712Z [3D3E2B90 info 'DiskLib' opID=A5004DDC-000000ED] DISKLIB-LINK  : "/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-flat.vmdk" : failed to open (The file specified is not a virtual disk).

2013-09-29T18:38:48.714Z [3D3E2B90 verbose 'Default' opID=A5004DDC-000000ED] [Search::GetInfo] Successfully opened disk: /vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh.vmdk

2013-09-29T18:38:48.718Z [3D3E2B90 verbose 'Default' opID=A5004DDC-000000ED] [Search::GetInfo] Successfully opened disk: /vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000001.vmdk

2013-09-29T18:38:48.718Z [3D3E2B90 info 'Default' opID=A5004DDC-000000ED] [Search::GetInfo] Failed to read disk hardware version.

2013-09-29T18:38:48.718Z [3D3E2B90 error 'Default' opID=A5004DDC-000000ED] [Search::GetInfo] Failed to read disk adapter type directly

2013-09-29T18:38:48.718Z [3D3E2B90 error 'Default' opID=A5004DDC-000000ED] [Search::GetInfo] Failed to read disk adapter type.

2013-09-29T18:38:48.729Z [3D3E2B90 info 'Libs' opID=A5004DDC-000000ED] FileIOErrno2Result: Unexpected errno=22, Invalid argument

...

2013-09-29T18:40:13.728Z [3DD9DB90 info 'Libs' opID=A5004DDC-00000121] AIOGNRC: Failed to open '/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000008-delta.vmdk' : Failed to lock the file (40003) (0x2001).

2013-09-29T18:40:13.729Z [3DD9DB90 info 'Libs' opID=A5004DDC-00000121] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000008-delta.vmdk' : Failed to lock the file (262146).

...

2013-09-29T18:46:38.233Z [3D2DEB90 info 'ha-eventmgr'] Event 1099 : Device t10.ATA_____ST3000DM0012D1CH166__________________________________Z1F0SBZ4 performance has deteriorated. I/O latency increased from average value of 4160 microseconds to 129322 microseconds.

...

2013-09-29T18:54:08.022Z [3DEEBB90 info 'Libs' opID=A5004DDC-0000045F] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/test-redmine/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000001-flat.vmdk' : Failed to lock the file (262146).

2013-09-29T18:54:08.022Z [3DEEBB90 info 'DiskLib' opID=A5004DDC-0000045F] DISKLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/test-redmine/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000001-flat.vmdk': Failed to lock the file (40002)

2013-09-29T18:54:08.022Z [3DEEBB90 info 'DiskLib' opID=A5004DDC-0000045F] DISKLIB-LINK  : "/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/test-redmine/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000001-flat.vmdk" : failed to open (Failed to lock the file).

2013-09-29T18:54:08.093Z [3DEEBB90 verbose 'Default' opID=A5004DDC-0000045F] [Search::GetInfo] Successfully opened disk: /vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/test-redmine/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000001.vmdk

2013-09-29T18:54:12.008Z [3DEEBB90 info 'Libs' opID=A5004DDC-0000045F] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/test-redmine/bitnami-redmine-2.3.1-1-ubuntu-12.04fresh-000007-flat.vmdk' : Failed to lock the file (262146).

2013-09-29T18:54:12.008Z [3DEEBB90 info 'DiskLib' opID=A5004DDC-0000045F] DISKLIB-DSCPTR: DescriptorDetermineType: failed to open

...

2013-09-29T18:54:45.475Z [3DC87B90 info 'ha-eventmgr'] Event 1109 : Lost access to volume 507dc398-dd0fe102-6b03-c86000be4670 (datastore1) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

2013-09-29T18:55:01.245Z [3DCC8B90 verbose 'SoapAdapter'] Responded to service state request

2013-09-29T18:55:49.337Z [3D2DEB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root

2013-09-29T18:56:13.126Z [3DD9DB90 verbose 'DvsManager'] PersistAllDvsInfo called

2013-09-29T18:56:52.999Z [3E0F2B90 verbose 'Hostsvc::DatastoreSystem'] StorageApdUpdate: got Storage APD message [N11HostdCommon24VmkernelUpdateStorageApdE:0xffecd628] timestamp=18990789192748 updated=t10.ATA_____ST3000DM0012D1CH166__________________________________Z1F0SBZ4 eventtype=1 devicetype=1 with identifier t10.ATA_____ST3000DM0012D1CH166__________________________________Z1F0SBZ4

2013-09-29T18:56:52.999Z [3E0F2B90 verbose 'Hostsvc::DatastoreSystem'] StorageApdUpdate: Received APD_START

2013-09-29T18:56:52.999Z [FFAC2D20 verbose 'StorageSystem'] StoragePathUpdate: Invalidate Path Cache immediately

2013-09-29T18:56:52.999Z [3E0F2B90 verbose 'FSVolumeProvider'] SetAPDStatus: Added 507dc398-dd0fe102-6b03-c86000be4670 to apd start list

2013-09-29T18:56:52.999Z [FFAC2D20 verbose 'StorageSystem'] SendStorageInfoEvent() called

2013-09-29T18:56:52.999Z [3E0F2B90 verbose 'FSVolumeProvider'] RefreshOneVmfsVolume on /vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670

2013-09-29T18:56:52.999Z [3E0F2B90 verbose 'FSVolumeProvider'] RefreshOneVmfsVolume 507dc398-dd0fe102-6b03-c86000be4670 calling ProcessVmfs

...

2013-09-29T18:56:52.999Z [3E0F2B90 verbose 'FSVolumeProvider'] DeleteApdStarted: Clearing volume 507dc398-dd0fe102-6b03-c86000be4670 from APD Started list

2013-09-29T18:56:53.000Z [3E0F2B90 verbose 'FSVolumeProvider'] ProcessVmfs: Inaccessible datastore 507dc398-dd0fe102-6b03-c86000be4670, storage t10.ATA_____ST3000DM0012D1CH166__________________________________Z1F0SBZ4, reason APD Start

2013-09-29T18:56:53.000Z [3D2DEB90 info 'ha-eventmgr'] Event 1110 : Device or filesystem with identifer [t10.ATA_____ST3000DM0012D1CH166__________________________________Z1F0SBZ4] has entered the All Paths Down state.

2013-09-29T18:56:53.000Z [3E0F2B90 verbose 'Hostsvc::Datastore'] NotifyIfCapacityChanged:: -- notify that datastore 507dc398-dd0fe102-6b03-c86000be4670 now has capacity 0

2013-09-29T18:56:53.000Z [3E0F2B90 verbose 'Hostsvc::Datastore'] NotifyIfAccessibleChanged -- notify that datastore 507dc398-dd0fe102-6b03-c86000be4670 at path /vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670 now has accessibility of false due to AllPathsDown_Start

2013-09-29T18:56:53.000Z [3DD9DB90 warning 'VmkVprobSource'] Can't find datastore ''.

...

2013-09-29T18:56:53.001Z [3D31FB90 verbose 'vm:/vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670/winplaut2012/winplaut2012.vmx'] Got DSSYS change: [N11HostdCommon18DatastoreSystemMsgE:0xffd2e5d0]/vmfs/volumes/, 507dc398-dd0fe102-6b03-c86000be4670, /vmfs/volumes/507dc398-dd0fe102-6b03-c86000be4670;

2013-09-29T18:56:53.001Z [3DD9DB90 info 'ha-eventmgr'] Event 1111 : Lost connectivity to storage device t10.ATA_____ST3000DM0012D1CH166__________________________________Z1F0SBZ4. Path vmhba0:C0:T0:L0 is down. Affected datastores: "", "datastore1".

2013-09-29T18:56:53.001Z [3E0F2B90 verbose 'Hostsvc::DatastoreSystem'] DatastoreSystemImpl.UpdateSystemSwapConfigIssue : System swap is not active

2013-09-29T18:56:53.002Z [3DE69B90 error 'Default' opID=CFE526C3-0000011B] Failed to send response to the client: Broken pipe

from this moment I've got

2013-09-29T18:59:13.003Z [3D678B90 info 'ha-eventmgr'] Event 1114 : Device or filesystem with identifer [t10.ATA_____ST3000DM0012D1CH166__________________________________Z1F0SBZ4] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will be fast failed.

How to repair broken datastore???

Please help!

Tags (2)
0 Kudos
0 Replies