Re: suddenly i lost my DATASTORE

waeel_ahmed · ‎05-23-2016

suddenly all my VMs hang from yesterday , for that i restart my hypervisor Esxi 6

But after that i cant run any VM or delete,copy or edit anything from datastore

I have the following :

- Server Dell R930 , 12 HDD 1.2TB , Raid 60 , 2 Group

- one datastore 8.72 TB

- Esxi 6.0.0 Build 3620759

If i tried to run VM this is the message :

Failed to start the virtual machine.

Failed to power on VM.

Could not power on virtual machine:msg.vmk.status.VMK_INVALID_METADATA.

Current swap file size is 0 KB.

Failed to extend swap file from 0 KB to 8192000 KB.

If i tried to delete a VM not in my inventory files from datastore this is the message :

Error Caused by file

/vmfs/volumes/572.............................../............

What i can do , please any support, I can't do anything

LucianoPatrão · ‎05-23-2016

Hi,

First you need to check you Datastore in that ESXi

Just connect trough ssh and check the free space etc. 8Tb is not small, so check the free space on that datastore. You need to troubleshooting why is not able to create swap files.

PS(no related to the issue): I will never create a 9Tb Datastore only. I would divided at least in 3.

Datastore troubleshooting: VMware KB: Troubleshooting a datastore or VMFS volume that is full or near capacity

One workaround to power up the VMs is to do a memory reservation of all the allocated VM memory. This will prevent the VM to create swap file. This is just a workaround to troubleshoot the issue and put the VM running. Should not be a long term solution

Check here how to: https://kb.vmware.com/kb/1030719

Then provide here the information about the datastore

Luciano Patrão

VCP-DCV, VCAP-DCV Design 2023, VCP-Cloud 2023
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard
Solutions Architect - Tech Lead for VMware / Virtual Backups

________________________________
If helpful Please award points
Thank You
Blog: https://www.provirtualzone.com | Twitter: @Luciano_PT

waeel_ahmed · ‎05-23-2016

Many Thanks for your replay.

Could you please support me what is the command for free space from ssh.

So , if i can rebuild my Esxi from the beginning ( the current data almost not important ) , what is the best way i can follow it for that , and how i can avoid the same issue in the future?

LucianoPatrão · ‎05-23-2016

Hi,

First you need to connect to host by ssh and do it manually. Mostly is using Linux commands.

Here you can find some of the commands and some of the troubleshooting that you need to do to check the datastore issues: VMware KB: Investigating disk space on an ESX or ESXi host

Check and try that and let me know

Luciano Patrão

VCP-DCV, VCAP-DCV Design 2023, VCP-Cloud 2023
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard
Solutions Architect - Tech Lead for VMware / Virtual Backups

________________________________
If helpful Please award points
Thank You
Blog: https://www.provirtualzone.com | Twitter: @Luciano_PT

impranayk · ‎05-23-2016

Check if LUNs are present on server. Run below esxcli command to list of all LUNs currently connected to ESXi host.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10149...

Check active path status of datastores.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10039...

Check free space on datastores.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10035...

Check with storage team and make sure physical connectivity or there is no such storage issue from where these datastores are zoned.

-------------------------------------------------------------------------
Follow me @ www.vmwareinsight.com
Please consider marking this answer "correct" or "helpful" if you found it useful

Pranay Jha | Blog: http://vmwareinsight.com
vExpert 2016/2017, VCAP5-DCD/DCA, VCP5-DCV, VCA-Cloud, VCE-CIA, MCSE, MCITP

waeel_ahmed · ‎05-23-2016

That is the disk space i have free around 6 TB

Filesystem Size Used Available Use% Mounted on

VMFS-5 8.7T 2.5T 6.2T 29% /vmfs/volumes/datastore1

vfat 249.7M 167.9M 81.8M 67% /vmfs/volumes/690bc47a-a400b745-1982-417 8261c043c

vfat 4.0G 39.4M 4.0G 1% /vmfs/volumes/5742895c-f76f4c3d-f3a6-141 8776ab898

vfat 249.7M 167.9M 81.8M 67% /vmfs/volumes/20fc5879-a3575a40-9190-1e4 a57af9f35

vfat 285.8M 202.6M 83.3M 71% /vmfs/volumes/5729fe80-54746626-fc7f-141 8776ab898

waeel_ahmed · ‎05-23-2016

[root@ESXI1:~] esxcli storage core path list

sas.514187706fbb3400-sas.60bc7462039a6135-naa.614187706fbb34001ebc7462039a6135

UID: sas.514187706fbb3400-sas.60bc7462039a6135-naa.614187706fbb34001ebc7462039a6135

Runtime Name: vmhba0:C2:T0:L0

Device: naa.614187706fbb34001ebc7462039a6135

Device Display Name: DELL Serial Attached SCSI Disk (naa.614187706fbb34001ebc7462039a6135)

Adapter: vmhba0

Channel: 2

Target: 0

LUN: 0

Plugin: NMP

State: active

Transport: sas

Adapter Identifier: sas.514187706fbb3400

Target Identifier: sas.60bc7462039a6135

Adapter Transport Details: 514187706fbb3400

Target Transport Details: 60bc7462039a6135

Maximum IO Size: 131072

sata.vmhba37-sata.0:0-mpx.vmhba37:C0:T0:L0

UID: sata.vmhba37-sata.0:0-mpx.vmhba37:C0:T0:L0

Runtime Name: vmhba37:C0:T0:L0

Device: mpx.vmhba37:C0:T0:L0

Device Display Name: Local PLDS CD-ROM (mpx.vmhba37:C0:T0:L0)

Adapter: vmhba37

Channel: 0

Target: 0

LUN: 0

Plugin: NMP

State: active

Transport: sata

Adapter Identifier: sata.vmhba37

Target Identifier: sata.0:0

Adapter Transport Details: Unavailable or path is unclaimed

Target Transport Details: Unavailable or path is unclaimed

Maximum IO Size: 131072

[root@ESXI1:~] esxcli storage core device list

naa.614187706fbb34001ebc7462039a6135

Display Name: DELL Serial Attached SCSI Disk (naa.614187706fbb34001ebc7462039a6135)

Has Settable Display Name: true

Size: 9152512

Device Type: Direct-Access

Multipath Plugin: NMP

Devfs Path: /vmfs/devices/disks/naa.614187706fbb34001ebc7462039a6135

Vendor: DELL

Model: PERC H730P Adp

Revision: 4.25

SCSI Level: 5

Is Pseudo: false

Status: degraded

Is RDM Capable: true

Is Local: false

Is Removable: false

Is SSD: false

Is VVOL PE: false

Is Offline: false

Is Perennially Reserved: false

Queue Full Sample Size: 0

Queue Full Threshold: 0

Thin Provisioning Status: unknown

Attached Filters:

VAAI Status: unsupported

Other UIDs: vml.0200000000614187706fbb34001ebc7462039a6135504552432048

Is Shared Clusterwide: true

Is Local SAS Device: false

Is SAS: true

Is USB: false

Is Boot USB Device: false

Is Boot Device: true

Device Max Queue Depth: 128

No of outstanding IOs with competing worlds: 32

Drive Type: logical

RAID Level: RAID6

Number of Physical Drives: 12

Protection Enabled: false

PI Activated: false

PI Type: 0

PI Protection Mask: NO PROTECTION

Supported Guard Types: NO GUARD SUPPORT

DIX Enabled: false

DIX Guard Type: NO GUARD SUPPORT

Emulated DIX/DIF Enabled: false

mpx.vmhba37:C0:T0:L0

Display Name: Local PLDS CD-ROM (mpx.vmhba37:C0:T0:L0)

Has Settable Display Name: false

Size: 0

Device Type: CD-ROM

Multipath Plugin: NMP

Devfs Path: /vmfs/devices/cdrom/mpx.vmhba37:C0:T0:L0

Vendor: PLDS

Model: DVD+-RW DS-8ABSH

Revision: LD51

SCSI Level: 5

Is Pseudo: false

Status: on

Is RDM Capable: false

Is Local: true

Is Removable: true

Is SSD: false

Is VVOL PE: false

Is Offline: false

Is Perennially Reserved: false

Queue Full Sample Size: 0

Queue Full Threshold: 0

Thin Provisioning Status: unknown

Attached Filters:

VAAI Status: unsupported

Other UIDs: vml.0005000000766d68626133373a303a30

Is Shared Clusterwide: false

Is Local SAS Device: false

Is SAS: false

Is USB: false

Is Boot USB Device: false

Is Boot Device: false

Device Max Queue Depth: 1

No of outstanding IOs with competing worlds: 32

Drive Type: unknown

RAID Level: unknown

Number of Physical Drives: unknown

Protection Enabled: false

PI Activated: false

PI Type: 0

PI Protection Mask: NO PROTECTION

Supported Guard Types: NO GUARD SUPPORT

DIX Enabled: false

DIX Guard Type: NO GUARD SUPPORT

Emulated DIX/DIF Enabled: false

[root@ESXI1:~] esxcli storage vmfs extent list

Volume Name VMFS UUID Extent Number Device Name Partition

----------- ----------------------------------- ------------- ------------------------------------ ---------

datastore1 5729fe82-2b0bfe4c-6a8f-1418776ab898 0 naa.614187706fbb34001ebc7462039a6135 3

[root@ESXI1:~] esxcli storage filesystem list

Mount Point Volume Name UUID Mounted Type Size Free

------------------------------------------------- ----------- ----------------------------------- ------- ------ ------------- -------------

/vmfs/volumes/5729fe82-2b0bfe4c-6a8f-1418776ab898 datastore1 5729fe82-2b0bfe4c-6a8f-1418776ab898 true VMFS-5 9589051359232 6821649580032

/vmfs/volumes/690bc47a-a400b745-1982-4178261c043c 690bc47a-a400b745-1982-4178261c043c true vfat 261853184 85823488

/vmfs/volumes/5742895c-f76f4c3d-f3a6-1418776ab898 5742895c-f76f4c3d-f3a6-1418776ab898 true vfat 4293591040 4252172288

/vmfs/volumes/20fc5879-a3575a40-9190-1e4a57af9f35 20fc5879-a3575a40-9190-1e4a57af9f35 true vfat 261853184 85786624

/vmfs/volumes/5729fe80-54746626-fc7f-1418776ab898 5729fe80-54746626-fc7f-1418776ab898 true vfat 299712512 87310336

[root@ESXI1:~] ls -alh /vmfs/devices/disks

total 18744344465

drwxr-xr-x 2 root root 512 May 23 11:00 .

drwxr-xr-x 15 root root 512 May 23 11:00 ..

-rw------- 1 root root 8.7T May 23 11:00 naa.614187706fbb34001ebc7462039a6135

-rw------- 1 root root 4.0M May 23 11:00 naa.614187706fbb34001ebc7462039a6135:1

-rw------- 1 root root 4.0G May 23 11:00 naa.614187706fbb34001ebc7462039a6135:2

-rw------- 1 root root 8.7T May 23 11:00 naa.614187706fbb34001ebc7462039a6135:3

-rw------- 1 root root 250.0M May 23 11:00 naa.614187706fbb34001ebc7462039a6135:5

-rw------- 1 root root 250.0M May 23 11:00 naa.614187706fbb34001ebc7462039a6135:6

-rw------- 1 root root 110.0M May 23 11:00 naa.614187706fbb34001ebc7462039a6135:7

-rw------- 1 root root 286.0M May 23 11:00 naa.614187706fbb34001ebc7462039a6135:8

-rw------- 1 root root 2.5G May 23 11:00 naa.614187706fbb34001ebc7462039a6135:9

lrwxrwxrwx 1 root root 36 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048 -> naa.614187706fbb34001ebc7462039a6135

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:1 -> naa.614187706fbb34001ebc7462039a6135:1

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:2 -> naa.614187706fbb34001ebc7462039a6135:2

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:3 -> naa.614187706fbb34001ebc7462039a6135:3

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:5 -> naa.614187706fbb34001ebc7462039a6135:5

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:6 -> naa.614187706fbb34001ebc7462039a6135:6

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:7 -> naa.614187706fbb34001ebc7462039a6135:7

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:8 -> naa.614187706fbb34001ebc7462039a6135:8

lrwxrwxrwx 1 root root 38 May 23 11:00 vml.0200000000614187706fbb34001ebc7462039a6135504552432048:9 -> naa.614187706fbb34001ebc7462039a6135:9

[root@ESXI1:~] esxcli storage core path list

sas.514187706fbb3400-sas.60bc7462039a6135-naa.614187706fbb34001ebc7462039a6135

UID: sas.514187706fbb3400-sas.60bc7462039a6135-naa.614187706fbb34001ebc7462039a6135

Runtime Name: vmhba0:C2:T0:L0

Device: naa.614187706fbb34001ebc7462039a6135

Device Display Name: DELL Serial Attached SCSI Disk (naa.614187706fbb34001ebc7462039a6135)

Adapter: vmhba0

Channel: 2

Target: 0

LUN: 0

Plugin: NMP

State: active

Transport: sas

Adapter Identifier: sas.514187706fbb3400

Target Identifier: sas.60bc7462039a6135

Adapter Transport Details: 514187706fbb3400

Target Transport Details: 60bc7462039a6135

Maximum IO Size: 131072

sata.vmhba37-sata.0:0-mpx.vmhba37:C0:T0:L0

UID: sata.vmhba37-sata.0:0-mpx.vmhba37:C0:T0:L0

Runtime Name: vmhba37:C0:T0:L0

Device: mpx.vmhba37:C0:T0:L0

Device Display Name: Local PLDS CD-ROM (mpx.vmhba37:C0:T0:L0)

Adapter: vmhba37

Channel: 0

Target: 0

LUN: 0

Plugin: NMP

State: active

Transport: sata

Adapter Identifier: sata.vmhba37

Target Identifier: sata.0:0

Adapter Transport Details: Unavailable or path is unclaimed

Target Transport Details: Unavailable or path is unclaimed

Maximum IO Size: 131072

[root@ESXI1:~] esxcli storage nmp device list

naa.614187706fbb34001ebc7462039a6135

Device Display Name: DELL Serial Attached SCSI Disk (naa.614187706fbb34001ebc7462039a6135)

Storage Array Type: VMW_SATP_LOCAL

Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.

Path Selection Policy: VMW_PSP_FIXED

Path Selection Policy Device Config: {preferred=vmhba0:C2:T0:L0;current=vmhba0:C2:T0:L0}

Path Selection Policy Device Custom Config:

Working Paths: vmhba0:C2:T0:L0

Is USB: false

mpx.vmhba37:C0:T0:L0

Device Display Name: Local PLDS CD-ROM (mpx.vmhba37:C0:T0:L0)

Storage Array Type: VMW_SATP_LOCAL

Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.

Path Selection Policy: VMW_PSP_FIXED

Path Selection Policy Device Config: {preferred=vmhba37:C0:T0:L0;current=vmhba37:C0:T0:L0}

Path Selection Policy Device Custom Config:

Working Paths: vmhba37:C0:T0:L0

Is USB: false

impranayk · ‎05-23-2016

Where is your virtual machine swap files are located? does it have sufficient space?

-------------------------------------------------------------------------
Follow me @ www.vmwareinsight.com
Please consider marking this answer "correct" or "helpful" if you found it useful

Pranay Jha | Blog: http://vmwareinsight.com
vExpert 2016/2017, VCAP5-DCD/DCA, VCP5-DCV, VCA-Cloud, VCE-CIA, MCSE, MCITP

J1mbo · ‎06-03-2016

Controller is PERC H730P - interesting.

Run an array consistency check from the DRAC (which will take a while) then run tech support report including array logs, and see what is going on.

All

suddenly i lost my DATASTORE