If a datastore runs out of space ¿ it is unrecoverable or can be recovered in some way ?

[nachogonzalez]

Hey hope you are doing fine.

In order to provide a detailed answer I will need a little more information:
What kind of storage are you using? (vSAN, iSCSI, SAN, NAS, etc)
Are you provisioning your VMs as thin? think? both?

Now, a very wide answer to your question is: Yes it is recoverable.
Think of your datastore as the disk in your computer, if it is full (of whatever kind of files) there is always ways of recovering it (the easiest one emptying trash files)
The thing is, having the datastore full will cause issues on your environment: some unix based systems might become read only and other kind of nasty stuff might happen. But (in my experience) nothing unrecoverable.

[continuum]

If you have active sesparse snapshots at the moment you fill up the datastore expect that the latest sesparse files are damaged beyond repair. So make sure to power off affected VMs BEFORE you run out of space.

Do not assume that VMFS datastores behave like NTFS disks on your desktop.

Do not assume that running out of space will affect only open files.

For all decisions you need to make like shutting down mission critical VM assume that running out of space is an unacceptable risk.

In worst case all thin provisioned vmdks can be damaged.

Do not hope for self repairing features of VMFS - there also is no trashcan.

 -

Ulli

[KrizProgramm]

We lost access to 2 Datastores. When validating in the VSphere client, the console data does not appear and through the ESXi console we can still see the volume but it is inaccessible. In addition, the df command generates the following error:

Traceback (most recent call last):
File "/bin/df", line 101, in <module>
sys.exit(main(sys.argv))
File "/bin/df", line 55, in main
o = eval(output)
File "<string>", line 1
Errors:
^
SyntaxError: invalid syntax

In the VSphere Client the virtual machines that were in the datastore are shown as Unknown # (inaccessible).

Here is more information about our virtualization environment:

Server: IBM System x3650 M3
Version ESXi 5.5.0
Version VSphere Client 5.5.0
Types of provisioning Thin y Thick.
Type Datastore: VMFS5
Disc Types: No-SSD
Datastore1: 2 discos de 600 GB.
Datastore2: 1 disto de 1TB.

Results in ESXi console:
-sh: cd: can't cd to datastore1
du: ./5f3d9dba-8d9d1bf6-3da0-5cf3fcb91e9c: Device or resource busy

[nachogonzalez]

Hey @KrizProgramm what is the Device Backing for your Datastores? 
Are you using directly attached disks? are you using a storage array?

Can you log in to the esxi console and get vmkernel.log? (try alt+F12)

[KrizProgramm]

hey , @nachogonzalez  We do not have a backup of the datastore and all the disks are configured in RAID0 attached the file vmkernel.log

[nachogonzalez]

Hey, hope you are doing fine:
By device backing I mean the type of storage you are using for your datastores.
It can be a storage array (SAN, NAS, etc), Directly connected disks, vSAN, or others.

I will review the logs ASAP.

[nachogonzalez]

I just checked the logs, they are flooded with this kind of entries:
2021-01-20T11:36:16.174Z cpu11:32779)ScsiDeviceIO: 2337: Cmd(0x412e80420e40) 0x28, CmdSN 0x1c0558d from world 32841 to dev "naa.600605b003bf1fb026d05807539ef07d" failed H:0x1 D:0x0 P:0x3 Possible sense data: 0x0 0x0 0x0.
(all for LUN naa.600605b003bf1fb026d05807539ef07d)

As you can see in the image this means connection to the datastore backing LUN is lost.

- If you have some sort of storage array: I'd suggest checking what's going on over there.
- If you have local disks as datastores: I'd suggest running you server's hardware diagnosis tool (mainly focusing on the controllers).


Two other things to take into consideration:
- Have you made any changes? (Driver/Firmware upgrade)
- vSphere 5.5 is out of support for a while, if possible I'd suggest to upgrade to a newer version.

Let me know how that works.

[KrizProgramm]

@nachogonzalez  our disks are directly connected and not have third party diagnostic tool  .

[nachogonzalez]

Might sound silly but have you tried rebooting the server?

If there is some issue with the controllers/disks (as I suspect there is) server's self hardware diagnostics will let you know before BIOS POST message.

Let me know.