ESX suddenly seems to have only READ access to a LUN

[jjohnston1127]

One of my clients has an ESX cluster setup with 3 ESX hosts.

They have a 300GB LUN running several virtual machines at the moment.

When attempting to clone, create, or power on a virtual machine (that does not already have a swap file I've noticed so far) it gives an error saying "Unable to access file."

I attempted to power on one of the virtual machines and got the error about no swap file, unable to access file. After relocating the virtual machine storage to another datastore and powering it on, it works fine. When I looked at the datastore the VM was previously on, the files are still there. When they were copied over to the other datastore they apparently were not deleted.

When attempting to create a file using touch, I get the following:[i] touch: creating `test.txt': Cannot allocate memory[/i]

Another note: The virtual machines that are up and running on this datastore seem to be operating fine. Some that are up and tested OK that I try to access via command-line and display a directory structure for the VM times out. I would assume that it is still able to RW to the LUN to continue to let the VM run.

Has anyone seen this happen? Is it something along the lines of storage having problems or is it an ESX problem?

Message was edited by:

jjohnston1127

[mbrkic]

Which user are you using to do this? Does that user have proper permissions to access and/or modify the files?

[mbrkic]

Cannot allocate memory sounds suspicious as well. Check your SC memory with top, and check /var/log/messages for any warnings/errors.

[jjohnston1127]

It isn't a user permission error as I'm attempting to do this with root via the command line and administrators via the VC client.

I checked esxtop and everything looked ok. I'll check the /var/log/messages and see what it says there.

All of the ESX hosts can operate and maintain the other LUNs attached to them with no problem so I doubt it is a memory issue.

Nothing out of the ordinary in the messages file.

Message was edited by:

jjohnston1127

[piacas]

Check the LUN its self on the array. I've seem this happen on a clariion and 'somehow' the LUN was put in read only on the array......

[jjohnston1127]

Confirmed that the array and LUN is not in read-only mode. Thanks for the suggestion though

[jjohnston1127]

FYI - this was caused due to something I cannot really explain because my knowledge in file systems isn't that great, but here we go:

The LUN on the SAN (IBM DS4700) was refusing write attempts to itself. You could copy, delete, etc, but no writes of new files were allowed.

After investigating further, there were over 4000 log files per virtual machine (about 7 on this LUN) that had not been cleaned up.

As soon as I deleted one of the directories full of log files, file creation was allowed again and everything worked normal.

I'm wondering if there is some sort of file-count limitation on a LUN. If anybody has any information or experience with this please post a comment on here and explain to me maybe for the benefit of the general public.

[mbrkic]

I suspect that you ran out of inodes, though the maximums document on the vmware site states 'unlimited' for number of files in a directory/vmfs (for VMFS3 anyway).

I am running a little test creating a lot of files on a VMFS to see where/if it breaks. Ill report the results when it is done.

[jjohnston1127]

Right on. I'm interested to see the results.

[mbrkic]

After creating 30710 files of zero length my loop stopped creating files and I got the same error message that you were getting:

\[root@esx011c tmp]# pwd

/vmfs/volumes/esx011local/tmp

\[root@esx011c tmp]# touch test

touch: creating `test': Cannot allocate memory

So I guess that the 'unlimited' that VMware is referring to is a 'practically unlimited', as opposed to 'real life unlimited'. Unless there is a way to change this parameter on vmfs creation or otherwise?

[jjohnston1127]

Interesting results. I suppose they don't figure a single datastore will ever grow to 30710 files. I wouldn't imagine a real life scenario in where that would happen other than my problem I experienced.

Thanks for running the test. Maybe someone can run this by the ESX developers as a bug(feature?) or for an explanation?

[Texiwill]

Hello,

Outside of this issue which is intriguing as it is not on power of 2 boundary....

You have other issues.... 4000 Log files per VM sounds way to high to me. Before you removed them did you investigate why they were created? Generally that many logfiles implies a problem.

Best regards,

Edward