Unable to create files in VMFS

[baselinept]

Hi,

I'm having a problem I've never seen before, hope someone can help me.

All of a suden I can't power on any of my VM's. They all give the error "no swap file"

It seems files can't be created on the vmfs volume and it still has about 64 GBs free. Whenever I try to create a file in the ESX console, i.e using touch, it gives out the error: "touch: creating `/vmfs/volumes/vms/teste.txt': Cannot allocate memory"

Can someone help?

tks

[baselinept]

bump

Anyone?

I'm really at a loss... Never encountered this error before... Starting to think that the only way out is to format the ESX Host.

[GBromage]

It sounds like the Service Console is running out of memory. When you created the host, did you give it a swap partition? And is the swap partition actually mounted?

[baselinept]

Hi,

Here is the output from df:

df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 1.6G 3.0G 35% /

/dev/sda1 99M 30M 65M 32% /boot

none 131M 0 131M 0% /dev/shm

/dev/sda6 2.0G 56M 1.8G 3% /var/log

This is an ESX 3.0. I think the VM's create the swap file on demand when they're started.

cheers

[baselinept]

Upps... sorry... Here it is in human readable format ...

df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 1.6G 3.0G 35% /

/dev/sda1 99M 30M 65M 32% /boot

none 131M 0 131M 0% /dev/shm

/dev/sda6 2.0G 56M 1.8G 3% /var/log

[mbrkic]

It does sound like it might be the SC memory. How big is it?

Can you 'touch' a file on a regular fs?

You will have to look at 'free -l', 'top' or 'cat /proc/meminfo' for example to get stats on the memory usage in the SC.

Also see if you have any errors in /var/log/messages about 'out of memory'.

[GBromage]

Also look at 'cat /proc/swaps' to see if the swap is actually being used.

[baselinept]

Tks for the help guys.

I've got it running again. I've deleted some files from the vmfs partition and now it's working again...

Can only attribute this to a bug... vdf -h at the time reported more than 50 gigs free. I don't understand why I had to delete some files..

As anyone experienced something like this before?

Here is the complete output from vdf:

vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 4.9G 1.6G 3.0G 35% /

/dev/sda1 99M 30M 65M 32% /boot

none 131M 0 131M 0% /dev/shm

/dev/sda6 2.0G 51M 1.8G 3% /var/log

/vmfs/devices 402G 0 402G 0% /vmfs/devices

/vmfs/volumes/46153cf2-ad56f2bc-72a7-00137268670a

129G 43G 85G 33% /vmfs/volumes/vms

here is the output from esxcfg-vmhbadevs:

vmhba0:0:0 /dev/sda

vmhba0:1:0 /dev/sdb

cheers

[GBromage]

Can only attribute this to a bug... vdf -h at the

time reported more than 50 gigs free. I don't

understand why I had to delete some files..

As anyone experienced something like this before?

I've never seen it myself, but I suppose it's possible, if you were VERY low on memory.

The service console is based on Linux, which can support a number of different file systems. In order to do so, it links the various physical file systems (ext, ext2, fat, vmfs(?) ) through a Virtual File System layer, which is naturally held in memory.

Theoretically, if you didn't have enough service console RAM to hold the file system, you might see this error. But I'd expect to see other memory errors occuring before you reached this stage.

How much memory does the host have?

[spseabra]

The host has 4 GBs RAM... The VM's collectivly don't consume more than 2,5 GBs...

Don't understand how this can happen... If fact the "cannot allocate memory" error when I tried to create any file in the VMFS partition is IMO pretty misleading...

cheers

[mbrkic]

But how much of that memory is allocated to the Service Console?

Post the output of 'free -l' , 'cat /proc/meminfo' and 'grep -i memory /var/log/messages' so we can get an idea of how your SC memory is configured and doing.

[systest]

I've the same problem, following is my related info, any suggestion?

\----

---

grep -i memory /var/log/messages

Aug 7 15:05:02 xxxxxxxx0300 kernel: Memory: 260404k/278528k available (1649k kernel code, 15628k reserved, 1207k data, 376k init, 0k highmem)

Aug 7 15:05:03 xxxxxxxx0300 kernel: Freeing initrd memory: 5356k freed

Aug 7 15:05:03 xxxxxxxx0300 kernel: Freeing unused kernel memory: 376k freed

\----

---

free -l

total used free shared buffers cached

Mem: 268252 261144 7108 0 41156 91396

Low: 268252 261144 7108 0 0 0

High: 0 0 0 0 0 0

-/+ buffers/cache: 128592 139660

Swap: 554200 0 554200

\----

---

cat /proc/meminfo

total: used: free: shared: buffers: cached:

Mem: 274690048 267411456 7278592 0 42151936 93589504

Swap: 567500800 0 567500800

MemTotal: 268252 kB

MemFree: 7108 kB

MemShared: 0 kB

Buffers: 41164 kB

Cached: 91396 kB

SwapCached: 0 kB

Active: 186608 kB

ActiveAnon: 101492 kB

ActiveCache: 85116 kB

Inact_dirty: 33384 kB

Inact_laundry: 10972 kB

Inact_clean: 3084 kB

Inact_target: 46808 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 268252 kB

LowFree: 7108 kB

SwapTotal: 554200 kB

SwapFree: 554200 kB

CommitLimit: 688324 kB

Committed_AS: 286648 kB

MachineMem: 33552992 kB

Message was edited by:

systest

[systest]

Additional info.

While error occurs (cannot touch any file in /vmfs/storage folder), just delete any file in the folder, even empty file, the system would work well again.

Currently, I can only copy file to /vmfs/storage one by one, if want to copy 2 or more or a folder to /vmfs/storage, the error will happen.

And I already set ESX server console memory to 800M (maximum value in my env)

Any suggestion?

[griffmeister]

I'm also seeing this problem and think I may have found a bug or a feature depending on your point of view

We have been trying to copy a large number of files into a VMFS-3 file system and the copy always fails at the same point even though the file system is only 80% (39.23Gb free) full.

To investigate this issue further, I created a small (30Gb) VMFS-3 file system and managed to successfully fill it to 100% by creating 6 files of 5Gb without any errors. I then deleted my last file, created a new one so that the file system was 99% full and could still create further files without error.

As the 'touch'[/i] command only creates a directory entry and does not consume space I began to think there must be a limit to the number of files a particular VMFS file system can hold (similar to i-nodes in UNIX file systems) even though the Configuration Maximums guide says this is [i]unlimited[/i][/b] in VMFS-3.

To test this theory, I am attempting to copy my data to the smaller VMFS-3 system and see if it fails at the same (or some earlier) point. If this fails, then I will create a standard EXT3 file system and store the data there as it is only needed to build the VMs. As this is still running, I will have to report the results later.

FYI, my VMFS-3 file system is 203.25Gb in size and contains 30,720 files. If I attempt to create another file, I get the 'Cannot allocate memory'[/i] error from all file creation commands. If I delete a file, I can create a new one but cannot go above a file count of 30,720.

If you are still seeing this problem, then check how many files you have in your VMFS-3 file system. I would be interested to see if there is a limit, whether this is related to file system size. Also, may need to report this as a 'bug'[/i] if there appears to be some sort of limit when there shouldn't be.

[systest]

FYI.

\----

---

\[root@xxxxxx /vmfs/volumn/storage]ls -aR|wc -l

48668

\----

---

[griffmeister]

Interesting ...

Is your VMFS partition larger than mine as you seem to have managed to create more files than I could. If so, then as with UFS i-node allocation the number of files the file system can hold is related to the file system size.

Also, the test with the smaller VMFS file system failed as it filled up. This was because my colleague was attempting to copy across lots of small files which would have a minimum file size of 1Mb (due to the VMFS block size).

In the end, I created a VM and then stored the data inside a virtual hard-disk (<VM-name>.vmdk file) attached to the VM and made the data available from there. This was because there appears to be no way to map an arbitrary EXT3 directory into a VMware disk/share that I could find (except for /vmimages which can be used to store .iso images). This seems to have got round the problems.

Moral of the story is that there does appear to be a limit on the number of files that a VMFS-3 file system can hold and that you should really not use native VMFS to store large numbers of files. Also, storing files that are less than the VMFS block-size in size makes inefficient use of the storage available. Better to do this using a native file system inside a VM and then sharing the resultant .vmdk file amongst multiple VMs if required.

Any other observations from anyone?