Cant add more disk space, error: "Insufficient disk space on datastore"

[dzenansirco]

Hi,

We have some problems with Vmware ESXi, we cant find solution for this problem.

We have ESXi 4.1. When we try add more provisioned space for hard disk on one of our virtual machine:

We got error:

So, we cant add more space for this hard disk for this virutal machine, but we have enough free disk space. This error happening only for this virtual machine, for other its works. How we can resolve this issue?

[zXi_Gamer]

I find that you have given 8GB for your VM. Also, please list out the datastore listing from Configuration>Storage.

One more reason could be that you have created VMs that have been assigned thick disks and are yet powered off, once you power on the other VMs also, you will be able to find out the actual disk size available for addding more disk space.

[dzenansirco]

Hi, thank you on quick answer!

As you can see, we have 1.5 TB free space on datastore where is this virtual machine:

Of course, I power off virtual machine before attempt add more space for hard disk. I cant add more space for anyone hard disks for this virtual machine (but I need only for first hard disk)! Also, I cant add more space for only 1 MB! We have about 10 other virtual machine and everythings works fine about adding more space for hard disks, only this virtual machine is problem. I think that is problem in some configugation file of this virtual machine or something, but I dont know what?

[zXi_Gamer]

Hi,

    Could you tell us what the vmware.log is saying about this when you try to increase the disk space? I was saying in previous post that if you have provided any shares to the disks of any other VMs either powered on or powered off, then yes, you might be facing this issue.

[rickardnobel]

dzenansirco wrote:

As you can see, we have 1.5 TB free space on datastore where is this virtual machine:

I see that the Datastore2 is VMFS3 on a ESXi 4.1, but with a size of 3.8 TB. Is this made up of several LUNs into extents? Could you verify that all of these combined LUNs are actually available?

[dzenansirco]

Hi,

@zXi_Gamer: "

    Could you tell us  what the vmware.log is saying about this when you try to increase the  disk space? I was saying in previous post that if you have provided any  shares to the disks of any other VMs either powered on or powered off,  then yes, you might be facing this issue."

vmware.log nothing saying when I got this error, vmware.log not creating during when I try add more space on hard disk1. No, there is no sharing disk for any other VMs.

@rickardnobel,

all LUN's are avaible (I can create new VM, a cant add more disk space but only for other VM's, not for hard disk 1 for this EMC-SHP problematic machine).

So, there is configuration files for this virtual machine:

Configuration files:

/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP # ls
EMC-SHP.nvram      EMC-SHP.vmdk       EMC-SHP.vmsd       EMC-SHP.vmx        EMC-SHP.vmxf       EMC-TMP-flat.vmdk
/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP # more EMC-SHP.vmx 
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "7"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
nvram = "EMC-SHP.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "soft"
--More-- (24% of 2746 bytes)
                           
displayName = "EMC-SHP"
extendedConfigFile = "EMC-SHP.vmxf"
floppy0.present = "TRUE"
numvcpus = "2"
scsi0.present = "TRUE"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsisas1068"
memsize = "4096"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "EMC-SHP.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide1:0.present = "TRUE"
ide1:0.clientDevice = "TRUE"
ide1:0.deviceType = "cdrom-raw"
ide1:0.startConnected = "FALSE"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.clientDevice = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network 3"
ethernet0.addressType = "generated"
disk.EnableUUID = "TRUE"
--More-- (48% of 2746 bytes)
                           
guestOS = "windows7srv-64"
uuid.location = "56 4d 37 c2 3d d1 c8 bc-ac 94 82 a4 0e 9b 39 4a"
uuid.bios = "56 4d 37 c2 3d d1 c8 bc-ac 94 82 a4 0e 9b 39 4a"
vc.uuid = "52 dd 72 7b 71 43 b2 b3-95 7a 0f 5c 1a cd 8b c1"
bios.forceSetupOnce = "FALSE"
ethernet0.generatedAddress = "00:0c:29:9b:39:4a"
svga.vramSize = "8388608"
vmci0.id = "245053770"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-TMP/EMC-TMP-cec22e1c.vswp"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "33"
scsi0.sasWWID = "50 05 05 62 3d d1 c8 b0"
vmotion.checkpointFBSize = "8388608"
--More-- (79% of 2746 bytes)
                           
ethernet0.generatedAddressOffset = "0"
tools.remindInstall = "TRUE"
hostCPUID.0 = "0000000b756e65476c65746e49656e69"
hostCPUID.1 = "000206c200200800029ee3ffbfebfbff"
hostCPUID.80000001 = "0000000000000000000000012c100800"
guestCPUID.0 = "0000000b756e65476c65746e49656e69"
guestCPUID.1 = "000206c200010800829822030febfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000b756e65476c65746e49656e69"
userCPUID.1 = "000206c200200800029822030febfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"

/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP # more EMC-SHP.vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=cc9b690b
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 104857600 VMFS "EMC-TMP-flat.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.sectors = "63"
ddb.geometry.heads = "255"
ddb.geometry.cylinders = "6527"
ddb.uuid = "60 00 C2 91 c2 bb cc 61-44 2c a1 32 d9 33 fe 83"
ddb.longContentID = "3fe9537e5187426eea474bedcc9b690b"
ddb.virtualHWVersion = "7"

That is for first problematic hard disk. There is configuration for others hard disks:

/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP_1 # more EMC-SHP_1.vmx
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "7"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
nvram = "EMC-SHP_1.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "soft"
--More-- (20% of 3328 bytes)
                           
displayName = "EMC-SHP"
extendedConfigFile = "EMC-SHP_1.vmxf"
floppy0.present = "TRUE"
numvcpus = "4"
scsi0.present = "TRUE"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsisas1068"
memsize = "8192"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide1:0.present = "TRUE"
ide1:0.clientDevice = "TRUE"
ide1:0.deviceType = "cdrom-raw"
ide1:0.startConnected = "FALSE"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.clientDevice = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "generated"
disk.EnableUUID = "TRUE"
--More-- (41% of 3328 bytes)
                           
guestOS = "windows7srv-64"
uuid.location = "56 4d 05 48 fb c8 97 84-5d 54 52 17 bc f1 ee 40"
uuid.bios = "56 4d 05 48 fb c8 97 84-5d 54 52 17 bc f1 ee 40"
vc.uuid = "52 a4 21 5a c5 84 36 d4-46 64 ad 9b d5 6c 67 75"
ethernet0.generatedAddress = "00:0c:29:f1:ee:40"
svga.vramSize = "8388608"
vmci0.id = "-1124995520"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP_1/EMC-SHP_1-87f15b26.vswp"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "33"
scsi0.sasWWID = "50 05 05 68 fb c8 97 80"
vmotion.checkpointFBSize = "8388608"
ethernet0.generatedAddressOffset = "0"
--More-- (67% of 3328 bytes)
                           
tools.remindInstall = "TRUE"
hostCPUID.0 = "0000000b756e65476c65746e49656e69"
hostCPUID.1 = "000206c200200800029ee3ffbfebfbff"
hostCPUID.80000001 = "0000000000000000000000012c100800"
guestCPUID.0 = "0000000b756e65476c65746e49656e69"
guestCPUID.1 = "000206c200010800829822030febfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000b756e65476c65746e49656e69"
userCPUID.1 = "000206c200200800029822030febfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"
scsi0:1.present = "TRUE"
scsi0:1.fileName = "EMC-SHP_1.vmdk"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:2.present = "TRUE"
scsi0:2.fileName = "EMC-SHP_1_1.vmdk"
scsi0:2.deviceType = "scsi-hardDisk"
scsi0:1.redo = ""
scsi0:2.redo = ""
scsi0:3.present = "TRUE"
scsi0:3.fileName = "EMC-SHP_1_2.vmdk"
scsi0:3.deviceType = "scsi-hardDisk"
scsi0:3.redo = ""
--More-- (94% of 3328 bytes)
                           
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
sched.scsi0:3.shares = "normal"
sched.scsi0:3.throughputCap = "off"
tools.syncTime = "FALSE"
sched.swap.hostLocal = "inherit"

mfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP_1 # more EMC-SHP_1.vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=4443c850
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 536870912 VMFS "EMC-SHP_1-flat.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.sectors = "63"
ddb.geometry.heads = "255"
ddb.geometry.cylinders = "33418"
ddb.uuid = "60 00 C2 9c b9 85 7d b8-42 59 16 9d 6a 84 d3 a0"
ddb.longContentID = "ded6d0de9dc67983678f43d84443c850"
ddb.virtualHWVersion = "7"

[zXi_Gamer]

@dzenansirco: My apologies. I had meant to ask about vmkernel.log when you extend the disk. But do not post the vmkernel entire message, as it might have other critical information also.

From your vmx file, the only thing i can see different is your extends

# Extent description
RW 104857600 VMFS "EMC-TMP-flat.vmdk"  for first vm.

I am reading more into your vmx. Will post asap

[dzenansirco]

OK, but vmkernel.log on ESXi doesnt exist, it is var/log/message file, I found some interesting in that file:

May 15 07:10:48 Hostd: [2012-05-15 07:10:48.779 7C1C2B90 verbose 'vm:/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP_1/EMC-SHP_1.vmx' opID=9FD2C069-000000CB] Current VM Tracking state: disabled
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.811 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-VMFS  : "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-TMP-flat.vmdk" : open successful (16) size = 53687091200, hd = 66476976. Ty
May 15 07:10:48 pe 3
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.811 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-DSCPTR: Opened [0]: "EMC-TMP-flat.vmdk" (0x10)
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.811 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-LINK  : Opened '/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk' (0x10): vmfs, 104857600 sectors / 50 GB.
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.811 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-LIB   : Opened "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk" (flags 0x10, type vmfs).
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.812 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-LIB   : Growing disk '/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk' : createType = vmfs
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.812 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-LIB   : Growing disk '/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk' : capacity = 104857600 sectors - 50 GB
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.812 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-LIB   : Growing disk '/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk' : new capacity = 106954752 sectors - 51 GB
May 15 07:10:48 sfcb-vmware_base[3954869]: LsaFindUserByName: 40008
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.902 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-VMFS  : "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-TMP-flat.vmdk" : closed.
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.902 7C1C2B90 info 'DiskLib' opID=9FD2C069-000000CB] DISKLIB-LIB   : Failed to grow disk '/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk' : There is not enough space on the file syste
May 15 07:10:48 m for the selected operation (13).
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.902 7C1C2B90 info 'vm:/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP_1/EMC-SHP_1.vmx' opID=9FD2C069-000000CB] Error extending disk /vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SH
May 15 07:10:48 P.vmdk to capacity 53477376: There is not enough space on the file system for the selected operation
May 15 07:10:48 Hostd: [2012-05-15 07:10:48.903 7C1C2B90 info 'vm:/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP_1/EMC-SHP_1.vmx' opID=9FD2C069-000000CB] Reconfigure failed

For extent

# Extent description
RW 104857600 VMFS "EMC-TMP-flat.vmdk"  for first vm.

What is problem in that line? How can I fix that? Are you find something else that could be problematic?

[john23]

If my understanding is right, you want to extend the vmdk.

can you try to use command line : "vmkfstools -X 5g disk name.vmdk"

[dzenansirco]

I want extend my hard disk 1 - it is EMC-TMP-flat.vmdk

I cant use this command vmkfstools -X 5g disk name.vmdk as "vmkfstools -X 5g disk EMC-TMP-flat.vmdk" because nothing happening, what is a parametar disk? I thing that it will not working because over vSphare not working.

[john23]

not run on flat vmdk,run on the "vmkfstools -X 5g disk EMC-TMP.vmdk".

Then do ls -lh and check the size of flat vmdk.

Check in vSphere as well and check the data store size as well

[dzenansirco]

Thank you. I try but nothing. If is everything OK with configurations file, it must works via vSphare.

[zXi_Gamer]

I am just finding that when the scsi0:0 disk file is scsi0:0.fileName = "EMC-SHP.vmdk", the swap and extent that got created is sched.swap.derivedName = "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-TMP/EMC-TMP-cec22e1c.vswp and  RW 104857600 VMFS "EMC-TMP-flat.vmdk" respectively.

I am trying to link why a EMC-TMP had kicked in for swap space. It is likely you have added another disk to the VM. Can you confirm this swap file size of this VM.

And btw, you can check vmkernel logs only, by pressing alt+f12 in the display screen of the ESXi server to enter non-interactive screen.

[dzenansirco]

Hm..... Yes! It could be probem!  I dont know why sched.swap.derivedName is to mfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-TMP/EMC-TMP-cec22e1c.vswp because I cant find that file in /EMC-TMP, file .vswp file doesnt exist. Swap file must be to EMC-SHP. It can be problem, EMC-TMP must not have any connection with EMC-SHP, EMC-TMP is totaly sperate machine. Do you have idea how I can fix that?

vmkernel.log I was copy/paste in my last post - it is only part od vmkernel.og (var/log/message)

[continuum]

I dont understand the command you use
you already have a 50Gb vmdk and want to expand it to the new size of 5 Gb ? - makes no sense

[dzenansirco]

No! I have 50 GB and I want extend to 55 GB! Or 100 GB, or 120 GB never mind, I cant extend on 51 GB!

[continuum]

whats the output of
vmkfstools -X 55Gb vmdk-name.vmdk

can you create other vmdks in that directory ?

[zXi_Gamer]

One more thing, in EMC-SHP_1.vmx, scsi0:0.fileName = "/vmfs/volumes/4d4b22d3-62171bd7-2be4-3c4a92758c72/EMC-SHP/EMC-SHP.vmdk" is appearing.

I do not understand why this has been made. Can you try and ONLY if you are allowed, try powering off EMC-TMP VM and extend the disks.

I also understand that you have not created the disks while creating a VM, but you have opted for directing to already existing vmdks while creating the other VMs.

Also, one more query, are the disks by any chance shared between VMs?

[dzenansirco]

Hi,

@zXi_Gamer:

"I do not understand why this has been made. Can you try and ONLY if you  are allowed, try powering off EMC-TMP VM and extend the disks."

Yes, I try and nothing same error.

@zXi_Gamer:"I also understand that you have not created the disks while creating a  VM, but you have opted for directing to already existing vmdks while  creating the other VMs.

Also, one more query, are the disks by any chance shared between VMs?"

Yes! I copied existing vmdk (from EMC-TMP) but I was make a copy to EMC-SHP. For example, hard disk 4 I can extend because it was new hard disk, but hard disk 1 (EMC-TMP-flat.vmdk) was copied from EMC-TMP to EMC-SHP. I know that there is not shared any disk between VMs, how I can check that? And, what I need change to config file for fixing this problem?

[zXi_Gamer]

We should now confirm if you are able to create vmdk or extend by command @continuum has said.

Longback, I had this prob of the VI Client changing the GB to TB when I click ok. After upgrading the vi client, this prob was not seen. Do let us know, if you are able to extend using command line the same Harddisk1 vmdk