VMware Cloud Community
paulkbeyer
Contributor
Contributor

Svmotion Received an error from the server: Insufficient disk space on datastore

Hi All,

I'm using svmotion to shifty some vdisks around on my datastores and I'm having a couple of quirky problems using the svmotion plugin for VC v1.0 from Andrew Kutz and also using the Remote CLI when the VC plugin doesn't behave as expected.

VC is 2.5u3, ESX 4 blade cluster is all ESX 3.5u3, remote CLI is 3.5 build 104314. Storage is all FC shared.

Firstly - when using the VC Plugin, some datastores report negative free space values when there is blatantly space still available on the datastore whilst I try to move the VM plus one of it's vdisks. Plus datastores that do report positive free space thought initally in the plugin allow you to move the components of the VM to the new datastore, when clicking apply, it decides there isn't really enough space by returning an error : Insufficient disk space on datastore in the VI Client recent tasks pane.

Secondly, which is related I'm sure, when using the remote cli to use svmotion --interactive and move a specific disk because of the dumb errors from the VC plugin, it's also reporting back when trying to move one disk rather than the entire VM:- Use of uninitialized value in concatenation (.) or string at C:/Program Files/VMware/VMware VI Remote CLI/Perl/lib/VMware/VICommon.pm line 1502, <STDIN> line 10. Received an error from the server: Insufficient disk space on datastore 'datastore'.

It's really dumb that svmotion thinks there's no space available when there is. Anyone think of anything that might help convince the svmotion process that there really is space on my datastores?

Restarting VC possibly an option?

Cheers in advance chaps.

0 Kudos
8 Replies
Texiwill
Leadership
Leadership

Hello,

I would send email to akutz about the plugin. I have been using it with no major issues for quite sometime. I would check to make sure you have the latest version. Have you updated the RCLI lately as well?

As for the RCLI issue, I would open a support request with your VMware Support Representative. Also, how much disk space is available on the LUNs in question? How big is the VMFS/NAS volume to which you are placing the VM? Are RDMs involved?


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
paulkbeyer
Contributor
Contributor

Hi Texiwill,

All software's are the bleeding edge latest levels. I was hoping to hear from anyone elses experience on the forum before bothering the vendor and software writer respectively direct. But If no-one else is experiencing anything like this then I guess I might well do. I'll give it a day or two and see if there's any activity on the thread, if not go to VMware support with the issues on the rCLI.

Thanks anyhow.

Regards

Paul.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Also, how much disk space is available on the LUNs in question? How big is the VMFS/NAS volume to which you are placing the VM? Are RDMs involved?

Also are there any errors within /var/log/vmkernel during the SVMotion?


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
paulkbeyer
Contributor
Contributor

Hi Texiwill,

There are errors in the VirtualCenter log file, but they tell me little more than the output from the rCLI when used, or the VIClient for that matter:-

-- BEGIN task-internal-594 -- -- vim.LicenseManager.queryUsage -- 71E4FAEA-E123-4327-ADC8-A640A6F4F872

-- FINISH task-internal-594 -- -- vim.LicenseManager.queryUsage -- 71E4FAEA-E123-4327-ADC8-A640A6F4F872

-- BEGIN task-4039 -- vm-127 -- vim.VirtualMachine.relocate -- 71E4FAEA-E123-4327-ADC8-A640A6F4F872

Checking admissibility of VM id: vm-127, name: jockey under respool id: resgroup-243, name: Resources

Checking admissibility of VM id: vm-127, name: jockey under respool id: resgroup-243, name: Resources

(jockey) Unexpected exception (vim.fault.NoDiskSpace) while relocating VM. Aborting.

Starting journal rollback

Vm provision transaction rollback done

-- BEGIN task-internal-595 -- host-1275 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esxhost.pretend.local (192.168.2.27)

Retrieved host update to 60

Completed host synchronization

-- FINISH task-internal-595 -- host-1275 -- VpxdInvtHostSyncHostLRO.Synchronize

-- FINISH task-4039 -- vm-127 -- vim.VirtualMachine.relocate -- 71E4FAEA-E123-4327-ADC8-A640A6F4F872

-- ERROR task-4039 -- vm-127 -- vim.VirtualMachine.relocate: vim.fault.NoDiskSpace:

(vim.fault.NoDiskSpace) {

dynamicType = <unset>,

file = "[eva_vmfs3_vm3] jockey/jockey.vmdk",

datastore = "msa_vmfs3_vm2",

msg = ""

}

There's 247GB of a 276GB datastore free which is the msa_vmfs3_vm2 datastore and the vDisk size I'm moving is 150GB and the source datastore is 410GB with 156GB free.

No RDM's on my system at all.

Hope this helps someone see my problem

Regards

Paul.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

There's 247GB of a 276GB datastore free which is the msa_vmfs3_vm2 datastore and the vDisk size I'm moving is 150GB and the source datastore is 410GB with 156GB free.

Ah ha! I believe SVMotion makes a shadow copy of the VMDK on the source before doing the transfer. It may be trying to allocate the space for a full copy of the VMDK on the source and failing. Does SVMotion work for a smaller than 150GB VM from the same source? If so that would be my thoughts on why this would be the case. Not sure why they would be allocating like this however....


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
paulkbeyer
Contributor
Contributor

Hiya,

Unless there's some overhead in there somewhere then I don't think even the size of the source datastore is the issue. It doesn't make much sense what you're saying though either chap. You're implying that if you've got a datastore for a file server for example that you've setup as 1.5TB, you're going to need it to be a 3TB+ datastore to ever be able to sVmotion that vmdk file off the datastore onto another datastore.

As it is, the source datstore I have in question is 6GB larger than the vmdk file I'm trying to move and I don't have anything I can re-create the situation with right now as I've evacuated the other datastores due to failing hardware so I'd be putting myself at risk just to test this out.

Let me attach a vdf -h output and a screen of the sVmotion to show you what I'm looking at.

/vmfs/volumes/45faba1a-88dc94c4-cfbc-0017a4aa0616

699G 620G 79G 88% /vmfs/volumes/msa_vmfs3_vm1

/vmfs/volumes/46752043-4e3b4258-998b-0016357f2b89

276G 29G 247G 10% /vmfs/volumes/msa_vmfs3_vm2

/vmfs/volumes/46c019b4-12142c0d-e4ee-0017a4aa0616

410G 253G 156G 61% /vmfs/volumes/msa_vmfs3_carestore

/vmfs/volumes/46c44ca9-bbfa1fdf-a721-0017a4aa0616

399G 696M 399G 0% /vmfs/volumes/emc_vmfs3_mirror

/vmfs/volumes/47bc4bc5-fc6aa7e8-815d-0017a4aa0616

199G 30G 169G 15% /vmfs/volumes/emc_vmfs3_vm1

/vmfs/volumes/47bec684-a551ffba-d1de-001321f27405

511G 489G 22G 95% /vmfs/volumes/eva_vmfs3_vm2

/vmfs/volumes/48970b02-c8d25684-0101-001f296b0768

1.5T 1.3T 183G 88% /vmfs/volumes/emc_vmfs3_dragonet

/vmfs/volumes/4909c78a-db0eddc4-ad9d-001cc4452714

511G 455G 55G 89% /vmfs/volumes/eva_vmfs3_vm1

/vmfs/volumes/4909c847-ab767649-b0b3-001cc4452714

511G 486G 25G 94% /vmfs/volumes/eva_vmfs3_vm3

/vmfs/volumes/49252a2c-394eedfa-84c0-0017a4770000

683G 600G 82G 87% /vmfs/volumes/msa_vmfs3_vm3

As you can see, there's no negative spaces reported here. And check the screen to see how sVmotion is reporting datastore space.

Regards

Paul

0 Kudos
Texiwill
Leadership
Leadership

Hello,

As stated I am not sure why it would be doing this.....

Howeve SVMotion does make a shadow copy of the VMDK on the source. THink of it as a snapshot. Before moving the VMDK to the new location. Then the delta file is committed on the new location. SO it is using space on the source. The question is how much.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
justinsmith
Enthusiast
Enthusiast

Is there a tool/plugin that allows SVmotion from shared to shared storage? Not local to shared?

0 Kudos