tcador
Contributor
Contributor

Virtual Machines over NFS: Permission Denied on Power On

Hi Guys,

I'm stumped on this one, hopefully you all can point out what I'm doing wrong. I've got an NFS server added to one of my ESX 3.5 servers as a shared folder for virtual machines. Whenever I try to power on the virtual machine, I get this error: "Could not power on VM : Permission denied. Failed to power on VM". Here are the details of the setup:

1. NFS server with root squash enabled (can't disable this). This means i need to setup a delegate user...

2. NFS directory owner from the NFS server : 'testuser'(uid=10032) and 'testgroup' (gid=11032)

3. Created matching testuser / testgroup on ESX server

4. Set ESX delegated user to be 'testuser' and rebooted.

When I create a new virtual machine, everything seems to work fine. All files created with the correct owner and group. But powering on throws the permission denied error. From vmware.log, I see that the vswp file doesn't is what is causing the problems:

Jul 25 14:01:07.401: vmx| VMMon_CreateVM: vmmon.numVCPUs=1

Jul 25 14:01:07.403: vmx| Swap file path: '/vmfs/volumes/54811446-bdb157d8/testvm/testvm-eca36cfa.vswp'

Jul 25 14:01:07.460: vmx| Msg_Post: Error

Jul 25 14:01:07.460: vmx| http://msg.vmmonVMK.creatVMFailed Could not power on VM : Permission denied.

Jul 25 14:01:07.460: vmx| http://msg.monitorLoop.createVMFailed Failed to power on VM----


Jul 25 14:01:07.471: vmx| Module MonitorLoop power on failed.

Jul 25 14:01:07.471: vmx| VMX_PowerOn: ModuleTable_PowerOn = 0

Jul 25 14:01:07.539: vmx| vmdbPipe_Streams Couldn't read: OVL_STATUS_EOF

Directory listing (ls -ln):

-rw------- 1 10032 11032 0 Jul 25 14:05 testvm-eca36cfa.vswp

-rw------- 1 10032 11032 8589934592 Jul 25 14:05 testvm-flat.vmdk

-rw------- 1 10032 11032 373 Jul 25 14:05 testvm.vmdk

-rw------- 1 10032 11032 0 Jul 25 14:05 testvm.vmsd

-rw-rr 1 10032 11032 1213 Jul 25 14:05 testvm.vmx

-rw------- 1 10032 11032 261 Jul 25 14:05 testvm.vmxf

-rw-rr 1 10032 11032 15827 Jul 25 14:05 vmware.log

Finally, if I change permissions on the vswp file to 620 instead of 600 everything works fine. I'm wondering why I have to do this? I can change the permissions using the testuser on command-line without any problems. Anyone know what I'm doing wrong here?

Thanks

Tags (2)
0 Kudos
10 Replies
Texiwill
Leadership
Leadership

Hello,

Who is the owner and group of the .vswp file? THat could make a difference in how a VM works.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIII: 2009-2021,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
RParker
Immortal
Immortal

> -rw------- 1 10032 11032 373 Jul 25 14:05 testvm.vmdk

testvm.vmdk needs user w permission. (read/write).

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Do you not mean 'x' or execute permission as it already has 'w' permissions. But it should not need 'x' perms either.

If the .vswp needs 620 perms then I would say the owner of the file is incorrect as that should definitely not need any group perms to run unless the owner is not really root.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIII: 2009-2021,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
tcador
Contributor
Contributor

The NFS /etc/exports looks like this:

"/nfs/nova" *(rw,no_subtree_check,insecure,anonuid=99,anongid=100)

Because root squash is not enabled, I have to set the permissions to 620. I guess my question is, why is the vswp file being created with 600 permissions initially, when other files in the same directory are different permission (such as 640). I thought it was related to NFS server setup, and maybe it still is, it just seems strange that the permissions on all the files when cloning / creating a new virtual machine are all correct. It just runs into problems when powering on due to the wrong permissions on that particular file.

-rw------- 1 10032 11032 0 Jul 25 14:05 testvm-eca36cfa.vswp

-rw------- 1 10032 11032 8589934592 Jul 25 14:05 testvm-flat.vmdk

-rw------- 1 10032 11032 373 Jul 25 14:05 testvm.vmdk

-rw------- 1 10032 11032 0 Jul 25 14:05 testvm.vmsd

-rw-rr 1 10032 11032 1213 Jul 25 14:05 testvm.vmx

-rw------- 1 10032 11032 261 Jul 25 14:05 testvm.vmxf

-rw-rr 1 10032 11032 15827 Jul 25 14:05 vmware.log

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Which NFS server are you using?

Actually, root_squash is enabled (which implies that root writes get mapped to nobody.nobody. If you enable no_root_squash then root is not mapped to anything.

ESX generally requires 'no_root_squash' to be set as it does all its writing/reading as root. The default for any NFS mount is 'root_squash'.

Therefore, you do need to set the group permissions appropriately as root can not access something as the user if the user of the files is nobody.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIII: 2009-2021,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
tcador
Contributor
Contributor

Edward,

You're exactly right, I meant to say root_squash IS enabled, meaning the groups and permissions must be set accordingly as you stated. I'm using what I consider I pretty standard NFS server, running on RHEL4 with the exports file: /nfs/nova *(rw,insecure,async,no_subtree_check,anonuid=99,anongid=100)

I guess what I need to do is take a closer look at how the NFS server is setup, as it seems the permissions must be too strict by default (even though all the other files look right). One thing to note here. If I logon to the ESX server as the 'testuser', then change directories to the virtual machine directory on the NFS server, I can create and delete files no problem. Touching a file in this directory ends up with default permissions of 644, with no problems changing permissions to the required 620. If the NFS server permissions weren't correct, I wouldn't think this test would work.

Any other suggestions based on this information? Thanks!

0 Kudos
Texiwill
Leadership
Leadership

Hello,

The NFS Datastore should be allowed to mount using the no_root_squash directive in your /etc/exports. It will fix any issues with needing group perms for starting a VM. In addition it will not require any mapping of root perms to a non-root UID.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIII: 2009-2021,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
tcador
Contributor
Contributor

I'm located in the Eastern Hemisphere and had the better part of today devoted to further investigation into this issue. Unfortunately, I was unable to come up with any solution that worked, other than temporarily disabling root_squash. I also applied the ESX 3.5/VC 2.5 Update 2 with no luck.

Since I appear to be the only one with this problem, it has got to be something mis-configured. Any suggestions on further debugging this issue?

Thanks for your time.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Your NFS server is misconfigured. THe NFS server should have no_root_squash defined for the ESX server. This is a requirement from the VMware ESX documentation. Granted, I agree it is not desirable, but it is the current requirement.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIII: 2009-2021,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
tcador
Contributor
Contributor

Thanks for the clarification/confirmation on this requirement.

0 Kudos