VMware Cloud Community
BenignSage
Contributor
Contributor

NFS Trouble (vSphere 4.0 and Windows Server 2008)

Hello everybody. I am at my wits end trying to get a simple NFS  datastore working between vSphere 4.0 (ESX 4.0) and Windows Server 2008.  I was able to set this up in Windows Server 2003 R2 a few months ago  with none of the problems I am experiencing now. I have scoured the  internet for solutions, tried everything and have not been able to get  this to work. Please see below for more details:

Current Set-Up

Installed Windows Server 2008 File Services

Installed Services for Network File System

Created a new folder and shared it for NFS

  • share name: NFS
  • allow anonymous access checked
  • anonymous uid: -2
  • anonymous gid: -2
  • NFS Permissions:
    • [IP Address] Read-Write ANSI Root Access Allowed
    • ALL MACHINES Read-Only ANSI Root Access Disallowed
    • (I have even tried this with all machines root access and read-write allowed)
  • NTFS Security
    • ANONYMOUS LOGON - Full Control
    • SYSTEM - Full Control
    • CEATOR OWNER - Full Control (Special)
    • Administrator - Full Control

Added datastore to ESX host via GUI

  • Host > Confiuration > Storage > Add Storage
  • Added NFS datastore with name of NFS
  • ERROR: A general system error occurred: Timed-out while waiting to get datastore information from host
  • Datastore appears "normal" in the list of datastores (green check mark, status: normal, capacity and free space correct)

Tried removing NFS datastore from GUI

  • ERROR: The object has already been deleted or has not been completely created (due to error above)

After refreshing more datastore objects begin appearing NFS(1), NFS(2), NFS(3), NFS(4), NFS(5) (and continues)

Tried browsing datastore

  • Datastore appears empty although files exist within
  • Tried creating a folder
    • ERROR: Cannot complete file operation

Connected to NFS share from another computer

  • Connected successfully
  • Could see files
  • If I clicked on a file it would disappear, a refresh would show it again, multiple refreshes would hide/show the file

Removed datastore using CLI (success)

Restarted management services to refresh the GUI (success)

Datastores are now gone from host

I hope the  above helps, I have tried to be as descriptive as possible. I have  tried so many different "solutions" to try to solve this so I may have  forgotten a few things. I noticed this time around that when I mounted  the datastore I could not see anything. In the past I COULD see files  (browse datastore in the GUI), but if I refreshed they would dissapear,  another refresh would make them appear. I could also (again previous to  this attempt) create a folder, but not delete it (from browse datastore  in the GUI).

I remember when I set this up in Windows  Server 2003 R2 that there was a step where I needed to do something  either with the passwd file or user mapping or something. I am not sure  if this exists or not in 2008 as I do not recall performing this step  nor do I recall the option even being available (but they could have  renamed it like they did pretty much everything else).

If there is any additional information you may need please don't hesitate to ask. I appreciate any assistance anyone can give.

Thanks!

Benign Sage

Reply
0 Kudos
12 Replies
kjb007
Immortal
Immortal

Go into your local security policy, and make sure under Security Settings -> Local Polciies -> Security Options, that yo uhave 'Network access:  Let Everyone permissions apply to anonymous users' set to Enabled.  Reboot your server, and try again.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
BenignSage
Contributor
Contributor

KJB,

Thanks for your reply. I followed your advice, changed the setting in the local security policy, rebooted the server, tried to add the datastore again and I am still having the same issues:

Tried to add datastore

  • ERROR: A general system error occurred: Timed-out while waiting to get datastore information from host

Datastores began multiplying (NFS (1), NFS(2), NFS (3))

Browsed datastore

  • CAN see files/folders, however
    • folders show up as files
    • refresh many times and folders will show up as folders
    • can NOT delete folders/files
      • ERROR: Object reference not set to an instance of an object

Used CLI to remove NFS datastore

Restarted management services

These seem to be some more of the same errors I was receiving the other day. I can only imagine that it is something small that I am missing here.

Any other thoughts?

Thanks!

Benign Sage

Reply
0 Kudos
kjb007
Immortal
Immortal

Did you have the Everyone group in the security settings to allow all permissions?

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
BenignSage
Contributor
Contributor

KJB,

I did NOT have the Everyone group in my list of permissions. I configured the Everyone group to have Full Control of the folder, tried the same procedure mentioned previously (adding the datastore, testing it) and am still having the same errors occur.

Benign Sage

Reply
0 Kudos
BenignSage
Contributor
Contributor

Just as another note:

I have checked my Event Viewer and I am noticing the following warning has begun to show up:

Log Name:      Application
Source:        Microsoft-Windows-IDMU-PSync
Date:          9/22/2011 8:19:45 AM
Event ID:      8245
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      STORAGE.mydomain.com
Description:
Password propagation is not done. Either default encryption key is configured or no UNIX hosts configured to propagate password
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-IDMU-PSync" Guid="{E8DAF010-F14B-435F-BE59-93EDE773763C}" EventSourceName="NT to UNIX Password Sync Service" />
    <EventID Qualifiers="49152">8245</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-09-22T12:19:45.000Z" />
    <EventRecordID>47216</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>STORAGE.mydomain.com</Computer>
    <Security />
  </System>
  <EventData>
  </EventData>
</Event>

Reply
0 Kudos
kjb007
Immortal
Immortal

Are you still limiting your hosts in the NFS permissions?  Do you have multiple vmkernel ports on your ESX host that can reach the storage?

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
BenignSage
Contributor
Contributor

I still had the hosts limited by IP. I took those out and gave ALL MACHINES Read-Write and root access. I had an existing VMKernel port but created another one called NFS just to be sure. Tested connectivity to the file server successfully. Was able to mount the datastore successfully (unless it lies), however when trying to delete folders I received an error.

Below are steps I have taken:

Removed host-based permissions from the NFS share

Granted ALL MACHINES Read-Write and Allow Root Access

Created VMKernel port called NFS with a static IP address

Using the CLI, tested connectivity to the file server using vmkping

  • Pinged successfully

Created NFS datastore on host

  • Created successfully
  • No error as before

Browsed datastore

  • Created folder
    • Success
  • Tried to delete folder
    • ERROR: Object reference not set to an instance of an object.

Double-checked NTFS permissions on NFS share

  • Made sure that Everyone group had Full Control and that permissions were inherited
  • Noticed the permissions on the folder created that Everyone had Read/Write permissions but not Delete/Full Control
  • Tried manually assigning full control to the folder and deleting it from Browse Datastore, same error

When I am browsing the datastore sometimes the folder I create does not show up, I have to refresh multiple times for it to show up, sometimes other folders disappear, sometimes they have a folder icon, sometimes it's a file icon.

Getting really close now! Almost there!

Benign Sage

Reply
0 Kudos
kjb007
Immortal
Immortal

Can you turn off permission inheritance on the NFS folder, so that the new folders take their permissions from what you set?  I'm running out of ideas at this point.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
BenignSage
Contributor
Contributor

KJB,

Thanks for all your input, it has helped to get me closer to solving the problem.

I have turned off inheritance, double checked all permissions, Everyone, Anonymous Users both have Full Access. Tried deleting and recreating the datastore. Even tried it all from the command line. Mounts the datastore but can not modify/delete files.

Benign Sage

Reply
0 Kudos
kjb007
Immortal
Immortal

Are you using a plain vanilla Windows 2K8 build, or is it a custom / hardened os?  Maybe there's additional security processes at work here?

The service seems to be working, at this point, you are definitely running into a permission problem somewhere, I just can't see it.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
BenignSage
Contributor
Contributor

I am wondering as well. I have a suspicion... one of the tutorials I went through for a "solution" suggested that the server needs to be a domain controller. I was hesitant, but after trying everything else made this server into a domain controller. The only thing I can think of is that being a domain controller has tightened security and caused these issues. The next chance I get I am going to revert all the changes I made (uninstall NFS services, demote it as a domain controller, etc.) and then try setting up NFS again. It's so close to working, I can only help but think it's something small causing these issues, it should not be this hard to set up. I appreciate all your input and will be referring back to it when I try to set up NFS again on this box.

Thanks so much for your help!

Benign Sage

Reply
0 Kudos
zotamus
Contributor
Contributor

I don't know if this will help, but I found this article on troubleshooting NFS:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100396...

Also this information might be what you are looking for specifically:

To make sure that you can create and manage datastores from your host, the NFS administrator must turn off the root squash feature or add the VMware Server host’s physical network adapter to the list of trusted hosts.

Syntax:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100594...

Other info:

http://www.vmware.com/files/pdf/VMware_NFS_BestPractices_WP_EN.pdf - page 11

Cheers,

Zot

Reply
0 Kudos