Postgres data directory and shared folder issues on Workstation 9.x

[carze]

Posting this in hope that someone might be able to provide some guidance (also posted on the Postgres mailing list and got some suggestions that didn't solve my issue).

Have been putting together a VM with a bunch of tools, including Postgres, on an Ubuntu 10.04 LTS guest OS for some time now. Recently we upgraded to Workstation 10 and have been using this as a base to update and build a new VM that is distributed to our users. Unfortunately it looks like the upgrade to Workstation 10 has caused some issues in Postgres that we did not see before. Our host OS' are Redhat 5.9 and it looks like the issues I am running to stem from running Postgres (and its data directory) out of a shared folder. The initdb step of postgres fails with the following error (snippet below):

DEBUG:  TZ "posix/Universal" gets max score 5200

DEBUG:  TZ "posix/W-SU" scores 0: at 1389225600 2014-01-09 04:00:00 std versus 2014-01-09 00:00:00 std

DEBUG:  TZ "posix/GMT-0" scores 0: at 1389225600 "GMT" versus "UTC"

DEBUG:  TZ "posix/GB" scores 0: at 1389225600 "GMT" versus "UTC"

DEBUG:  TZ "posix/EST5EDT" scores 0: at 1389225600 2014-01-08 19:00:00 std versus 2014-01-09 00:00:00 std

DEBUG:  TZ "posixrules" scores 0: at 1389225600 2014-01-08 19:00:00 std versus 2014-01-09 00:00:00 std

DEBUG:  TZ "localtime" gets max score 5200

DEBUG:  TZ "W-SU" scores 0: at 1389225600 2014-01-09 04:00:00 std versus 2014-01-09 00:00:00 std

DEBUG:  TZ "GMT-0" scores 0: at 1389225600 "GMT" versus "UTC"

DEBUG:  TZ "GB" scores 0: at 1389225600 "GMT" versus "UTC"

DEBUG:  TZ "EST5EDT" scores 0: at 1389225600 2014-01-08 19:00:00 std versus 2014-01-09 00:00:00 std

DEBUG:  invoking IpcMemoryCreate(size=29278208)

DEBUG:  creating and filling new WAL file

LOG:  could not link file "pg_xlog/xlogtemp.4163" to "pg_xlog/000000010000000000000000" (initialization of log file 0, segment 0): Operation not permitted

FATAL:  could not open file "pg_xlog/000000010000000000000000" (log file 0, segment 0): No such file or directory

DEBUG:  shmem_exit(1): 3 callbacks to make

DEBUG:  proc_exit(1): 3 callbacks to make

DEBUG:  exit(1)

DEBUG:  shmem_exit(-1): 0 callbacks to make

DEBUG:  proc_exit(-1): 0 callbacks to make

child process exited with exit code 1

I can run the initdb step on the VM and install to a non-shared folder directory (i.e. /tmp/) and do not run into any issues. I had thought it might be tied to an older version of vmware-tools (we were manually installing an 8.x version of vmware-tools) so I went ahead and installed the 9.6.x version of vmware-tools and on one of my test machines running Workstation 10 everything worked out perfectly. I replicated the process on a machine running Workstation 9 and it still failed. So I am not sure if some fixes in Workstation 10 are taking care of the problem that are not present in Workstation 9 but we have users who are using older versions of Workstation, Player and Fusion so having them update to a newer version isn't an optimal solution for us.

Any help would be appreciated.

[admin]

I am guessing that you are talking about the PostgreSQL distribution that came with redhat?

From the error it seems like permission error to write on the shared directory (but needs to be confirmed)?

Can you provide the PostgreSQL version?

[fakber]

From what you have shown above, I have a few questions for you.

1 - What is the user you are running initdb as?

2 - What is the exact command that you used when running initdb?  That is, what were the exact command-line arguments used?

3 - What are the permissions on the directory that you've chosen for pg_xlog?  Please run ls -la on the directory where you've chosen pg_xlog to be.

I have run Ubuntu 10.x, 12.x and 14.x in VMs and have Postgres running in all of them.  In fact, the VMs are spread across on Workstation 9, Fusion 7 and ESXi (various versions).  In most cases, I have built Postgres from source and built the DB cluster using initdb. 

As Jignesh has mentioned above, based upon the log messages below, it seems like a permissions issue.  By answering the questions above, we will know better what the problem may be.

LOG:  could not link file "pg_xlog/xlogtemp.4163" to "pg_xlog/000000010000000000000000" (initialization of log file 0, segment 0): Operation not permitted

FATAL:  could not open file "pg_xlog/000000010000000000000000" (log file 0, segment 0): No such file or directory

If the permissions are fine, then please run initdb with strace.  Also, show us the contents of the vmware.log file of the VM in question during the period that you run strace. 

If needed, please file a Support Request to have the vmware.log file and strace output reviewed in more detail. 

[michaelpq]

Additionally, what is the exact version of workstation 10 used? Fusion 6.0.3 and Workstation 10.0.2 have fixed some errors related to shared folder usages.

[carze]

Postgres version is 8.4

After more poking around, as well as detailed logging I get something like this:

DEBUG:  invoking IpcMemoryCreate(size=29278208)

DEBUG:  creating and filling new WAL file

LOG:  could not link file "pg_xlog/xlogtemp.30519" to "pg_xlog/000000010000000000000000" (initialization of log file 0, segment 0): Operation not permitted

FATAL:  could not open file "pg_xlog/000000010000000000000000" (log file 0, segment 0): No such file or directory

DEBUG:  shmem_exit(1): 3 callbacks to make

DEBUG:  proc_exit(1): 3 callbacks to make

DEBUG:  exit(1)

DEBUG:  shmem_exit(-1): 0 callbacks to make

DEBUG:  proc_exit(-1): 0 callbacks to make

child process exited with exit code 1

initdb: data directory "/mnt/pg_data" not removed at user's request

It looks like there's some sort of issue with linking of the file here, after digging around Postgres' messageboards it looks like this step is executing a hard link operation, which I cannot perform on the VM at the command-line as well.  After some more digging around it looks like VMware has some weird issues with hard links and shared folders?

[carze]

1 - What is the user you are running initdb as?

initdb is running as the postgres user, who also owns the data directory I am trying to write too.

2 - What is the exact command that you used when running initdb?  That is, what were the exact command-line arguments used?

Running the following from the command-line replicates the error each time for me:

sudo -u postgres /usr/lib/postgresql/8.4/bin/initdb --debug --noclean -D /mnt/pg_data/

3 - What are the permissions on the directory that you've chosen for pg_xlog?  Please run ls -la on the directory where you've chosen pg_xlog to be.

Permissions for the directory follow:

drwx------ 1 postgres postgres 4.0K 2014-09-15 19:16 pg_xlog/

I've attached both the strace output and the vmware.log that should cover the times that I ran initdb.

[michaelpq]

> After some more digging around it looks like VMware has some weird issues with hard links and shared folders?

Well, You should try to see if this is reproducible with at least workstation 10.0.2 in a shared folder.

[carze]

I get the same error on Workstation 10.0.2, but I still am not sure if this is a permissions error or just some weird issue with hard-linking from a shared folder.

Not 100% sure where to go from here at this point.

EDIT - I can replicate this behavior on fresh Ubuntu 10.04 and 14.04 LTS VM's using both postgres 8.4 and 9.3 -- I'm guessing that installing postgres in a shared folder just doesn't work out because of the hard link during the initdb step.