stale NFS handles make sigar hang

[admin]

Hi Doug,

sigar_file_system_usage_get() is using statvfs() which blocks on a stale
NFS mounts.

$ mount 10.0.0.1:/nfs/path /mnt/nfs/

shut down nfs-server

$ sigar-test-all
will hang in statvfs().

To fix the hang you may have to use:

fd = open(..., O_RDONLY | O_NONBLOCK);
if (fd == -1) ...

fstatvfs(fd, ...);

open() will (hopefully) fail if it can't get a lock on the file.

I have not tested this yet, but it points into the right direction.

cheers,
Jan
--
Jan Kneschke, Enterprise Tools Dev, MySQL GmbH,
D-81373 München, Radlkoferstr. 2, GF: Kaj Arnö - HRB München 162140

[admin]

Jan,
A stale NFS mount should make everything hang, try a df -k while it is
hanging and you should see the same. The problem is in NFS, set the soft
mount option when mounting either in the fstab, vfstab, or mount command,
depending on situation and OS. The soft mount will allow a failure to
return, otherwise it will hang indefinitely.
Good Luck, John

John Buren Southerland
Principal Consultant
Southerland Consulting
801.467.8090(office)
214.734.8099(cell)
john@southerland-consulting.com

> -----Original Message-----
> From: sigar-dev-bounces@hyperic.org [mailto:sigar-dev-bounces@hyperic.org]
> On Behalf Of Jan Kneschke
> Sent: Monday, May 07, 2007 11:51 AM
> To: sigar-dev@hyperic.org
> Subject: stale NFS handles make sigar hang
>
> Hi Doug,
>
> sigar_file_system_usage_get() is using statvfs() which blocks on a stale
> NFS mounts.
>
> $ mount 10.0.0.1:/nfs/path /mnt/nfs/
>
> shut down nfs-server
>
> $ sigar-test-all
> will hang in statvfs().
>
> To fix the hang you may have to use:
>
> fd = open(..., O_RDONLY | O_NONBLOCK);
> if (fd == -1) ...
>
> fstatvfs(fd, ...);
>
> open() will (hopefully) fail if it can't get a lock on the file.
>
> I have not tested this yet, but it points into the right direction.
>
> cheers,
> Jan
> --
> Jan Kneschke, Enterprise Tools Dev, MySQL GmbH,
> D-81373 München, Radlkoferstr. 2, GF: Kaj Arnö - HRB München 162140

[dougm_hyperic]

Hi Jan,

I haven't tried this, but as John pointed out, it is unlikely to help.  The Java api has a getMountedFileSystemUsage method which checks that the nfs server is reachable via rpc ping.  If the ping fails, don't attempt to get the stats, avoiding the hang.  I can move this to the C api if you're interested.

On May 7, 2007, at 8:50 AM, Jan Kneschke wrote:

Hi Doug,

sigar_file_system_usage_get() is using statvfs() which blocks on a stale
NFS mounts.

$ mount 10.0.0.1:/nfs/path /mnt/nfs/

shut down nfs-server

$ sigar-test-all
will hang in statvfs().

To fix the hang you may have to use:

  fd = open(..., O_RDONLY | O_NONBLOCK);
  if (fd == -1) ...

  fstatvfs(fd, ...);

open() will (hopefully) fail if it can't get a lock on the file.

I have not tested this yet, but it points into the right direction.

cheers,
  Jan
--
Jan Kneschke, Enterprise Tools Dev, MySQL GmbH,
D-81373 München, Radlkoferstr. 2, GF: Kaj Arnö - HRB München 162140

[dougm_hyperic]

Jan, I added a sigar_file_system_ping() function you can use to check prior to calling sigar_file_system_usage_get().  We could probably make this more transparent for the C api as it is for Java, but its a start:

http://svn.hyperic.org/?view=rev&root=Hyperic+SIGAR&revision=3246

On May 10, 2007, at 6:08 AM, Doug MacEachern wrote:

Hi Jan,

I haven't tried this, but as John pointed out, it is unlikely to help.  The Java api has a getMountedFileSystemUsage method which checks that the nfs server is reachable via rpc ping.  If the ping fails, don't attempt to get the stats, avoiding the hang.  I can move this to the C api if you're interested.

On May 7, 2007, at 8:50 AM, Jan Kneschke wrote:

Hi Doug,

sigar_file_system_usage_get() is using statvfs() which blocks on a stale
NFS mounts.

$ mount 10.0.0.1:/nfs/path /mnt/nfs/

shut down nfs-server

$ sigar-test-all
will hang in statvfs().

To fix the hang you may have to use:

  fd = open(..., O_RDONLY | O_NONBLOCK);
  if (fd == -1) ...

  fstatvfs(fd, ...);

open() will (hopefully) fail if it can't get a lock on the file.

I have not tested this yet, but it points into the right direction.

cheers,
  Jan
--
Jan Kneschke, Enterprise Tools Dev, MySQL GmbH,
D-81373 München, Radlkoferstr. 2, GF: Kaj Arnö - HRB München 162140