Troubleshooting NFS

Common NFS Errors

• "No such host" - Name of the server is specified incorrectly
• "No such file or directory" - Either the local or remote file system is specified incorrectly.
• "No such device" - NFS is not configured into the client's kernel.
• "NFS server is not responding" message followed by "NFS server xxxx OK" - Server is heavily loaded causing RPC timeouts, or server has crashed.
• "Stale file handle" - The file is no longer available.
• "MOUNT_PROG not registered" - rpc.mountd daemon never started up and registered.
• "Too many levels of remote in path" - Attempting to mount a file system which is already an NFS mounted file system.
• "Permission denied" - Accessing as root on the client and root is mapped to nobody. Or the user on the client does not have corresponding UID on the server.
• "No space" - The server is out of space on the file system. 

Troubleshooting

The showmount command may be used to display server-side mount information. The option -a displays all remote mounts showing the name of the client and the directory, separated by a colon. The -d option displays only the names of the directories mounted by the clients. And the -e option displays the list of file systems exported by the server.

     # showmount -a
     All mount points on local host:
     edcert20.ucs.indiana.edu:/home
     edcert21.ucs.indiana.edu:/usr/local

     # showmount -d
     Directories on local host:
     /home
     /usr/local

     # showmount -e
     Export list on local host
     /home           edcert21.ucs.indiana.edu edcert20.ucs.indiana.edu     
     /usr/local      edcert21.ucs.indiana.edu

The df command may be used to display information on the file systems mounted remotely, the mount point and the amount of available space. The -F option may be specified to list only a specified file system type.

     # df -F nfs
     Filesystem                      Type  blocks     use   avail %use  Mounted on
     edcert21.ucs.indiana.edu:/home  nfs    68510   55804   12706  81%  /usr/share/help

BSD systems use -t option to specify the fstype. The output from the df command also varies among the different operating systems. df also resolves the symbolic links and determines the file system mounted at the link's target. For example:

     # ls -l /usr/local/man
     lrwxr-xr-x    1 root     sys            6 Mar 17  1995 /usr/local/man -> catman/
     # df /usr/local/man
     Filesystem              Type  blocks     use   avail %use  Mounted on
     orange:/usr/local        nfs   68510   55804   12706  81%  /usr/local

Use the command nfsstat -s to display NFS activity on the server side. For example:

     # nfsstat -s

     Server RPC:
     calls      badcalls   nullrecv   badlen     xdrcall    duphits    dupage
     50852      0          0          0          0          0          0.00    

     Server NFS:
     calls        badcalls     
     50852        0            
     null         getattr      setattr      root         lookup       readlink  
     1  0%        233  0%      0  0%        0  0%        1041  2%     0  0%  
     read         wrcache      write        create       remove       rename    
     49498 97%    0  0%        0  0%        0  0%        0  0%        0  0% 
     link         symlink      mkdir        rmdir        readdir      fsstat 
     0  0%        0  0%        0  0%        0  0%        75  0%       4  0%     

The output may be interpreted using the following guidelines.

• badcalls > 0 - RPC requests are being rejected by the server. This could indicate authentication problems caused by having a user in too many groups, attempts to access exported file systems as root, or an improper Secure RPC configuration.
• nullrecv > 0 - NFS requests are not arriving fast enough to keep all of the nfsd daemons busy. Reduce the number of NFS server daemons until nullrecv is not incremented.
• symlink > 10% - Clients are making excessive use of symbolic links that are on file systems exported by the server. Replace the symbolic link with a directory, and mount both the underlying file system and the link's target on the client.
• getattr > 60% - Check for non-default attribute caching (noac mount option) on NFS clients.

On the client side use the command nfsstat -c to display the client statistics. For example:

     # nfsstat -c

     Client RPC:
     calls      badcalls   retrans    badxid     timeout    wait       newcred
     369003     62         1998       43         2053       0          0 

     Client NFS:
     calls        badcalls     nclget       nclsleep     
     368948       0            368948       0            
     null         getattr      setattr      root         lookup       readlink  
     0  0%        51732 14%    680  0%      0  0%        95069 25%    542  0% 
     read         wrcache      write        create       remove       rename 
     210187 56%   0  0%        2259  0%     1117  0%     805  0%      337  0%   
     link         symlink      mkdir        rmdir        readdir      fsstat    
     120  0%      0  0%        7  0%        0  0%        5510  1%     583  0% 

This output may be interpreted using the guidelines given below.

• timeout > 5% - The client's RPC requests are timing out before the server can answer them, or the requests are not reaching the server. Check badxid to determine the problem.
• badxid ~ timeout - RPC requests are being handled by the server, but too slowly. Increase timeo parameter value for this mount, or tune the server to reduce the average request service time.
• badxid ~ 0 - With timeouts greater than 3%, this indicates that packets to and from the server are getting lost on the network. Reduce the read and write block sizes (mount parameters rsize and wsize) for this mount.
• badxid > 0 - RPC calls on soft-mounted file systems are timing out. If the server is running, and badcalls is growing, then soft mounted file systems should use a larger timeo or retrans value.

---

Read More on http://www.cs.bgu.ac.il/~arik/usail/network/nfs/tips.html