Why does `ls` without any arguments (only flags) give me "no such file or directory"?

Question

When I pass --color=always to ls, it occasionally outputs a number of No such file or directory errors, like this:

~/svn/projects/submm/adda/scat$ /bin/ls --color=always
ls: cannot access adda_output_f89: No such file or directory
ls: cannot access adda_output_f150: No such file or directory
ls: cannot access adda_output_f183: No such file or directory
ls: cannot access adda_output_f186: No such file or directory
ls: cannot access adda_output_f190: No such file or directory
...

Later follow the contents of the directory, including the subdirectory adda_output_f89 coloured as a directory.

There is a process running that is operating on files in this directory, but I don't think it's doing anything with the directories that ls is mentioning.

It is not fully reproducible. I have so far not succeeded in finding out a pattern when it happens and when it doesn't happen. It appears to happen in waves. Perhaps a process is rapidly creating and removing directories, but I don't think that's true.

It appears to be happening only when I pass --color=always, but I am not 100% sure that this is the case. Normally I use an alias, ls='ls --classify --color=always --human-readable' where it does happen, but when I call /bin/ls it appears that it does not happen.

Edit:

ls -i gives for those files:

? adda_output1_f243/  ? adda_output_f243/

etc.

Edit:

This is a nfs filesystem.

What might cause this behaviour? Is it some kind of race condition?

Answer 1

As mentioned in the comments, ls in an NFS mount will result in two separate NFS calls, with a slight delay between them. If you suspect some process might be adding and removing entries in the directory I would definitely pursue that as an explanation. Here is how I would test that theory.

If the problem occurs some significant number of times you run ls (i.e., it would be easy to reproduce by manually running ls multiple times) you could simply:

{ ls;ls;ls; } 2>&1 | tee ls.out | grep 'No such file'

Re-run that command until you get the error, then inspect the ls.out file, find which files it was complaining about, then see if those files 1) exist in the first 1/3 of the output and 2) don't exist in the last 1/3 of the output. Since the files were (presumably) deleted while one of the ls commands was being run it shouldn't be too hard to figure out if they were there before and removed after.

If reproduction is significantly time-consuming you could write a script along the lines of (untested!):

#!/bin/bash

while /bin/true; do
  /bin/ls -lc >ls1.out 2>&1
  /bin/ls -lc >ls2.out 2>&1
  /bin/ls -lc >ls3.out 2>&1
  grep 'No such file' ls2.out >missing.out 2>&1
  if (( $? != 0 )); then
    echo "Found the error!"
    break
  fi
done

(Add sleeps if you're worried about the performance impact of running this in an unbridled tight loop.)

Run this in a screen session and check on it later. When the script exits reporting that it found the error, check missing.out to see which files ls couldn't find, then verify whether those files are listed in ls1.out but not ls3.out.

Don't forget to verify the ctime reported by ls, in case this mystery process happens to delete the file, then recreate a new file with the same name just in time for the third ls to list it. :)

If it turns out that for some reason the cause isn't that the process had deleted the file while ls was running, report back here and we'll figure out what else to try.