3

Today I found something really interesting (at least to me) on one of our test servers:

I can change into an existing directory from my actual working directory using a relative path, but that very same directory is not listed when using ls -a.

Here is the shell session (as root):

$ pwd
/you/are/here
$ ls -a
. ..                       <-- Note: "somedir" is not shown to root
$ echo $CDPATH

$ cd somedir               <-- But still: "cd" works fine
$ pwd
/you/are/here/somedir
$ cd ..
$ pwd
/you/are/here
$ ls -a
. ..

Could someone tell me, how is this possible at all? I have checked: ls is from /bin/ls, and pwd is /bin/pwd, both from their original package (I mean: not hacked).

/you is a mounted EMC disk (ext3). And somedir exists as I can list the contents of it (there are several subdirs, files). Its name does not start with a dot.

Some more shell session, with more info about the commands and the ls output:

root@U-TEST@AT$/bin/ls -ali
total 4
16515074 drwxrwxr-x  2 U8000966 test 2048 Sep  1 07:39 .
16515073 drwxrwxr-x  3 U8000966 test 2048 Apr 27  2006 ..
root@U-TEST@AT$ls -ali somewhere | head -5
total 182
16515075 drwxrwxr-x  43 U8000966 test  2048 Sep  1 07:39 .
16515074 drwxrwxr-x   2 U8000966 test  2048 Sep  1 07:39 ..
16519169 drwxrwxrwx   4 U8000966 test  2048 Jul 25  2007 AAA
16515124 drwxrwxr-x   3 U8000966 test  2048 May 12  2006 BBB
root@U-TEST@AT$type ls
ls is aliased to `/bin/ls $LS_OPTIONS'
root@U-TEST@AT$type pwd
pwd is a shell builtin
root@U-TEST@AT$/bin/pwd
/you/are/here
root@U-TEST@AT$cd somewhere
root@U-TEST@AT$/bin/pwd
/you/are/here/somewhere
root@U-TEST@AT$type cd
cd is a shell builtin

Please note the Total 4 after the first ls -ali. (I don't know if it's relevant...)

Some more tests:

root@UR-TEST@AT$ls
.  ..
root@U-TEST@AT$touch somewhere/testfile
root@U-TEST@AT$ls
.  ..
root@U-TEST@AT$cp somewhere/testfile ./
root@U-TEST@AT$ls
.  ..  testfile
root@U-TEST@AT$du .
2       .
root@URBIS-TEST@AT$

And EMC is: http://www.emc.com/products/family/disk-library-family.htm , but they are only a disk provider in this case, with hard disks, formatted as ext3.

UPDATE

(Sorry, but yesterday I had to leave)

I did check echo *, and its output is: . ... Here are the LS_OPTIONS: -a -N --color=tty -T 0.

I had checked the automount thing mentioned by Gilles, but as I had changed to somewhere and issued a mount|grep somewhere there were no output.

Here is the lsattr and strace output as suggested: http://gist.github.com/566947

15
  • Anyone else not able to follow this question? Can you please revise as I cannot understand what you are asking.
    – Chris
    Sep 2, 2010 at 12:20
  • I assume that somedir indeed exists? Any clues when typing ls -a /you/are/here/somedir?
    – Arjan
    Sep 2, 2010 at 12:23
  • 1
    @Chris, ls -a does not list somedir (to root), but still cd somedir works just fine.
    – Arjan
    Sep 2, 2010 at 12:24
  • 2
    ls is aliased to /bin/ls $LS_OPTIONS. So, what does echo $LS_OPTIONS give you then? (Though I doubt that is relevant while you were also explicitely using /bin/ls with the same result, and the intriguing "total 4"...)
    – Arjan
    Sep 2, 2010 at 14:31
  • 1
    Do any of echo *, ls -ld somedir, ls -lb or ls -l|cat -v show anything helpful? Sep 2, 2010 at 14:46

8 Answers 8

3

Zsolt, please try these three steps:

    1. cd /
    2. exec bash
    3. /usr/bin/find /you/are/here -ls

It's probably time for fsck... :-(

But before that, you could try (after backing up anything inside somedir):
cd /you/are/here && mkdir somedir.
cd /you/are/here && ln somedir newdir (as root).

Also check mount | egrep -e 'somedir|you|are|here' for any oddities.

3
  • Did so, except the reboot/fsck cycle (as I'm not allowed to do so). The find also not found somewhere. I can create the the new somedir directory, and can link to it. I can't create a somewhere dir, as it complains, the file already exists. I can link to somewhere as well, and after an ls -l I can see a listing where there is a link to a directory within the same parent, and can't see the target. Sep 6, 2010 at 9:32
  • @Zsolt: Excuse me, I don't understand; did the mkdir somedir pass or fail? I wouldn't do anything more on this filesystem (or in the here directory) except backup. It's broken, or the cache of it (in the kernel) is corrupt. If you are lucky, fsck can relink the somedir directory again. If not, you could lose it. Sounds to me like it's time for some downtime. Try mv if you managed to link to it. (mv somedir olddir; mv newdir somedir) if that works, maybe you dare to unlink olddir?
    – MattBianco
    Sep 6, 2010 at 12:15
  • 1
    Fsck found it: most probably it was some filesystem error. The somedir directory was moved to lost+found. Now we are testing the disk for more errors. Sep 13, 2010 at 19:48
5

At the time I write this, you haven't ruled out the effects something in $LS_OPTIONS. GNU ls has a some file ignoring options, and ls -I foo -a still ignores foo. But the rest of my answer assumes you get the same results without $LS_OPTIONS.

The total 4 line is not, in fact, surprising. This is the total number of blocks used by . and ... If . is empty and .. is small and on a filesystem whose block size is equal to 4 ls blocks (which is common: GNU ls defaults to 1kB blocks, and ext[234] often uses to 4kB blocks), then the expected total is 0 + 1 * 4 = 4.

There is something unusual, but not unheard of, going on with the filesystem that /you/are/here is on. When you ask for the contents of /you/are/here (with opendir() and readdir(3)), the filesystem responds only with . and ..; yet when you assume that /you/are/here/somedir exists, you are told that it does. This is surprising but possible behavior.

A possible but highly unlikely explanation is a demon (as in Maxwell's demon, not as in daemon program) who moves somedir into place just when you access it and moves it out of the way when you list the directory. Thus the peculiarity you observe could be caused by an ordinary program that just happens to make the right guesses, it doesn't indicate anything wrong with the operating system.

In fact, the operating system probably is behaving in a peculiar way. A common culprit is an automounting system. The way an automounting system works is typically something like this:

  • A directory, say /you/are/here, is set up as a location for mount points. A special-purpose filesystem (possibly called autofs) is mounted there.

  • When you try to access an entry in /you/are/here, say /you/are/here/somedir, the filesystem driver tries to mount the somedir filesystem. For example, it might look for a line like somedir = /dev/foo or somedir = server:/loca/tion in its configuration file and mount the indicated device or NFS location as /you/are/here/somedir.

  • When you list the directory /you/are/here, you see a subdirectory for each filesystem that is currently mounted.

  • When you stop using /you/are/here/somedir, perhaps after a delay, the automounter unmounts somedir. So somedir no longer appears in the listing of /you/are/here.

1
  • 1
    Thanks for the Maxwell's demon link, interesting read, and the automount explanation is a possibility. Yet, I assume, if I change to somewhere, then issue a mount|grep somewhere and it's mounted at that time, it should be listed, which is not the case. Sep 3, 2010 at 8:55
3

What does "which ls" or "alias" show? Perhaps your ls command is being overridden by an alias or the like? Running the following might rule that out:

/bin/ls -a /you/are/here

Some background:

An alias takes precedence over /bin/ls, yet which ls still shows /bin/ls. You can recreate what I'm referring to by this:

  1. Create a bash script called ls, holding:

    #!/bin/bash
    echo this is not ls
    
    • Create an alias to the bash script:

      alias ls='~/ls'

    • Now, run ls. You should get "this is not ls", yet which ls shows /bin/ls.

It might also be beneficial to rule out autofs/automount.

After reviewing the posted updates...

Perhaps LS_OPTIONS contains a --hide or --ignore?

~/dirtest$ ls -a
.  ..  somewhere
~/dirtest$ ls -a --ignore='some*'
.  ..

What does...

echo $LS_OPTIONS

...show? Maybe even try...

unset LS_OPTIONS

...then re-run the ls -a.

It could be that LS_OPTIONS is being set in your .bashrc or other config file. (maybe even at the global level, from /etc/bash* (or whichever appropriate config file for the shell in question: .login, .profile, etc)

5
  • 1
    I'd guess I have checked: ls is from /bin/ls, and pwd is /bin/pwd, both from their original package refers to that?
    – Arjan
    Sep 2, 2010 at 13:09
  • 1
    Still, @Zsolt, explicitly running /bin/ls -a /you/are/here might rule out this, and is easily tested?
    – Arjan
    Sep 2, 2010 at 13:33
  • 1
    Not necessarily. An alias takes precedence over /bin/ls... yet 'which ls' still shows /bin/ls. I'd rule out aliases and such, still. You can recreate what I'm referring to by this: Create a bash script called "ls". Add the typical #!/bin/bash and a second line with something like: echo this is not ls Create an alias to the bash script... something like: alias ls='~/ls' Now, run ls. You should get "this is not ls", yet "which ls" shows /bin/ls. It might also be beneficial to rule out autofs/automount...
    – Matt
    Sep 2, 2010 at 13:40
  • Yep, I'd think the /bin/ls -a /you/are/here would be a great test.
    – Matt
    Sep 2, 2010 at 13:42
  • Did so, question updated. Sep 2, 2010 at 14:06
2

Can you try other tools that might walk the directory tree?

For example du -h to get the size of directories? What behaviour do you observe if you launch that from the drive root, or from /you/are/here as compared to /you/are/here/somedir?

Also - can you create a file in /you/are/here/somedir? If so, does it persist once you move out of the directory? Does creating a file cause the directory to be visible to ls command?

0

The following may or may not help, but I would like to see the output of the following two commands:

$ lsattr -av somedir
$ strace ls -a somedir
1
0

Perhaps some process is doing something with the directory. Have you tried lsof dirname, lsof /path/to/parent/* and lsof dirname/*?

What about stat dirname?

What about readlink -ev dirname?

I see you did type ls - what does type -a ls show?

Does $PATH include anything unusual? Does it include . (dot)?

What shell are you using? Have you tried doing your original ls dirname in another shell (especially after a completely new session/login?

If you're using a shell with tab completion will it do a completion for the directory name?

Do you have access to a file manager either GUI or a text-mode on such as mc (Midnight Commander) or emacs dired? Can they navigate into and out of this directory?

I would seriously suspect that something is corrupted or hacked.

0

LS help provides this, which may be of some use.

-a, --all                  do not ignore entries starting with .
-A, --almost-all           do not list implied . and ..
    --author               with -l, print the author of each file
-1

At this point, it seems to me the most likely thing is that your server was hacked and a rootkit was loaded that keeps the directory from showing up in listings. The root kit would do this with a kernel module that hooks or redirects the readdir and getdents system calls. If the module stays resident then it is probably also cloaking itself from showing up in /proc/modules or by lsmod.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .