9

The POSIX shell standard says on this site

http://pubs.opengroup.org/onlinepubs/9699919799/

about how shells use PATH to look for executables:

"The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found."

Well, this is not how this appears to work in real POSIX implementation:

man which says:

"returns the pathnames of the files (or links) which would be executed in the current environment, had its arguments been given as commands in a strictly POSIX-conformant shell. It does this by searching the PATH for executable files matching the names of the arguments. It does not follow symbolic links."

OK, let's look at this situation:

$ pwd /home/mark

$ echo $PATH /home/mark/bin:...

$ ls -l bin/foobar
lrwxrwxrwx 1 mark mark 18 Dec 12 22:51 bin/foobar -> /home/mark/foobar1
$ touch foobar1
$ which foobar
$ chmod a+x foobar1
$ which foobar
/home/mark/bin/foobar

OK, here is a symbolic link in PATH with the correct name, and it is reported by ls to be executable.

which does not look at it at all, but is only interested in what it points to.

That despite the fact that both man which explicitly says that it does not follow symbolic links (and indeed we see it doesn't, because which foobar does not print foobar1), and also that the POSIX shell documentation quoted above, never ever mentions following symlinks in the PATH algorithm.

So, is which and the existing shells wrong, or am I not understanding the documentation?

TO CLARIFY:

I know and can explain the existing behaviour. My question is not "how does this work?". That I know.

My question is about documentation: where is my mistake in following the documentation that I quoted. Or is the documentation wrong?

MOTIVATION: Why do I care?

Well, I am an implementer. Different implementers have different requirements. For me, the requirement is that the word of the current POSIX standard MUST be followed EXACTLY (or, more precisely, the best it can be, because, the standard itself is somewhat buggy). Like as it were the word of God.

Now, the standard wording is pretty clear - following symlinks is not mentioned, where in many other places, it is mentioned where it needs to be done. So in this case, don't.

However, I always double check how dash and bash behave, just to make sure. Now of course, there is a little problem here as well, dash even though it is billed as POSIX, has plenty of small bugs with conformance to POSIX. bash, I have yet to find any bugs with POSIX, but... bash isn't actually POSIX, it is much more than that.

So there you have it. That is why I care.

  • You don't understand: which doesn't follow symlinks on files. $PATH can contains symlinks. Try which sh. – Ipor Sircer Dec 13 '17 at 7:33
  • OK but, in my case $PATH does not have any symlinks. – user322908 Dec 13 '17 at 7:44
  • In almost all situations, symlinks are followed transparently. The cases where they aren't are usually mentioned explicitly (e.g. system calls like lstat(2)), following them is not usually stated. For instance, the description of open(2) only mentions symlinks when talking about the behavior of O_CREAT | O_EXCL. It doesn't need to state that the target file will be opened. – Barmar Dec 15 '17 at 17:51
10

The permissions of the symlink itself are irrelevant. You couldn't even change them if you tried.

What matters are permissions of the underlying file.

It is fine to have directories in your PATH include symlinks to executables. In fact, it is likely that many executables in your PATH are symlinks. For example, on debian/ubuntu-like systems:

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Jan 23  2017 /bin/sh -> dash

Documentation

From man chmod:

chmod never changes the permissions of symbolic links; the chmod system call cannot change their permissions. This is not a problem since the permissions of symbolic links are never used. However, for each symbolic link listed on the command line, chmod changes the permissions of the pointed-to file. In contrast, chmod ignores symbolic links encountered during recursive directory traversals. [Emphasis added.]

Example

The shell has a test, -x, to determine if a file is executable. Let's try that:

$ ls -l
total 0
lrwxrwxrwx 1 john1024 john1024 7 Dec 12 23:36 foo -> foobar1
-rw-rw---- 1 john1024 john1024 0 Dec 12 23:36 foobar1
$ [ -x foo ] && echo foo is executable
$ chmod +x foobar1
$ [ -x foo ] && echo foo is executable
foo is executable

So, just like you found with which, the shell does not consider a softlink executable unless the underlying file is executable.

How which works

On a Debian system, which is a shell script. The relevant section of the code is:

 case $PROGRAM in
  */*)
   if [ -f "$PROGRAM" ] && [ -x "$PROGRAM" ]; then
    puts "$PROGRAM"
    RET=0
   fi
   ;;
  *)
   for ELEMENT in $PATH; do
    if [ -z "$ELEMENT" ]; then
     ELEMENT=.
    fi
    if [ -f "$ELEMENT/$PROGRAM" ] && [ -x "$ELEMENT/$PROGRAM" ]; then
     puts "$ELEMENT/$PROGRAM"
     RET=0
     [ "$ALLMATCHES" -eq 1 ] || break
    fi
   done
   ;;
 esac

As you can see, it uses the -x test to determine is a file is executable.

POSIX specifies the -x test as follows:

-x pathname
True if pathname resolves to an existing directory entry for a file for which permission to execute the file (or search it, if it is a directory) will be granted, as defined in File Read, Write, and Creation. False if pathname cannot be resolved, or if pathname resolves to an existing directory entry for a file for which permission to execute (or search) the file will not be granted. [Emphasis added.]

So, POSIX checks what the pathname resolves to. In other words, it accepts symlinks.

POSIX exec function

The POSIX exec function follows symlinks. The POSIX spec goes on at length to specify error conditions it may report if symlinks are circular or too deep, such as:

[ELOOP]
A loop exists in symbolic links encountered during resolution of the path or file argument.

[ELOOP]
More than {SYMLOOP_MAX} symbolic links were encountered during resolution of the path or file argument.
[ENAMETOOLONG]
As a result of encountering a symbolic link in resolution of the path argument, the length of the substituted pathname string exceeded {PATH_MAX}.

  • I KNOW everything you wrote in your answer. I know how things "work". That is not my question. My question is about documentation. Point me where am I not understanding the documentation. Or tell me that the documentation is incorrect. – user322908 Dec 13 '17 at 7:46
  • @user322908 On most systems, which is a shell script. Thus, it is likely just using the -x test that I showed. According to POSIX, -x tests whether a file "resolves" to an executable. If you are seeing something different, where are you looking? – John1024 Dec 13 '17 at 7:55
  • Thank you and I am sorry for being such a pain... I realize that my question is different than 99% questions so it is hard to understand what I am after. Again, I am not interested in "how" which works, or whether it is -x or something else. I am interested to know where am I not following correctly the documentation that I quoted. – user322908 Dec 13 '17 at 7:58
  • 4
    @user322908 The POSIX exec function follows symlinks. That seems to make pretty clear that symlinked files can be executables under POSIX. – John1024 Dec 13 '17 at 8:13
  • 2
    I also checked the Ubuntu 17.10 man which which says "It does not canonicalize path names." That doesn't mean it doesn't follow the links. That only means, as you observed, that it doesn't "canonicalize" the names. – John1024 Dec 13 '17 at 8:28
3

In this case symlinks are followed transparently, without canonicalizing the final path. In other words, which does not care about whether /home/mark/bin is a symlink or not. What it cares about is whether the file /home/mark/bin/foobar exists or not. It does not need to manually flatten symlinks along the path – the OS can do that just fine on its own.

And indeed, when which asks about file information of /home/mark/bin/foobar, the OS notices /home/mark/bin being a symlink, follows it, and successfully finds foobar in the target directory.

This is the default behavior unless the program uses open(…, O_NOFOLLOW) or fstatat(…, AT_SYMLINK_NOFOLLOW) to access the file.

[comments merged in]

While you say that shell utilities do it on a case-by-case basis, it is not the same with kernel syscalls: all file-related calls do follow symlinks by default, unless the "nofollow" flag is given. (Even lstat follows symlinks in all path components except the last one.)

When the specification does not explicitly mention what to do with symlinks, it implies the default behavior will be used. That is, a shell following the path algorithm neithers resolve symlinks manually nor does it explicitly opt out of the OS doing the same. (It just concatenates each $PATH component with the executable name.)

When the which(1) manual page says it does not follow symlinks, it can mean several things, but the GNU coreutils version states it this way:

Which will consider two equivalent directories to be different when one of them contains a path with a symbolic link.

That is much narrower in scope – it only means which will not try to manually canonicalize all paths to weed out duplicates, but it does not imply that the tool will opt out of symlink following by the OS in general. For example, if /bin is a symlink to /usr/bin, running which -a sh will return both /bin/sh and /usr/bin/sh.

  • Yes thank you, I know all of this. My question is not how things "work". I know how they work. That is not my point. My point is about documentation. Where am I following the documentation incorrectly. Or is the documentation incorrect. – user322908 Dec 13 '17 at 7:47
  • 2
    You are understanding the documentation incorrectly – if it doesn't mention following symlinks, that means it doesn't manually resolve the symlinks, but the regular OS behavior still applies. The GNU which manual page states it differently: "Which will consider two equivalent directories to be different when one of them contains a path with a symbolic link." – grawity Dec 13 '17 at 7:51
  • OK, that is better thank you! I am trying to understand... But... pardon my being a pain in the neck: "the regular OS behaviour" is NOT to always implicitly follow symlinks. There are plenty of utilities that don't. It's on a case-by-case basis. – user322908 Dec 13 '17 at 7:54
  • 1
    All kernel calls – chdir, open, chmod, execve... – will follow symlinks both in the path and the tail, unless you specify AT_SYMLINK_NOFOLLOW or similar. (lstat is the only one which does not dereference symlinks at the tail, but still does so for the remaining path.) Therefore the default behavior is to follow symlinks. For example, when a shell calls execve("/home/mark/bin/foobar", …) it will result in all symlinks being followed. – grawity Dec 13 '17 at 8:04
  • OK I think I am buying the execve argument, actually, in my implementation it is execl(), same thing. Please if you include this in your answer I will accept. – user322908 Dec 13 '17 at 8:13
1

The shell conforms to its documentation in that it follows the rules for pathname resolution. which conforms to its documentation. The two do slightly different things.

The output of which is the link's file name and path, not the path to what the symlink points to. This is spelled out in the man page.

When a command is executed, the link is "followed" as per Section 4.13 Pathname Resolution in the same. The relevant clause for executing a file is:

In all other cases, the system shall prefix the remaining pathname, if any, with the contents of the symbolic link, except that if the contents of the symbolic link is the empty string, then either pathname resolution shall fail with functions reporting an [ENOENT] error and utilities writing an equivalent diagnostic message, or the pathname of the directory containing the symbolic link shall be used in place of the contents of the symbolic link. If the contents of the symbolic link consist solely of characters, then all leading characters of the remaining pathname shall be omitted from the resulting combined pathname, leaving only the leading characters from the symbolic link contents. In the cases where prefixing occurs, if the combined length exceeds {PATH_MAX}, and the implementation considers this to be an error, pathname resolution shall fail with functions reporting an [ENAMETOOLONG] error and utilities writing an equivalent diagnostic message. Otherwise, the resolved pathname shall be the resolution of the pathname just created. If the resulting pathname does not begin with a , the predecessor of the first filename of the pathname is taken to be the directory containing the symbolic link.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.