13

I'm creating a simple Bash script to extract the file modification time/date of a remote file via HTTP.

Example file: http://example.com/bar/example.pdf

Can this be done without downloading the actual file? If not, what's the best alternative?

13

To be honest, not directly.

You will have to fetch data from the remote site to get information about the file. Usually this is done with a HEAD request, but some (most?) servers haven't implemented it correctly and deliver the whole file, just like doing a GET request. Assuming that you have curl installed:

curl -s -v -X HEAD http://foo.com/bar/baz.pdf 2>&1 | grep '^< Last-Modified:'

might give you what you want, but as said, it highly depends on the server.

| improve this answer | |
  • 6
    "Most"? I would be surprised if any of the popular HTTP servers violated the protocol in such a way. – user1686 Jul 14 '13 at 9:33
  • It should have changed of course. Some time ago, when I dealt with such problems, it was like that. However, time passes. If you find sites that are still running cgi whatever "apps", they are most likely not handling HEAD. Nevertheless, also those sites will give you the result, because they are supposed to deliver everything. – Karsten S. Jul 14 '13 at 9:39
  • 3
    I suggest using the --head option instead of -X HEAD as its more terse so the command would become: curl -s -v --head http://foo.com/bar/baz.pdf 2>&1 | grep '^< Last-Modified:' – Gautham C. May 28 '14 at 16:13
  • 1
    curl -svX HEAD is even more terse... – Karsten S. Sep 27 '14 at 10:09
  • 1
    @Hi-Angel No, there generally isn't. Wget uses the If-Modified-Since header to say "hey only send that file if it's newer than this date", it's then up to the server to implement and respect that header. If the server thinks the file hasn't changed it sends a 304 NOT MODIFIED response. – antonagestam Jul 18 '18 at 6:58
13

The server response does usually have Last-Modified field, you can check it without downloading the file. No need to use -X HEAD, there's a special option -I for that (the -s suppresses progress output):

curl -sI http://example.com/bar/example.pdf | grep -i Last-Modified

Also in my case there's no curl installed (I'm doing a script for an embedded device), just wget. The way with wget is:

wget --server-response --spider http://example.com/bar/example.pdf 2>&1 | grep -i Last-Modified

The --server-response prints headers, and the --spider option forces to not download pages, but rather check their existence.

| improve this answer | |
  • 2
    This curl is a better answer than the accepted one. Perhaps using grep -i since often the "last-modified" has a different case. – not2qubit Sep 14 '18 at 19:20
  • @not2qubit thx for the note, edited. – Hi-Angel Sep 15 '18 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.