1

i'm downloading some measurements with wget, which are in the form of

http://www.somewhere.com/sub-somewhere052/image0001.jpg  
http://www.somewhere.com/sub-somewhere052/file0001.txt  

up to

http://www.somewhere.com/sub-somewhere052/image0099.jpg  
http://www.somewhere.com/sub-somewhere052/file0099.txt  

The problem is, these two numbers change from test to test (this was test 052), and each test has an unknown (in advance to me) number of files. However, they are all called the same (they're not called somewhere, of course) and that is known.

How can I get wget to "go through" sub-somewhere001/image0001.jpg to imagexxxx.jpg (where xxxx is unknown), until it reaches the one file that doesn't exist anymore, and then move to (after filexxxx.txt's) to somewhere002/... and so on?

In short, how to deal with the "go until there is nothing any more" situation?

3
  • 1
    Oh, wow ... the "somewhere" domain actually exist !??
    – Rook
    Mar 9, 2012 at 3:46
  • 1
    You're really supposed to use example.com for things like this - it's a reserved address.
    – Bob
    Mar 9, 2012 at 3:52
  • @Bob - Yeah, well ... the point is, I believe, clear and I doubt anyone's going to go seeing whether there is a file0099.txt somewhere out there (no pun indended) :)
    – Rook
    Mar 9, 2012 at 3:58

2 Answers 2

3

From man wget, you can see that it uses the usual Unix return value convention - 0 means no error, anything else is an error. Provided you don't expect other kinds of errors (e.g. network failure or things like that), i.e. you expect that if it doesn't download anything that means there's no file, you can use something like this:

get_tf_simulated() {
  t=$1
  if [ $t -lt 3 ]; then
    f=$3
    s=$((2 * $t))
    if [ $f -lt $s ]; then
      return 0
    fi
  fi
  return 1
}

get_tf_real() {
  tp=$2
  fp=$4
  inf=$5
  ext=$6
  # Get http://example.com/test<test number>/<image or file><file number>.<jpg or txt>
  wget -Otest$tp_file$fp_$inf.$ext http://example.com/test$tp/$inf$fp.$ext
}

get_tf() {
  echo --- Getting $*
  get_tf_simulated $*
  #get_tf_real $*
}

get_all() {
  get_tf $t $tp $f $fp image jpg
  ret_val=$?
  if [ $ret_val -ne 0 ]; then
    return $ret_val
  fi
  get_tf $t $tp $f $fp file txt
}

for t in {1..999}; do
  tp=`printf %3.3d $t`
  got_one=no
  for f in {1..9999}; do
    fp=`printf %4.4d $f`
    get_all $t $tp $f $fp
    if [ $? -ne 0 ]; then
      echo Failed, going next
      break
    fi
    got_one=yes
  done
  if [ $got_one == 'no' ]; then
    echo Nothing more
    break
  fi 
done

Uncomment the right line in get_all function. Currently, it will simulate it and the output will be like this (provided you saved the above to mkt.sh):

$ ./mkt.sh 
--- Getting 1 001 1 0001 image jpg
--- Getting 1 001 1 0001 file txt
--- Getting 1 001 2 0002 image jpg
Failed, going next
--- Getting 2 002 1 0001 image jpg
--- Getting 2 002 1 0001 file txt
--- Getting 2 002 2 0002 image jpg
--- Getting 2 002 2 0002 file txt
--- Getting 2 002 3 0003 image jpg
--- Getting 2 002 3 0003 file txt
--- Getting 2 002 4 0004 image jpg
Failed, going next
--- Getting 3 003 1 0001 image jpg
Failed, going next
Nothing more

Note I did not test the wget one, but you can use this to test on a couple of files:

wget -Otest$tp_file$fp_$inf.$ext http://example.com/test$tp/$inf$fp.$ext; echo $?

Just replace $tp, $fp, $inf and $ext as necessary, e.g. for similar example to what you gave:

wget -Otest052_file0001_file.txt http://www.example.com/sub-somewhere052/file0001.txt; echo $?

This should echo 8 for 404, from the man wget:

8   Server issued an error response.

If this works, then the script should work, hope there are no typos in that one line. :)

5
  • I'll try to copy that to windows. I'm using a port of wget on winxp, therefore having no man pages.
    – Rook
    Mar 9, 2012 at 5:25
  • @Idigas: here, poor fella, your own copy of wget, just for you: gnu.org/software/wget/manual/wget.html :)
    – akira
    Mar 9, 2012 at 5:49
  • @akira - i already have a copy of wget. what I don't have is a *nix shell ...
    – Rook
    Mar 9, 2012 at 6:00
  • @Idigas: yah, i meant MANUAL of wget .. but i spoiled the joke :)
    – akira
    Mar 9, 2012 at 6:01
  • @Idigas You can install Cygwin. It has a bash shell and wget, so you should be able to run the above without a problem. Mar 10, 2012 at 0:45
0

If the site returns a 404 response, wget will set the $? variable to a non-zero value (specifically 8, but who cares). You can test for that.

I find bash rather confusing, so here's one in Python (2.7.2). It should work, but I can't directly test without a handy website. it depends on the server returning a proper 404 response.

#! /usr/bin/python

basepath = "http://www.somewhere.com/sub-somewhere"
imgpre = "/image"
imgpost = ".jpg"
txtpre = "/txt"
txtpost = ".txt"

import os
import urllib2

directorynum = 1
filenum = 1

while True:
    pathdir = basepath + str(directorynum).zfill(3)

    if filenum == 1:
        try:
            os.makedirs(pathdir[7:])
        except OSError, e:
            print "Error creating directory: " + e.strerror

    pathimg = pathdir + imgpre + str(filenum).zfill(4) + imgpost
    pathtxt = pathdir + txtpre + str(filenum).zfill(4) + txtpost
    try:        
        print "Getting " + pathimg
        resp = respimg = urllib2.urlopen(pathimg)
        with open(pathimg[7:], "wb") as f:
            f.write(respimg.read())

        print "Getting " + pathtxt
        resp = resptxt = urllib2.urlopen(pathtxt)
        with open(pathtxt[7:], "w") as f:
            f.write(resptxt.read())

        filenum += 1

        continue
    except urllib2.HTTPError, e:
        if e.code == 404:
            print "Error: 404"
            print "Got " + str(filenum - 1) + " from directory " + str(directorynum) + ", incrementing directory."
            directorynum += 1
            filenum = 1
            continue
        else:
            print "An unexpected error (" + resp.code + resp.msg + ") has occurred."
            break

It should also run fine on Windows (just get rid of the #! /usr/bin/python and save as a .py file, though a python interpreter must be installed)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .