73

Let's say I want to download something with wget but the website that has the files I need redirects to a site which automatically chooses a mirror for me (and there's no static file URL provided).

Downloading from such sites works with Firefox, but how do I get real url (not only the link to the redirect page) so I could download these files with wget?

5
  • 3
    Breakthrough's answer works, but what's the point then? Just let FF finish the download. Can you add more details to your question? I'm going to guess that you're trying to automate downloading via some sort of script, and not use wget just for gits-n-shiggles.
    – Yoopergeek
    Commented Aug 21, 2009 at 16:40
  • 2
    when you download with firefox right click on your download in downloads windows and select copy download link
    – Am1rr3zA
    Commented Sep 10, 2009 at 13:11
  • 2
    @Yoopergeek: His answer helps if the file is extremely big and you want to put it into a download manager. ;)
    – cp2141
    Commented Sep 10, 2009 at 19:17
  • @Yoopergeek: The OP also mentioned "wget" which is a linux/unix command-line tool to transfer a file without any windowsy UI at all. Tools like this come in handy when you're running a virtual machine in the cloud and need to download software. Since wget is just trying to get a file, not display advertising while it waits for a mirror to be selected, the direct url is essential.
    – mightypile
    Commented Jul 1, 2014 at 23:29
  • 1
    Cliget addon gives you a curl command to copy and paste in the download dialog. addons.mozilla.org/en-US/firefox/addon/cliget
    – Dagelf
    Commented Aug 9, 2017 at 5:24

15 Answers 15

66

When you've started the transfer in Firefox, cancel it, and right click the download and hit "Copy download link". If you're using an older version, right click it and click on "Properties", and the link will be in the window which opens.

In Chrome - run download as normal - then go to Menu - Downloads - and you should see the direct link which was used. Or press Ctrl + J to open the window.

7
  • 1
    Yeah, that would be a lot simpler than what I suggested!
    – arathorn
    Commented Aug 21, 2009 at 15:09
  • my thoughts as well, arathorn. I had started typing up a response to use the 'net' tab in Firebug, but this is far simpler!
    – Matt
    Commented Aug 21, 2009 at 15:10
  • 1
    This does not seem to work for me on github (I tried in Firefox, Chrome, Safari). eg.: nodeload.github.com/jverzani/gWidgets2/tarball/master Commented Apr 23, 2012 at 1:39
  • cmd+shift+j on Mac Commented Jun 3, 2014 at 18:53
  • This works only if the file you are downloading is big. if the file is small, it is pain to use this method. I managed to use this method and got the link from paused download. I get exactly the same url and what I copy from a webpage. However, it is not direct link to a file. accessdata.fda.gov/scripts/sda/…
    – Valentyn
    Commented Jan 30, 2023 at 16:20
18

You can use the LiveHTTPHeaders extension to determine the actual URL of the file being downloaded. (Keep an eye on the GETs in particular.)

3
  • 1
    that's cool idea, i'll try this. i would vote up if not my low rep ;)
    – Phil
    Commented Aug 21, 2009 at 15:08
  • anything for chrome?
    – Anwar
    Commented Sep 16, 2015 at 10:12
  • I know this is an old thread but wanted to give my +1 for this response. The best solution is definitely acceptable for most cases. In mine, I have an inventory web gui that I wanted to easily export all my devices from in CSV format using their export button. To make this scriptable, I took your "keep an eye on GET" advice and checked the network tab in F12 to see the file that was being downloaded by export. From there, in chrome at least, you can right click that file and do copy > copy as cURL. I substituted the device's name as a variable in the link & put it in a for loop in bash to curl Commented Aug 7, 2019 at 15:06
16

While I like wget too, I use another similar tool: cURL. cURL specifically has a feature to follow redirects (-L / --location), and it's also free like wget. I suggest keeping both in your toolbelt; they have some complementary features.

Here's an interesting article I came across on how to use either wget or cURL to download from a site that uses cookies for authentication. There's a mention of the cURL --location feature to follow redirects. (In fact, based on no special flag mentioned for wget, it would seem to me that wget may follow redirects implicitly..?)

0
16

This is the answer you have been looking for!

There is only one fool-proof one-click solution: cliget

It gives you a curl command with headers, cookies and all, with a copy to clipboard button, right on the download dialog.

For Firefox: https://addons.mozilla.org/en-US/firefox/addon/cliget

For Google Chrome: https://chrome.google.com/webstore/detail/curlwget/dgcfkhmmpcmkikfmonjcalnjcmjcjjdn/related

5
  • 2
    Thank you! It worked like a charm Commented Sep 20, 2019 at 13:04
  • 2
    Great this work well.
    – Ahdee
    Commented Dec 16, 2019 at 16:36
  • 1
    Thank you! This should be the accepted answer! No other solution works for login only assets Commented Apr 28, 2022 at 10:26
  • I have gotten this addon for Google Chrome but I don't understand how to use it.
    – Valentyn
    Commented Jan 30, 2023 at 16:23
  • 1
    @Valentyn After you download a file, you cancel the download, then you click on the cliget icon, then it gives you a choice of 3 commands to copy, with the url and request details included, which if you paste it in the terminal on the machine where you want to download the file, the file will be downloaded with all the same cookies and request and login details as in your browser.
    – Dagelf
    Commented Jan 31, 2023 at 17:13
4

In those situations, First I starts the actual download and pause it and copy the URL.

3
  • 1
    copy it from where?
    – Phil
    Commented Sep 10, 2009 at 13:18
  • 1
    If you pause the download in the firefox download window before it completes, you can right-click and grab the URL (have done that to fire the URL on wget sometimes).
    – nik
    Commented Sep 10, 2009 at 13:19
  • You can usually right click on the file in the download dialog or page, and just choose "Copy URL". But that URL often doesn't work without the right cookie, which is where my answer comes in.
    – Dagelf
    Commented Sep 26, 2022 at 4:55
4

I usually use the firefox DownThemAll addon when there are a lot of links to download and I need to select specific URLs.

It shows the full URL and allows you to add in paused mode so you can start the download when you like.

As an additional feature, you can grab the URL and skip the download altogether.
However, if you need to download the file, DownThemAll is quite good as an accelerator.

2

Can't you use wget directly using the --trust-server-name flag ? It will download and save using the filename it redirects to.

1

Your question is a bit unclear. There are 2 urls here : the one you use, and the one that came back after redirection. The first you should know, since it originated from your machine, while the second is the one that displayed in your browser. If the question pertains to where the file is on the redirected server, then there's no way to know.

If you're trying to debug the calls issued towards the server, you can use the firefox add-on of firebug, older versions of it exist here : https://addons.mozilla.org/en-US/firefox/addons/versions/1843.

3
  • 1
    honestly, i don't understand what's so unclear in my question, some posters below even actually understood ;)
    – Phil
    Commented Sep 10, 2009 at 13:24
  • If your only problem is just getting the url, then pausing the download only works for large files, unless you're very fast. You can much much simpler just find this information in the download manager's list : right-click and do Copy download link.
    – harrymc
    Commented Sep 10, 2009 at 13:55
  • To the downvoter: Don't waste your time here - this is a very old post and may not apply to you.
    – harrymc
    Commented Aug 9, 2017 at 16:06
1

You can use Curl to download a redirected URL:

curl --location http://domainName/fileName 2> /dev/null > fileName
0

You can use IDM (Internet Download Manager) when you click on the not direct download link it will redirect the link and showing up the direct link.

0

Have you tried to pass the redirect URL to wget? Sometimes they work for me.

Here a example from SourceForge:

$ wget http://sourceforge.net/projects/emule/files/eMule/0.49c/eMule0.49c-Installer.exe/download?use_mirror=ufpr,Centro%20de%20Computacao%20Cientifica%20e%20Software%20Livre
--14:23:20--  http://sourceforge.net/projects/emule/files/eMule/0.49c/eMule0.49c-Installer.exe/download?use_mirror=ufpr,Centro%20de%20Computacao%20Cientifica%20e%20Software%20Livre
Resolvendo sourceforge.net... 216.34.181.60
Connecting to sourceforge.net|216.34.181.60|:80... conectado!
HTTP requisição enviada, aguardando resposta... 302 Found
Localização: http://downloads.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe?use_mirror=ufpr [seguinte]
--14:23:21--  http://downloads.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe?use_mirror=ufpr
Resolvendo downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... conectado!
HTTP requisição enviada, aguardando resposta... 302 Found
Localização: http://ufpr.dl.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe [seguinte]
--14:23:21--  http://ufpr.dl.sourceforge.net/project/emule/eMule/0.49c/eMule0.49c-Installer.exe
Resolvendo ufpr.dl.sourceforge.net... 200.236.31.1, 200.17.202.1
Connecting to ufpr.dl.sourceforge.net|200.236.31.1|:80... conectado!
HTTP requisição enviada, aguardando resposta... 200 OK
Tamanho: 3342809 (3,2M) [application/x-msdos-program]
Saving to: `eMule0.49c-Installer.exe'

100%[====================================================================================>] 3.342.809   2,35M/s   in 1,4s

14:23:23 (2,35 MB/s) - `eMule0.49c-Installer.exe' saved [3342809/3342809]

The version I use:

$ wget --version
GNU Wget 1.10.2 
0

There was an addon for Firefox called Splitlink. It doesn't work for Firefox 3+, but it did work in Firefox 2. If you can find it anywhere (it's not on the official addons page anymore), it will give you more information about the real URL.

-1

The firefox copy url is definitely simplest. wget (as the question was tagged) is also a nice solution because you can see all the intermediate bounces, not just the final url:

$ wget 'http://sourceforge.net/projects/pdfcreator/files/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe/download'
--2009-09-10 09:59:53--  http://sourceforge.net/projects/pdfcreator/files/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe/download
Resolving sourceforge.net... 216.34.181.60
Connecting to sourceforge.net|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://downloads.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe?use_mirror=voxel [following]
--2009-09-10 09:59:54--  http://downloads.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe?use_mirror=voxel
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://voxel.dl.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe [following]
--2009-09-10 09:59:54--  http://voxel.dl.sourceforge.net/project/pdfcreator/PDFCreator/PDFCreator%200.9.8/PDFCreator-0_9_8_setup.exe
Resolving voxel.dl.sourceforge.net... 74.63.52.167, 69.9.191.19, 69.9.191.18, ...
Connecting to voxel.dl.sourceforge.net|74.63.52.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17695920 (17M) [application/octet-stream]
Saving to: `PDFCreator-0_9_8_setup.exe'

10% [=======> ] 1,818,064 609K/s

1
  • Only works if cookies aren't needed and the site actually has redirects.
    – Dagelf
    Commented Mar 17, 2022 at 7:31
-1

If you want a download manager to handle the download instead of firefox, you could use the Flashgot firefox addon.

-1

You could use the netcat tool (which is often included with linux distributions) on port 80 of the website. This shows the full raw HTTP response, including the headers which contain the redirect destination URL. This is one of the most direct ways to get the redirect location, since you are doing the same thing as downloaders do when they follow redirects. Here is an example for the URL http://www.example.com/example.

printf 'GET /example HTTP/1.1\r\nHost: www.example.com\r\n\r\n' | netcat www.example.com 80

...or...

Alternatively, many browsers show the actual download URL in the Downloads section. In these, you can start the download and cancel it right away, then copy the URL (from the download history).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .