Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm looking for a solution between a proxy and wget, since neither fully addresses the issue I'm trying to solve.

I want to download files to a specific directory on my home machine (a Linux server - command-line only) from remote places (via smartphone, web browser, etc.). This sounds like a solution for wget, except many of the files I would download come from sources that create special download "sessions" for you via AJAX/Javascript (e.g. RapidShare.de, Hotfile.com).

I imagine I would have to use a proxy at some point in this process, but I don't want the download data to come to my remote device at all, since that would waste bandwidth. But, to use a proxy to access the sites and create the sessions would work for me.

So, solutions that don't work (out-of-the-box, at least):

  • Remote Desktop: Windows only
  • VNC: I only have command-line access; also, not so friendly with smartphones and the like
  • wget: download links generally only exist for a "session"; very difficult to emulate AJAX environment for sites
  • proxy: sends all data to remote client; mounting the remote directory would cause the data to take three hops (site to proxy, server/proxy to client, client to remote directory). I'm looking for a direct (site to remote directory) route for the download.

In my perfect world, I'm envisioning browsing to a site via a proxy, but then when the browser classifies something as a Download, it just downloads to the remote directory (or even opens a file Save dialog and lets me choose where to save it on the remote server). The solution doesn't have to have this level of simplicity, but hopefully this helps clarify the thrust of my question.

share|improve this question
    
How about SSH coupled with a Text-based web browser ? – harrymc Dec 6 '15 at 17:29

wget has options to save and load cookies. Sometimes depending on the site if you use wget to load the webpage that the link is on and tell it to save cookies, then make another request for the actual file and have it load the cookies it will work.

It's a two step process like this:

wget --cookies=on --keep-session-cookies --save-cookies=cookie.txt http://domain.com/page_with_link.html
wget --referer=http://domain.com/page_with_link.html --cookies=on --load-cookies=cookie.txt --keep-session-cookies --save-cookies=cookie.txt http://domain.com/some_file_to_download.zip
share|improve this answer
    
This could help. Most of the sites have an intricate Javascript procedures to generate the links, enter captchas, etc., so I would definitely have to make more wget calls, but it's a start. – palswim May 18 '11 at 22:56

For my needs, I have settled on JDownloader, an open-source Java application to handle downloads from various sites, including direct HTTP downloads. It handles page wait timers and presents the captchas to me when I need to solve one.

It has a slick GUI, but I have it running headless (no GUI) on my server machine and interface with it via JDownloader's web interface or the Android app, meaning I can initiate downloads from any browser or even my phone.

It still doesn't work perfectly with some sites, but it has certainly worked well for me overall. In my opinion the Android app has an even better interface than the browser site.

share|improve this answer

Navigate to the address and download the file with Links over a SSH connection. A GNU Screen could also be used to finish the download without having the SSH connection still open.

share|improve this answer

offhand the only software for downloading off one click hosters that you can run on a linux cli that i remember is ploughshare - its pretty much a dead simple, and handles loads of file hosts. Don't forget to run it in screen or tmux so you can detach the session rather than needing to keep the ssh session open

If you didn't mind a GUI i'd suggest looking at slimrat

share|improve this answer

Another possibility (apart from maybe custom modifying ziproxy to add a "save these types locally" option) would be to use a X server.

There are X servers for smartphones, and for Windows too. This answer hints at HTTP client X servers.

So on your typical Android smartphone you would install Android X server, and on the Linux box a simple Firefox browser theme for small displays. You now have a local screen on the remote process, and any download will happen remotely. Very similar to VNC, but friendlier to smartphones.

On large screen devices you'd still fire up X server, but this time launching a normal Firefox instance.

share|improve this answer

You have the following options:

  • run a virtual machine on the remote server, install your favourite OS to the VM, and run your favourite browser through the graphical console of the VM, accessed via the remote desktop protocol supported by your VM software (for example, VirtualBox can be accessed with rdesktop), optionally tunneled through SSH,
  • run your favourite browser natively on the server, via remote X protocol, optionally tunneled through SSH (it's wise to use the compression option of SSH as well),
  • run a text browser (for example: lynx) through SSH (optionally in screen),
  • run a dumb HTTP client (for example: wget, curl) through SSH (optionally in screen),
  • run an HTTP proxy (for example: squid) with non-standard settings: you'd probably have to increase the maximum allowed file size to be stored on disk, and it's practical to decrease the abort limit (the point at which the proxy will finish downloading even if the client has aborted, so that you can fire up the downloading, and then stop the client early, and let the proxy server finish the job),
  • write your own HTTP client :)
share|improve this answer

Install owncloud server (https://owncloud.org/) on your linux box, and owncloud clients on your devices (smartphone, remote pc, whatever). Config, authenticate, and Voila! you can now save downloads to your owncloud drive...

share|improve this answer
    
This does not answer the question: the OP wants its home machine to perform the download, while being piloted by the smartphone. The owncloud solution would have the smartphone downloading the file, and reuploading it to owncloud, effectively doubling the bandwidth required. – lserni Dec 6 '15 at 17:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.