Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to download around 15,000 individual user files from an FTP backup space. I know the files exist because if I put a certain username that was not downloaded with the regular wget it can be downloaded.

I have tried ncftp and ftp

share|improve this question
3  
Can you explain this part? "if I put a certain username that was not downloaded with the regular wget it can be downloaded". It might be clear to you but I have no idea what you mean with it. –  Hennes Apr 6 '13 at 10:50
    
I ment a certain filename. –  Legit Apr 6 '13 at 11:41
    
wget gets the filenames to download from the HTML structure. If there is no index.html (or similar) to feed the names to wget then you either need to give it a list (e.g. with a text file and the -i option) or use a FTP client and use the mget and prompt commands. –  Hennes Apr 6 '13 at 12:19
2  
What is it today with people needing to download 15,000 files one of which is not retrieved by wget? superuser.com/questions/578790/… –  lserni Apr 6 '13 at 12:54
1  
Maybe the server limits the number of files it will send for a single request? –  Fred Apr 6 '13 at 13:33

1 Answer 1

FTP example as requested (can't put it in a comment and keep the formatting)

user@mycomputer:/home/somedir> ftp
ftp> Open storage.provider.tld
Trying 127.0.0.1...
Connected to localhost.
220 storage.provider.tld FTP server (Version 6.00LS) ready.
name: YourUSERNAME
331 Password required for YourUSERNAME
Password: TypeYourPassword
230 User YourUSERNAME logged in.
Remote system type is UNIX.
Using binary mode to transfer files.

ftp> prompt
Interactive mode off.

ftp> mget *
local: test remote: test
229 Entering Extended Passive Mode (|||60396|)
150 Opening BINARY mode data connection for 'test' (10 bytes).
100% |*************************************|    10     184.25 KB/s    00:00 ETA
226 Transfer complete.
10 bytes received in 00:00 (34.75 KB/s)
local: test.iso remote: test.iso
229 Entering Extended Passive Mode (|||64833|)
150 Opening BINARY mode data connection for 'test.iso' (0 bytes).
     0       0.00 KB/s
226 Transfer complete.
local: test.uue remote: test.uue
229 Entering Extended Passive Mode (|||50714|)
150 Opening BINARY mode data connection for 'test.uue' (0 bytes).
     0       0.00 KB/s
226 Transfer complete.
(etc etc for all files)

Without the prompt command you would need to confirm all files. Not the best idea when there is 15000 of them.

Also, my client started in binary mode (Using binary mode to transfer files.). If yours starts in ASCII mode then type bin.

you can get subsets. E.g. all files starting with an a (mget a*) or cd into folders (cd to change dirs on the FTP computer, !cd to change dirs on your local computer. It just cd to it before you start ftp).

share|improve this answer
    
That just got index.html –  Legit Apr 6 '13 at 12:58
    
it also doesnt get every file. it limits it to around 5000 again. –  Legit Apr 6 '13 at 13:06
    
maybe if there was a way to zip it across ssh? –  Legit Apr 6 '13 at 13:25
    
If there is just a signel file (index.html) than it is not FTP based but web based. What is in the index.html? Just <HTML><head></head><body>Welcome!</body></HTML>, or are there filenames in there. IN that case wget -r http://storage.provider.tld. Or use it to construct a file and use the --input-file=file from wget. –  Hennes Apr 6 '13 at 13:35
    
As to 'zip over SSH'. 1) If you can log in via ssh, you can either tar it locally and ftp or wget the single file (tar cf my_archive.tar *) or you can use netcat. –  Hennes Apr 6 '13 at 13:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.