1

This question already has an answer here:

What is the Powershell equivalent of this wget command?

wget -nc -i downloadList.txt

Where

-i downloadList.txt
downloads a list of urls' in the specified file.

-nc
skips the already downloaded files.

marked as duplicate by DavidPostill, BlueBerry - Vignesh4303, Moses, Steven, mdpc Oct 19 '15 at 22:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • @DavidPostill Nope. I edited the question to make it clearer. – Renuka Oct 4 '15 at 8:01
  • It's not an exact duplicate but it is a starting point. There nothing stopping you modifying it to meet your exact needs. Hint Invoke-WebRequest -InFile<String> will do the 'i`' bit. – DavidPostill Oct 4 '15 at 8:02
  • 1
    I did my Googling before posting this question. Cannot find how to get -nc and -i parameters for PS. – Renuka Oct 4 '15 at 8:05
  • Try googling harder. Invoke-WebRequest Invoke-WebRequest is the PS command for "Gets content from a web page on the Internet." ... – DavidPostill Oct 4 '15 at 8:08
  • Still couldn't get it. Also tried extracting url by looping through the text file line by line . It was slow as hell ( on a large file ). – Renuka Oct 4 '15 at 8:17
1

You can use not only PowerShell cmdlets but .Net classes too.

For -nc part, get contents of a file and select only unique strings with cat (alias for Get-Content) and sort (alias for Sort-Object). Then use wget (alias for Invoke-WebRequest) on this list of strings, extracting output file name from URLs with GetFileName

cat downloadList.txt | foreach {wget $_ -OutFile ([System.IO.Path]::GetFileName($_))}
1

There is no Powershell switch which behaves exactly like -nc

It not only prevents overwriting a file. It also checks if the target file already exists and doesn't start a second download. The whole point of -nc is to prevent the actual download.

-nc

--no-clobber

If a file is downloaded more than once in the same directory, Wget's behavior depends on a few options, including -nc. In certain cases, the local file will be clobbered, or overwritten, upon repeated download. In other cases it will be preserved.

When running Wget without -N, -nc, or -r, downloading the same file in the same directory will result in the original copy of file being preserved and the second copy being named file.1. If that file is downloaded yet again, the third copy will be named file.2, and so on. When -nc is specified, this behavior is suppressed, and Wget will refuse to download newer copies of file. Therefore, "no-clobber" is actually a misnomer in this mode. It's not clobbering that's prevented (as the numeric suffixes were already preventing clobbering), but rather the multiple version saving that's prevented.

When running Wget with -r, but without -N or -nc, re-downloading a file will result in the new copy simply overwriting the old. Adding -nc will prevent this behavior, instead causing the original version to be preserved and any newer copies on the server to be ignored.

When running Wget with -N, with or without -r, the decision as to whether or not to download a newer copy of a file depends on the local and remote timestamp and size of the file. -nc may not be specified at the same time as -N.

Note that when -nc is specified, files with the suffixes .html or (yuck) .htm will be loaded from the local disk and parsed as if they had been retrieved from the Web.

So, in Powershell V3 you have to imitate this behavior. In a nutshell:

  • Get all basenames (no extension) of files in a given folder (download destination)
  • Get all urls of a given text file (downloadList.txt)
  • Compare both lists and retrieve missing URLs
  • Send only missing URLs to Invoke-Webrequest and append html as extension
$folder = "D:\my\folder"
Compare $(Dir $folder).BaseName (gc "D:\downloadList.txt")  -PassThru | 
    where {$_.SideIndicator -eq '=>'} | 
    foreach { wget $_ -OutFile "$folder\$_.html" }

And non-golfed

$folder = "D:\my\folder"
$exists = $(Get-ChildItem $folder).BaseName
$urls = Get-Content "D:\downloadList.txt" 
$missing = Compare $exists $urls  -PassThru | where {$_.SideIndicator -eq '=>'}
$missing  | foreach { Invoke-WebRequest -Uri $_ -OutFile "$folder\$_.html" }

Not the answer you're looking for? Browse other questions tagged or ask your own question.