2

As an example, here's one of the pages I'm trying to save:

https://www.oculus.com/experiences/rift/1233145293403213

When I use WGET, it downloads it as html which normally is fine. But when I open the html in a text editor, it's missing a bunch of text that's displayed on the website. Like everything in the "Additional Details" section on that page on missing from the html.

Here's the command in use in Windows:

wget --no-check-certificate -O test.html https://www.oculus.com/experiences/rift/1233145293403213/

Is my command line missing something or is WGET not the right tool for this site?

Thanks.

1
  • Tell on the question what editor you used to open up that file. Was it Notepad? – Jorge Luiz Jun 1 '20 at 11:25
0

Is my command line missing something or is WGET not the right tool for this site?

Looking at the source code for the web page, it seems likely that the "Additional Details" section is rendered with JavaScript. Unfortunately, wget does not support JavaScript.

A possible solution to this is to render the page fully with a browser and save that rendered source code. This can technically be automated with any automation tool that can interact with a browser, ranging from general purpose tools such as AutoHotkey to ones specifically designed to interact with them, such as Selenium.

Beware Headless Mode

Regarding Selenium specifically, it is possible to use it to interact with modern versions of Chrome and Firefox in "headless" mode, where a browser window isn't displayed.

However, window visibility can affect the final "rendered" HTML. And this certainly seems true for the example Oculus link in your original question. That is, the "Additional Details" section was apparently only included in the standard Selenium page_source attribute when the browser window was visible.


Python And Selenium

Selenium comes with a number of language bindings, but its Python bindings are relatively easy to use.

Below is a quick example of how you might go about retrieving the source HTML for your example page with Python, Selenium and Ungoogled Chromium:

Installation

  1. Install Python for Windows. If you run into trouble with the 3.8.x branch, the 3.7.x branch should definitely work for this.

    During installation, you will probably want select the option to install Python into your Windows path (so it is available from the command line) as well as selecting the option to install the Python py launcher.

    You will also likely want to install Python into a path that contains no spaces or special characters and that isn't a "special" folder in Windows (so avoid Program Files, Program Files (x86) and your User folder).

  2. Install a Windows build of Ungoogled Chromium. For this example, use the ones from Woolyss (either 32-bit or 64-bit). There are multiple builds of Chromium available on that page, so look for the "Marmaduke" ungoogled builds and download the appropriate 7-zip archive using the "Archive" link:

    Ungoogled Chromium - Woolyss

    Once downloaded, simply extract the archive wherever you want Ungoogled Chromium to live.

  3. Install the Python language bindings for Selenium. Assuming the copy of Python you installed from Step 1 is available from the command line (ex. you added it to your Windows path), simply run python -m pip install selenium to download and install the necessary files.

Create An Automation Script in Python

Below is a bare-bones example script that can download the HTML for the link in your original question, including the "Additional Details" section:

# --- Imports ---

# Python Standard Library
import time

# Selenium Python Bindings
from selenium import webdriver
from selenium.webdriver.chrome.options import Options


# --- Main ---

# Necessary Options for Selenium/Ungoogled Chromium.
options = Options()

options.binary_location = "C:/path/to/Ungoogled Chromium/chrome.exe"
chromedriver_path = 'C:/path/to/Ungoogled Chromium/chromedriver.exe'

# Create a Selenium webdriver object so we can issue commands to ex. Chromium.
driver = webdriver.Chrome(options=options, executable_path=chromedriver_path)

# Open Ungoogled Chromium to this web page.
driver.get('https://www.oculus.com/experiences/rift/1233145293403213/')

# Wait for 10 seconds.
time.sleep(10)

# Any page retrieved with get() has a page_source attribute. Running the code
# below with the example page above in non-headless mode should (generally)
# yield the same code as using "Save As" in the browser.
html_source = driver.page_source

# Write the returned page_source to a file. "encoding" should match the web
# page encoding of the original page to avoid write issues.
with open('page_source.html', 'w', encoding='UTF-8') as web_page_source:
    web_page_source.write(html_source)

# Wait for 10 seconds.
time.sleep(10)

# Shutdown (including closing Chrome)
driver.quit()
1
  • Thanks for the info... I did end up using Autohotkey which worked fine, and I was able to automate saving a bunch of webpages. Selenium does look interesting... I'll read your instructions and give it a try. Thanks for the help! – syrist May 28 '20 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.