1

I'm having trouble saving web pages as I build an archive of reference material for a project.

Some web pages - older styles of forums or blog pages with simple formatting - save just fine with the usual 'Save complete webpage' option in the standard browsers.

But much of the content on this topic is user-submitted posts on social media pages like Twitter and Facebook, where something else is needed. Content like comments, videos, and images must be included, and the formatting is wrecked and the content unreadable. This is because modern pages rely heavily on javascript, which needs to be saved.

How do I save a webpage for offline viewing, precisely in the state where I see it online? Obviously, I don't expect links to work, and I can tolerate having to rip video content separately, but I need the images and formatting to be the same as when seen online.

Wayback Machine is unreliable. This backup must exist in an offline state. Sadly, I cannot print the website as a PDF since it does not capture most images.

6
  • 2
    This question is being discussed on Meta. Jan 15, 2023 at 3:03
  • I can save a twitter page just fine, with layout intact in chrome > more tools > save page as > single file. Give page you definitively have problem with. Jan 15, 2023 at 16:18
  • 1
    @JoepvanSteen As for the question - Facebook is given as an example, but Twitter has the same problem -- If you look closely, you'll see that the images aren't saved in the "single file" -- They are still referenced with the original URL. If an image is changed or removed on the web, it will be lost in the file as well. Jan 15, 2023 at 17:48
  • @NotTheDr01ds, I tried to circumvent this by watching the saved page without being connected to internet. Perhaps it was using cached images. But I am indeed seeing same thing as in your answer, parts that I didn't scroll yet aren't saved. Jan 15, 2023 at 20:30
  • @JoepvanSteen Ah, that's interesting - If you can reproduce that offline/cache-save in a reliable way, then I'd recommend adding that as an answer! Thanks! Jan 15, 2023 at 21:29

1 Answer 1

2

There's probably never going to be a solution that works for all pages. There are multiple factors when attempting to save/archive a webpage as accurately as possible (e.g. videos, as you mention), but first and foremost, "infinite scroll" is the one that is likely causing you issues.

Twitter and Facebook (as you say, "modern web apps") both use JavaScript to continuously load new data from the backend as you scroll. The data that you've already seen at the "top" of the page is usually deleted to keep things flowing without lag (until, of course, you scroll back up). The data below hasn't been loaded yet.

This means that the browser itself only "knows* about the information on, slightly above, and slightly below the current view. When you Save or Print-to-PDF, only that information will be handled.

Any system that attempted to save or print an infinitely-scrolling page would ... never complete ;-).

Another problem, which you allude to in your question, is that images themselves can be hosted on any website, and those images can be deleted or removed. To truly capture the state of the page, you'll need a solution that either downloads the images locally and updates the links to point to a local, relative path; or you'll need a format which can include the images (e.g. PDF or an image).

As far as solutions go, neither are perfect, but I would recommend:

  • Print to PDF
  • The Vivaldi web browser (also Chrome-based) has the ability to capture "the page" as an image using the screen-capture tool. I've used it often here on Stack Exchange sites (which don't, of course, have infinite-scroll).

Of course, with either of these methods, you need to compensate for infinite-scroll. You'll need to scroll to the point in the page you need to archive, capture/print it, and then perhaps scroll down to load more data and repeat.

If printing to PDF, also make sure to turn on "Background images" in the print dialog.


Example of Vivaldi capture of this page:

Example of Vivaldi capture of this page

The biggest issue is that the header bar always "floats" and covers some content. In this case, the Site title is obscured.


Example of Vivaldi capture of Twitter home page:

Twitter home page

Again, the floating toolbar tends to always obscure something. Other than that, the main problem is that only the current viewport is captured. There's a lot of blank space below that hasn't been loaded yet. However, what was captured is fairly true to how it displayed on-screen.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .