I'm trying to parse the nickjr.com site, specifically "http://www.nickjr.com/paw-patrol/videos/" URL for full episodes only.

nickjr.com has their site laid out by show title, then by games or videos, at least with paw patrol, so the URL will remain the same until I actually click to play a video. There are multiple video's on the site, but the full episodes have the word "full-episode" in the URL.

Example that's not a full episode:

Example that's a full episode:

My idea is to use youtube-dl, which does work on their site (tested), to download episodes, but I don't want to download all their episodes because some are not full episodes. So what I'm doing now is going to each video and having to copy the URL, then copy that into youtube-dl. It works, but nickjrs site sucks trying to navigate it.

I know web spiders exist, but I wasn't really able to find exactly what I'm looking for and it didn't really make a whole lot of sense. But my plan would be to search the site, and use regex on the URL but only to write the URL's that match the full-episode string to a text file which I could then import into youtube-dl.

So, I really need help parsing the site for full episodes only, that can be saved into a text file.

Also, I would prefer the program to be compatible with Ubuntu 16.04, but I can also use Windows 10 if need be.

| improve this question | | | | |
  • Is it a single page with all these valid URLs on? Or do you need to crawl multiple pages? – Richard Aug 14 '17 at 7:50
  • Hmm...I'm not quite sure. When I click on a video, it creates a new URL at "nickjr.com/pawpatrol/videos/newvideo." So, I suppose I need it to crawl multiple pages. – tfrue Aug 14 '17 at 23:03

If you use Chrome, you can press F12 to bring up the developer tools and then run the following small snippet in the console to write the list of links:

$("a.route").each(function (i, a) {if (a.href.match(/full-episode/)) console.log(a.href)})

It might work on other browsers also, but I've only tested it on Chrome..

| improve this answer | | | | |
  • I will try this later, I've just returned from work. I will you when I've had some time to test this. Thank you! – tfrue Aug 14 '17 at 23:04
  • Worked like a charm! I had 32 hits! The great thing about this command, is that I can run this on other series with the same success! Thank you! – tfrue Aug 14 '17 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.