Technology1 min read

How to use browser DevTools for web scraping

By · Published by Everything Blog

In short

Developers can use the Elements panel to evaluate repeating containers, inspect hidden element attributes, and test selector queries in the search bar. The browser Console supports built-in query functions to preview extracted data prior to adding the logic into code. In addition, developers can disable JavaScript or inspect page sources to determine whether traditional scraping libraries can access the target information.

Key points

  • Pre-testing selectors in browser panels ensures that extraction queries function properly…: Pre-testing selectors in browser panels ensures that extraction queries function properly before they are transferred into external code.
  • Checking raw source code helps identify whether data is rendered dynamically or provided…: Checking raw source code helps identify whether data is rendered dynamically or provided in initial server responses.
  • Hidden attributes like class ratings and title values can be located directly by reviewin…: Hidden attributes like class ratings and title values can be located directly by reviewing container nodes in DevTools.

Sources checked by our analysis pipeline: 1

The scrapers that take the least time to write are the ones where most of the work happened in the browser before any code, working out where the data lives, how the page is put together and whether JavaScript builds any of it. Every modern browser ships with developer tools that answer those questions in a few minutes, and once you know five or six of their features you stop guessing at selectors and wondering why your script returns nothing.

This guide walks through those features in Chrome, with notes for Firefox where it differs, using books.toscrape.com and quotes.toscrape.com, two practice sites Zyte runs for people learning to scrape.

Opening DevTools

Press F12, or Ctrl+Shift+I, which is Cmd+Option+I on a Mac, or right-click anything on the page and choose Inspect. The last option is the one to get used to, because it opens DevTools with the element you clicked already selected.

For scraping, you will spend nearly all your time in three panels, Elements for the page structure, Console for testing selectors and Network for seeing what the page downloads.

Technology