Week 1 Tutorial:
Web Scraping

POP77142 Quantitative Text Analysis for Social Scientists

Webpage Elements & Source Code

  • Modern browsers come with very powerful built-in tools that help in web scraping
  • Two most useful for us are:
    • Element inspection (Move cursor to some element on the page, Right click, Pick ‘Inspect’ from a menu)
    • Viewing page source (Ctrl + U/Option + CMD + U in Windows(Linux)/Mac for Chrome and Firefox)
  • Knowing the source code and element attributes helps build XPaths for selection

Element Inspection (Firefox)

Element Inspection (Chrome)

Source Code Inspection

Exercise 1: Working with HTML

  • Consider a simple HTML code below.
  • Read it into R using rvest’s read_html() function.
  • How many children does the <body> element have?
  • Build an XPath to extract the first paragraph.
  • Build an XPath to extract the second sub-heading under a green header.

library("rvest")
html_txt <- "
<!DOCTYPE html> 
<html>
    <head>
        <title>A title</title> 
    </head>
    <body>
        <h1 style='color:Red;'>
        Heading 1
        <h2>Subheading</h2>
        </h1>
        <h1 style='color:Green;'>
        Heading 2
        <h2>Subheading 1</h2>
        <h2>Subheading 2</h2>
        </h1> 
        <p>A paragraph.</p>
        <p>Another paragraph.</p>
    </body>
</html>"

Exercise 2: Working with Webpages

  • Now let’s turn to a real website.
  • Here we will extract the table of countries with their GDP from a Wikipedia article
  • Start by loading in the webpage using rvest’s read_html() function.
  • Go the webpage of the article and locate the elements that would be helpful for table extraction.
  • Extract the <table> node that corresponds to the main table.
  • Extract <tbody> element as a child of this element
  • Extract the table of with data using rvest’s html_table() function
  • Tidy up the extracted table