---
title: "Collecting Your Own Data (Web Scraping)"
output:
  ioslides_presentation: 
    smaller: true
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

## Outline

- What is web scraping?
- Why do we need it?
- Scraping the web


## Scraping the Web

In short, web scraping is a way of converting the data in HTML over to a format that can be easily accessed and used. Namely, we are *scraping* the data off a website automatically into a table.

## Why Though?

Not all websites provide .xlsx or .csv files to download tables, and not everyone has time to copy and paste data into excel. If we can automate the process, a lot of time can be saved.

## Scraping the Web Example

```{r,message = F,warning = F}
# Load relevant libraries
library(rvest)

# URL of website to gather data
url <- 'http://www.espn.com/mlb/stats/batting'
# Setting url as html
webpage <- read_html(url)

# Commands to gather data
# 'table' is very specific to our task
data_table <- html_nodes(webpage, 'table')
# print out data as table
data <- html_table(data_table)
```

##

```{r}
data
```
##

If we clean this up a bit, we can make this look really good.


```{r}
# Create data frame
data = data.frame(data)
# Remove repeated rows
df <- unique(data)
# Rename columns
colnames(df) = df[2,]
# Delete first two rows
df<- df[-c(1:2),]
# Show first 5 rows
head(df)
```

## More Than One Table
The previous website consisted of a single table. However, the following website contains many.

```{r}
# Website URL
url <- 'http://www.nfl.com/player/antoniobrown/2508061/careerstats'
webpage <- read_html(url)

# Commands to gather data
# 'table' is very specific to our task
data_table <- html_nodes(webpage, 'table')
data <- html_table(data_table)
```

##

Since there is more than one table, the `data` is actually a `list` containing each table separately.

```{r}
# Table 1
data[1]
```

##

```{r}
# Table 2
data[2]
```

##

If each individual data frame in the list had the same number of columns, we could combine them into a single data frame as follows:

```{r,eval = F}
data.combine <- do.call("rbind",data)
```
Unfortunately the previous command does not work since each data frame has different number of columns.


## Your Turn

1. Scrape any statistics from the MLS data base. (https://www.mlssoccer.com/)

## Answers

```{r}
url <- 'https://www.mlssoccer.com/stats/season?year=2017&group=g'
webpage <- read_html(url)

data_table <- html_nodes(webpage, 'table')
data <- html_table(data_table)

head(data)

```



