bbest.Rmd

---
title: "bbest"
author: "Ben Best"
date: "January 15, 2016"
output:
  html_document:
    theme: united
    highlight: tango
---

```{r setup, echo=F}
# set working directory if has students directory and at R Console (vs knitting)
if ('students' %in% list.files() & interactive()){
    setwd('students' )
}

# ensure working directory is students
if (basename(getwd()) != 'students'){
  stop(sprintf("WHOAH! Your working directory is not in 'students'!\n   getwd(): %s", getwd()))
}
```

## Content
        
What is your burning environmental question that you'd like to address? Feel free to provide group project, dissertation, and/or personal interest. What's the study area?

![cool idea](images/bbest_cool-idea.png)
        
## Techniques
        
What techniques from the course do you think will be most applicable?
        
## Data
        
What data have you already identified? Feel free to provide a link and/or details on the variables of interest.
  
Here is some data from [Shipping in Canada (2011)](http://www.statcan.gc.ca/pub/54-205-x/2011000/part-partie1-eng.htm):
  
```{r}
ports_bc = read.csv('data/bbest_ports-bc.csv')
summary(ports_bc)
```

## Data Wrangling

```{r, eval=FALSE}
# present working directory
getwd()

# change working directory
setwd('.')

# list files
list.files()

# list files that end in '.jpg'
list.files(pattern=glob2rx('*.jpg'))

# file exists
file.exists('test.png')
```

# Install Packages

```{r, eval=FALSE}
# Run this chunk only once in your Console
# Do not evaluate when knitting Rmarkdown

# list of packages
pkgs = c(
  'readr',        # read csv
  'readxl',       # read xls
  'dplyr',        # data frame manipulation
  'tidyr',        # data tidying
  'nycflights13', # test dataset of NYC flights for 2013
  'gapminder')    # test dataset of life expectancy and popultion

# install packages if not found
for (p in pkgs){
  if (!require(p, character.only=T)){
    install.packages(p)
  }
}
```


## utils::read.csv

Traditionally, you would read a CSV like so:

```{r}
d = read.csv('../data/r-ecology/species.csv')
d
head(d)
summary(d)
```

## readr::read_csv

Better yet, try read_csv:

```{r}
library(readr)

d = read_csv('../data/r-ecology/species.csv')
d
head(d)
summary(d)
```


## dplry::tbl_df

Now convert to a dplyr table:

```{r}
library(readr)
library(dplyr)

d = read_csv('../data/r-ecology/species.csv')
d = tbl_df(d)

d = read_csv('../data/r-ecology/species.csv') %>%
  tbl_df()

d = tbl_df(read_csv('../data/r-ecology/species.csv'))
d
head(d)
summary(d)
glimpse(d)

```

## dplyr loosely

### What year does species 'NL' show up in the surveys.csv?


```{r}
library(readr)
library(dplyr)

read_csv('../data/r-ecology/surveys.csv') %>%
  select(species_id, year) %>%
  #filter(species_id == 'NL') %>%
  group_by(species_id, year) %>%
  summarize(count = n())

d = read_csv('../data/r-ecology/species.csv') %>%
  tbl_df()

d = tbl_df(read_csv('../data/r-ecology/species.csv'))
d
head(d)
summary(d)
glimpse(d)

```


## 4. Tidying up Data

### EDAWR

```{r EDAWR}
# install.packages("devtools")
# devtools::install_github("rstudio/EDAWR")
library(EDAWR)
help(package='EDAWR')
?storms    # wind speed data for 6 hurricanes
?cases     # subset of WHO tuberculosis
?pollution # pollution data from WHO Ambient Air Pollution, 2014
?tb        # tuberculosis data
View(storms)
View(cases)
View(pollution)
```

### slicing

```{r traditional R slicing}
# storms
storms$storm
storms$wind
storms$pressure
storms$date

# cases
cases$country
names(cases)[-1]
unlist(cases[1:3, 2:4])

# pollution
pollution$city[c(1,3,5)]
pollution$amount[c(1,3,5)]
pollution$amount[c(2,4,6)]

# ratio
storms$pressure / storms$wind
```

```{r dplyr on storms}
# better yet
library(dplyr)

pollution %>%
  filter(city != 'New York') %>%
  mutate(
    ratio = pressure / wind)
```


## tidyr

Two main functions: gather() and spread() 

```{r tidyr}
# install.packages("tidyr")
library(tidyr)
?gather # gather to long
?spread # spread to wide
```

### `gather`

```{r gather}
cases
gather(cases, "year", "n", 2:4)
```

### `spread`

```{r spread}
pollution
spread(pollution, size, amount)
```

Other functions to extract and combine columns...

### `separate`

```{r separate}
storms
storms2 = separate(storms, date, c("year", "month", "day"), sep = "-")
```

### `unite`

```{r unite}
storms2
unite(storms2, "date", year, month, day, sep = "-")
```

**Recap: tidyr**:

- A package that reshapes the layout of data sets.

- Make observations from variables with `gather()` Make variables from observations with `spread()`

- Split and merge columns with `unite()` and `separate()`

From the [data-wrangling-cheatsheet.pdf](./refs/cheatsheets/data-wrangling-cheatsheet.pdf):

  ![](wk04_tidyr/img/data-wrangling-cheatsheet_tidyr.png)

### tidy CO<sub>2</sub> emissions

_**Task**. Convert the following table [CO<sub>2</sub> emissions per country since 1970](http://edgar.jrc.ec.europa.eu/overview.php?v=CO2ts1990-2014&sort=des9) from wide to long format and output the first few rows into your Rmarkdown. I recommend consulting `?gather` and you should have 3 columns in your output._

```{r read co2}
library(dplyr)
library(readxl) # install.packages('readxl')

# xls downloaded from http://edgar.jrc.ec.europa.eu/news_docs/CO2_1970-2014_dataset_of_CO2_report_2015.xls
xls = '../data/co2_europa.xls'

print(getwd())
co2 = read_excel(xls, skip=12)
co2
```

_**Question**. Why use `skip=12` argument in `read_excel()`?_

## dplyr

A package that helps transform tabular data

```{r dplyr}
# install.packages("dplyr")
library(dplyr)
?select
?filter
?arrange
?mutate
?group_by
?summarise
```

See sections in the [data-wrangling-cheatsheet.pdf](./refs/cheatsheets/data-wrangling-cheatsheet.pdf):

- Subset Variables (Columns), eg `select()`
- Subset Observations (Rows), eg `filter()`
- Reshaping Data - Change the layout of a data set, eg `arrange()`
- Make New Variables, eg `mutate()`
- Group Data, eg `group_by()` and `summarise()`

### `select`

```{r select}
storms
select(storms, storm, pressure)
storms %>% select(storm, pressure)
```

### `filter`

```{r filter}
storms
filter(storms, wind >= 50)
storms %>% filter(wind >= 50)

storms %>%
  filter(wind >= 50) %>%
  select(storm, pressure)
```

### `mutate`

```{r mutate}
storms %>%
  mutate(ratio = pressure / wind) %>%
  select(storm, ratio)
```

### `group_by`

```{r group_by}
pollution
pollution %>% group_by(city)
```

### `summarise`

```{r summarise}
# by city
pollution %>% 
  group_by(city) %>%
  summarise(
    mean = mean(amount), 
    sum = sum(amount), 
    n = n())

# by size
pollution %>% 
  group_by(size) %>%
  summarise(
    mean = mean(amount), 
    sum = sum(amount), 
    n = n())
```

note that `summarize` synonymously works

### `ungroup`

```{r ungroup}
pollution %>% 
  group_by(size)

pollution %>% 
  group_by(size) %>%
  ungroup()
```

### multiple groups

```{r multiple groups}
tb %>%
  group_by(country, year) %>%
  summarise(cases = sum(cases))
  summarise(cases = sum(cases))
```

**Recap: dplyr**:

- Extract columns with `select()` and rows with `filter()`

- Sort rows by column with `arrange()`

- Make new columns with `mutate()`

- Group rows by column with `group_by()` and `summarise()`

See sections in the [data-wrangling-cheatsheet.pdf](./refs/cheatsheets/data-wrangling-cheatsheet.pdf):

- Subset Variables (Columns), eg `select()`

- Subset Observations (Rows), eg `filter()`

- Reshaping Data - Change the layout of a data set, eg `arrange()`

- Make New Variables, eg `mutate()`

- Group Data, eg `group_by()` and `summarise()`


### summarize CO<sub>2</sub> emissions

_**Task**. Report the top 5 emitting countries (not World or EU28) for 2014 using your long format table. (You may need to convert your year column from factor to numeric, eg `mutate(year = as.numeric(as.character(year)))`. As with most analyses, there are multiple ways to do this. I used the following functions: `filter`, `arrange`, `desc`, `head`)_. 

_**Task**. Summarize the total emissions by country  (not World or EU28) across years from your long format table and return the top 5 emitting countries. (As with most analyses, there are multiple ways to do this. I used the following functions: `filter`, `arrange`, `desc`, `head`)_.