dplyr-notes.Rmd

---
title: 'dplyr and pipes: the basics'
author: "Sean C. Anderson, sean@seananderson.ca"
date: "September 16, 2014"
output:
  pdf_document:
    toc: yes
  html_document:
    theme: united
    toc: yes
---

# Introduction

The [dplyr](http://cran.r-project.org/package=dplyr) R package is awesome. Pipes from the [magrittr](http://cran.r-project.org/package=magrittr) R package are awesome. Put the two together and you have one of the most exciting things to happen to R in a long time.

dplyr is [Hadley Wickham's](http://had.co.nz/) re-imagined plyr package (with underlying C++ secret sauce co-written by [Romain Francois](http://blog.r-enthusiasts.com/)). plyr 2.0 if you will. It does less than dplyr, but what it does it does more elegantly and much more quickly.

dplyr is built around 5 verbs. These verbs make up the majority of the data manipulation you tend to do. You might need to:

*Select* certain columns of data.

*Filter* your data to select specific rows.

*Arrange* the rows of your data into an order.

*Mutate* your data frame to contain new columns.

*Summarise* chunks of you data in some way.

Let's look at how those work.

# The data

We're going to work with a dataset of mammal life-history, geography, and ecology traits from the PanTHERIA database:

Jones, K.E., *et al*. PanTHERIA: a species-level database of life history, ecology, and geography of extant and recently extinct mammals. Ecology 90:2648. <http://esapubs.org/archive/ecol/E090/184/>

First we'll download the data:

```{r download-data, eval=FALSE}
pantheria <-
  "http://esapubs.org/archive/ecol/E090/184/PanTHERIA_1-0_WR05_Aug2008.txt"
download.file(pantheria, destfile = "mammals.txt")
```

Next we'll read it in and simplify it. This gets a bit ugly, but you can safely just run this code chunk and ignore the details:

```{r}
mammals <- read.table("mammals.txt", sep = "\t", header = TRUE, 
  stringsAsFactors = FALSE)
names(mammals) <- sub("X[0-9._]+", "", names(mammals))
names(mammals) <- sub("MSW05_", "", names(mammals))
mammals <- dplyr::select(mammals, Order, Binomial, AdultBodyMass_g, 
  AdultHeadBodyLen_mm, HomeRange_km2, LitterSize)
names(mammals) <- gsub("([A-Z])", "_\\L\\1", names(mammals), perl = TRUE)
names(mammals) <- gsub("^_", "", names(mammals), perl = TRUE)
mammals[mammals == -999] <- NA
names(mammals)[names(mammals) == "binomial"] <- "species"
mammals <- dplyr::tbl_df(mammals) # for prettier printing
```

Next we'll load the dplyr package:

```{r, message=FALSE}
library(dplyr)
```

# Looking at the data

Data frames look a bit different in dplyr. Above, I called the `tbl_df()` function on our data. This provides more useful printing of data frames in the console. Ever accidentally printed a massive data frame in the console before? Yeah... this avoids that. You don't need to change your data to a data frame tbl first — the dplyr functions will automatically convert your data when you call them. This is what the data look like on the console:

```{r}
mammals
```

dplyr also provides a function `glimpse()` that makes it easy to look at our data in a transposed view. It's similar to the `str()` (structure) function, but has a few advantages (see `?glimpse`).

```{r}
glimpse(mammals)
```

# Selecting columns

`select()` lets you subset by columns. This is similar to `subset()` in base R, but it also allows for some fancy use of helper functions such as `contains()`, `starts_with()` and, `ends_with()`. I think these examples are self explanatory, so I'll just include them here:

```{r}
select(mammals, adult_head_body_len_mm)
select(mammals, adult_head_body_len_mm, litter_size)
select(mammals, adult_head_body_len_mm:litter_size)
select(mammals, -adult_head_body_len_mm)
select(mammals, contains("body"))
select(mammals, starts_with("adult"))
select(mammals, ends_with("g"))
select(mammals, 1:3)
```

# Filtering rows

`filter()` lets you subset by rows. You can use any valid logical statements:

```{r}
filter(mammals, adult_body_mass_g > 1e7)[ , 1:3]
filter(mammals, species == "Balaena mysticetus")
filter(mammals, order == "Carnivora" & adult_body_mass_g < 200)
```

# Arranging rows

`arrange()` lets you order the rows by one or more columns in ascending or descending order. I'm selecting the first three columns only to make the output easier to read:

```{r}
arrange(mammals, adult_body_mass_g)[ , 1:3]
arrange(mammals, desc(adult_body_mass_g))[ , 1:3]
arrange(mammals, order, adult_body_mass_g)[ , 1:3]
```

# Mutating columns

`mutate()` lets you add new columns. Notice that the new columns you create can build on each other. I will wrap these in `glimpse()` to make the new columns easy to see:

```{r}
glimpse(mutate(mammals, adult_body_mass_kg = adult_body_mass_g / 1000))
glimpse(mutate(mammals, 
    g_per_mm = adult_body_mass_g / adult_head_body_len_mm))
glimpse(mutate(mammals, 
    g_per_mm = adult_body_mass_g / adult_head_body_len_mm,
    kg_per_mm = g_per_mm / 1000))
```

# Summarising columns

Finally, `summarise()` lets you calculate summary statistics. On its own `summarise()` isn't that useful, but when combined with `group_by()` you can summarise by chunks of data. This is similar to what you might be familiar with through `ddply()` and `summarise()` from the plyr package:

```{r}
summarise(mammals, mean_mass = mean(adult_body_mass_g, na.rm = TRUE))
# summarise with group_by:
head(summarise(group_by(mammals, order),
  mean_mass = mean(adult_body_mass_g, na.rm = TRUE)))
```

# Piping data

Pipes take the output from one function and feed it to the first argument of the next function. You may have encountered the Unix pipe `|` before.

The magrittr R package contains the pipe function `%>%`. Yes it might look bizarre at first but it makes more sense when you think about it. The R language allows symbols wrapped in `%` to be defined as functions, the `>` helps imply a chain, and you can hit these 2 characters one after the other very quickly on a keyboard by holding down the Shift key. Try it!

Try pronouncing `%>%` "then" whenever you see it. If you want to see the help page, you'll need to wrap it in back ticks like so:

```{r, eval=FALSE}
?magrittr::`%>%`
```

# A trivial pipe example

Pipes can work with nearly any functions. Let's start with a non-dplyr example:

```{r}
x <- rnorm(10)
x %>% max
# is the same thing as:
max(x)
```

So, we took the value of `x` (what would have been printed on the console), captured it, and fed it to the first argument of `max()`. It's probably not clear why this is cool yet, but hang on.

# A silly dplyr example with pipes

Let's try a single-pipe dplyr example. We'll pipe the `mammals` data frame to the arrange function's first argument, and choose to arrange by the `adult_body_mass_g` column:

```{r}
mammals %>% arrange(adult_body_mass_g)
```

# An awesome example

OK, here's where it gets cool. We can chain dplyr functions in succession. This lets us write data manipulation steps in the order we think of them and avoid creating temporary variables in the middle to capture the output. This works because the output from every dplyr function is a data frame and the first argument of every dplyr function is a data frame.

Say we wanted to find the species with the highest body-mass-to-length ratio:

```{r}
mammals %>%
  mutate(mass_to_length = adult_body_mass_g / adult_head_body_len_mm) %>%
  arrange(desc(mass_to_length)) %>%
  select(species, mass_to_length)
```

So, we took `mammals`, fed it to `mutate()` to create a mass-length ratio column, arranged the resulting data frame in descending order by that ratio, and selected the columns we wanted to see. This is just the beginning. If you can imagine it, you can string it together. If you want to debug your code, just pull a pipe off the end and run the code down to that step. Or build your analysis up and add successive pipes.

The above is equivalent to:

```{r}
select(
  arrange(
    mutate(mammals,
      mass_to_length = adult_body_mass_g / adult_head_body_len_mm),
    desc(mass_to_length)),
  species, mass_to_length)
```

But the problem here is that you have to read it inside out, it's easy to miss a bracket, and the arguments get separated from the function (e.g. see `mutate()` and `desc(mass_to_length))`). Plus, this is a rather trivial example. Chain together even more steps and it quickly gets out of hand.

Here's one more example. Let's ask what taxonomic orders have a median litter size greater than 3.

```{r}
mammals %>% group_by(order) %>%
  summarise(median_litter = median(litter_size, na.rm = TRUE)) %>%
  filter(median_litter > 3) %>%
  arrange(desc(median_litter)) %>%
  select(order, median_litter)
```

These examples don't even highlight one of the best things about dplyr. It's *really* fast. The internal C++ code makes quick work of massive data frames that would make plyr slow to a crawl.

dplyr can do much more, but the above are the basics of the 5 verbs and pipes. Try them for a bit. Once they click I think they'll revolutionize your data analysis.