datawrangling_tidyverse.Rmd

---
title: "Data wrangling with `tidyverse`"
subtitle: "Maria Novosolov"
date: "`r format(Sys.Date(),'%d-%m-%Y')`"
output:
  xaringan::moon_reader:
    lib_dir: libs
    css: ["default", "custom-fonts.css"]
    chakra: "libs/remark-latest.min.js"
    nature:
      highlightStyle: github
      highlightLines: true
      ratio: '16:9'
---
```{r setup, include=FALSE}
options(htmltools.dir.version = FALSE)
library(tidyverse)
library(ggplot2)
library(icon)
```

```{r use-logo, echo=FALSE}
xaringanExtra::use_logo("img/tidyverse.png")
```

# Tidyverse is a collection of packages
.center[
<img src="img/tidyverse_core.png" width=50%>
]

---

# The advantages

.large[
- shared syntax & conventions


- tibble/data.frame in, tibble out

- neat code

]
---

# Tidy data

>If I had one thing to tell biologists learning bioinformatics, it would be “write code for humans, write data for computers”.
>
>— Vince Buffalo (@vsbuffalo) July 20, 2013

---

|Film                       |Gender |Race   | Words|
|:--------------------------|:------|:------|-----:|
|The Fellowship Of The Ring |Female |Elf    |  1229|
|The Fellowship Of The Ring |Male   |Elf    |   971|
|The Fellowship Of The Ring |Female |Hobbit |    14|
|The Fellowship Of The Ring |Male   |Hobbit |  3644|
|The Fellowship Of The Ring |Female |Man    |     0|
|The Fellowship Of The Ring |Male   |Man    |  1995|
|The Two Towers             |Female |Elf    |   331|
|The Two Towers             |Male   |Elf    |   513|
|The Two Towers             |Female |Hobbit |     0|
|The Two Towers             |Male   |Hobbit |  2463|

---

# Does your code resemble this?

```{r}
starwars_human_subset <- subset(starwars,species == "Human")
starwars_human_subset$bmi <- starwars_human_subset$mass / 
  (0.01 * starwars_human_subset$height)^2
fattest_human_from_each_planet <- aggregate(bmi ~ homeworld,data = 
      starwars_human_subset, FUN = "max")
fattest_human_from_each_planet <- merge(
  x=fattest_human_from_each_planet,
  y=starwars_human_subset,by = c("homeworld","bmi"))
fattest_human_from_each_planet <- fattest_human_from_each_planet [,1:5]
```

![](https://jamesskemp.github.io/gits-matrix/images/green6.jpg)

---
# Code should be pleasant to read

![](https://media.giphy.com/media/OWyYSmZT43pxm/giphy.gif)

---
# Tibbles

>Tibbles are data.frames that are lazy and surly: **they do less** (i.e. they don't change variable names or types, and don't do partial matching) and **complain more** (e.g. when a variable does not exist). This forces you to confront problems earlier, typically leading to cleaner, more expressive code.

.center[
<img src="img/tibble.png" width=20%>
]

https://tibble.tidyverse.org/

---
# `data.frame`

```{r}
iris
```

---
# Tibbles print nicely!

```{r warning=FALSE,message=FALSE}
library(tidyverse)
as_tibble(iris)
```
---

# Pipe ("then")

.pull-left[
![](img/pipe.png)
]

.pull-right[
Data in, data out


```r
do_another_thing(do_something(data))

# versus

data %>% 
    do_something() %>% 
    do_another_thing() 
```
]

---

class: center, middle

# `readr` package

<img src="img/readr.png" width=30%>

---

# read_xxx function

* Neater import than `read.table` and `read.csv`

* Does data check and prints a report of the data imported

* Character columns are not converted to factors

* Most useful are `read_csv`, `read_table`, and `read_delim`

* Compatible with pipe workflow

---
# Example

```{r}
mydata<- read_csv("data/sub_PanTHERIA.csv")
```

---

```{r}
mydata
```


---
class: center, middle

# `janitor` package

<img src="img/janitor.png" width=40%>
---
# `clean_names()` function

* cleans the column names to something more computer friendly

* For example, brings all the column names to lowercase and adds underscores between words
---

# Regular column names
```{r}
mydata
```

---
# Clean column names

```{r}
mydata %>% 
  janitor::clean_names()
```

---
class: center, middle
# `dplyr` function


<img src="img/dplyr.png" width=30%>

---
# Load the Star Wars data

```{r}
library(tidyverse)
data(starwars)
starwars
```
---

# `select()`

![](img/xls-select.PNG)

---

# Example: Select the name, height, mass, and species columns only


```{r}
starwars %>% 
    select(name, height, mass, species)
```

---

# Select everything but the column mass
```{r}
starwars %>% 
    select(-mass)

```
---

# `mutate()`

![](img/xls-mutate.PNG)
---

# Example: Add a bmi column (`mass/(0.01*height)^2`)

```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2)
```

---
class: exercise

# Lets start practicing 

Select the columns sex, birth year, species, and height and add a column that converts the height to meters

---

# `arrange()`

![](img/xls-arrange.PNG)
---

# Sort the data based on the bmi

```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2) %>% 
    arrange(desc(bmi))
```

---
# `filter()`

![](img/xls-filter.PNG)

---


# Filter the data to have only Droids (found in species column)

--
```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2) %>% 
    filter(species == "Droid")
```

---
# Filter the same data to have only Droids shorter than 100 cm

--
```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    mutate(bmi = mass/(0.01*height)^2) %>% 
    filter(species == "Droid", height < 100)
```

---
# `group_by(), summarize()`

![](img/xls-summary.PNG)
---

## Create a summary data in which you have the average mass, maximum height, and the number of individuals from each species, and sort it by the number of individuals
--

```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    group_by(species) %>% 
    summarize(count = n(),
              avg_mass = mean(mass, na.rm = TRUE),
              max_height = max(height, na.rm = TRUE)) %>% 
    arrange(desc(count))
```

---

class: exercise

# Your turn

What is the most common eye color?

Who is the youngest human?

Which homeworld has the most characters?

---
# Rename columns with `rename()`
```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    rename(char_name = name)
```
---
# `rename_all()`

Change all the column names to upper case
```{r}
starwars %>% 
    select(name, height, mass, species) %>% 
    rename_all(toupper)
```

---
# `left_join()`

```{r}
starwars %>% 
  mutate(height_m = height*0.01) %>% 
  select(name,height_m) %>% 
  left_join(starwars,by = "name")
```

---
class: exercise

# Practice time!

Create a new data with species mass, calculate the bmi, and join it with the starwars data

---
# What else can you do?

- conditional functions: `*_at`, `*_if`, `*_all`
- `lead` & `lag` for time series
- `inner_join`,`semi_join`
- `bind_cols`, `bind_rows`

---
class: center, middle

# `tidyr` functions

<img src="img/tidyr.png" width=30%>

---

### spread == pivot_wider

### gather == pivot_longer

![](https://raw.githubusercontent.com/gadenbuie/tidyexplain/master/images/tidyr-spread-gather.gif)

---

# Community matrix!

```{r}
sw <- starwars %>% 
  select(name, films) %>% 
  unnest(films) %>% 
  mutate(present = 1) %>% 
  pivot_wider(names_from = name,values_from = present,values_fill = list(present = 0)) %>% 
  janitor::clean_names() %>% 
  print()
```

---

# Gather back the community matrix to a long format

```{r}
sw %>% 
  pivot_longer(cols = -films,names_to = "name",values_to = "present")
```

---
class: exercise

#Lets practice!

create a long format of the subset data of species, mass, height, and birth year with the species column, one column with the categories mass,height, birth year and one column with the values

---
# `nest` datasets

Useful when you want to run analyses group based but run it only ones

```{r}
nested_sw<- starwars %>% 
  select(name,height,films,mass, species,gender) %>% 
  group_by(gender) %>% 
  nest() %>% 
  print()

```

---
# To see the first sub data

```{r}
nested_sw$data[1]
```

---
class: center, middle

# `purrr` package

<img src="img/purrr.png" width=30%>

---

# `map` function

* transform the input by applying a function to each element. (similar to `apply` function)

```{r}
starwars %>% 
  split(.$gender) %>% 
  map(~ summary(lm(height~mass,data = .)))
```

---
# Combine `nest()` with `map()`

```{r}
lm_nest<- starwars %>% 
  group_by(gender) %>% 
  nest() %>% 
  mutate(lm_results = map(data,~ summary(lm(height~mass,data =.)))) %>% 
  print()

```

---
```{r}
lm_nest$lm_results[1]
```

---
class: center, middle

# `broom` package

<img src="img/broom.png" width=30%>

---
# `tidy()` function

Allows you to print and save model results in a tabular view

```{r}
broom::tidy(lm_nest$lm_results[[1]])
```


---
class: exercise

# Your turn!

Group the starwars dataset based on sex and nest it. 

Add a column with linear model summary for each species 

Add a column with a plot for the height as a function of mass for each sex

plot all the plots side by side (use `cowplot` package)

---


class: inverse, center, middle

# Other function that are compatible with ` %>% `

---

## `ggplot2` 

* can be piped into the sequence
```{r fig.height=4, fig.width=4, message=FALSE, warning=FALSE}
starwars %>% 
  select(height,mass,species) %>% 
  ggplot(.,aes(log10(mass),log10(height),color = species))+
  geom_point()+
  theme(legend.position = "none")
```

---
### omit all rows that contain `NA` somewhere in the data

```{r}
starwars %>% 
  na.omit()
```

---
# Run models on subset of the data

```{r}
starwars %>% 
  filter(species =="Droid") %>% 
  lm(height~mass,data = .) %>% 
  summary()
```

---

class: inverse, center, middle

# That's it!