# Lesson 9 Data Rodeo

By this stage you've read a number of different datasets into R and done a bit of data wrangling with dplyr. Now it's time for a full-fledged data rodeo! This goal of this lesson is to introduce you to more complicated data wrangling issues and more powerful tools to solve them.

## 9.1 Joins

A common task in data cleaning involves merging or joining datasets from multiple sources. Here's a very simple example. Suppose we have a student gradebook listing quiz scores, exam scores, student names, and student id numbers:

library(tidyverse)
set.seed(92815)

student_id = c(192297, 291857, 500286, 449192, 372152, 627561),
name = c('Alice', 'Bob', 'Charlotte', 'Dante', 'Ethelburga', 'Felix'),
quiz1 = round(rnorm(6, 65, 15)),
quiz2 = round(rnorm(6, 88, 5)),
quiz3 = round(rnorm(6, 75, 10)),
midterm1 = round(rnorm(6, 75, 10)),
midterm2 = round(rnorm(6, 80, 8)),
final = round(rnorm(6, 78, 11))
)
gradebook
## # A tibble: 6 × 8
##   student_id name       quiz1 quiz2 quiz3 midterm1 midterm2 final
##        <dbl> <chr>      <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl>
## 1     192297 Alice         64    96    68       81       90    99
## 2     291857 Bob           58    91    91       75       75    79
## 3     500286 Charlotte     70    94    71       81       70    74
## 4     449192 Dante         57    85    84       83       94    83
## 5     372152 Ethelburga    74    91    70       63       73    96
## 6     627561 Felix         77    86    68       78       83    75

Unfortunately, gradebook doesn't contain the students' email addresses. This information is contained in another tibble:

email_addresses <- tibble(
student_id = c(101198, 192297, 372152, 918276, 291857),
email = c('unclejoe@whitehouse.gov', 'alice.liddell@chch.ox.ac.uk',
'ethelburga@lyminge.org', 'mzuckerberg@gmail.com',
'microsoftbob@hotmail.com')
)
email_addresses
## # A tibble: 5 × 2
##   student_id email
##        <dbl> <chr>
## 1     101198 unclejoe@whitehouse.gov
## 2     192297 alice.liddell@chch.ox.ac.uk
## 3     372152 ethelburga@lyminge.org
## 4     918276 mzuckerberg@gmail.com
## 5     291857 microsoftbob@hotmail.com

Both gradebook and email_addresses contain a column of student id numbers, unique identifiers for each student in the university. We can use this column to merge the two tibbles via a dplyr mutating join. There are many different kinds of joins, and the right one to use depends on the problem you want to solve. All of the join functions have two required arguments: x and y are the tibbles we want to join. A left_join() returns a tibble with the same number of rows as x, the "left" tibble inside the statement left_joint(x, y). For example, the following command returns a tibble with one row for each student from gradebook:

left_join(gradebook, email_addresses) 
## Joining, by = "student_id"
## # A tibble: 6 × 9
##   student_id name       quiz1 quiz2 quiz3 midterm1 midterm2 final email
##        <dbl> <chr>      <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl> <chr>
## 1     192297 Alice         64    96    68       81       90    99 alice.liddell…
## 2     291857 Bob           58    91    91       75       75    79 microsoftbob@…
## 3     500286 Charlotte     70    94    71       81       70    74 <NA>
## 4     449192 Dante         57    85    84       83       94    83 <NA>
## 5     372152 Ethelburga    74    91    70       63       73    96 ethelburga@ly…
## 6     627561 Felix         77    86    68       78       83    75 <NA>

Students whose id appears in email_addresses but not in gradebook have been dropped. Students whose id appears in gradebook but not in email_addresses have been retained, but since we don't know their email address, left_join() has filled in an NA.

One of the goals of this book is to help you become comfortable learning new things about R own your own, through a combination of documentation files and internet searches. So rather than telling you how a right_join(), a full_join(), and an inner_join() work, I will ask you to discover this for yourself in the following exercise!

### 9.1.1 Exercise

1. Run the command right_join(gradebook, email_addresses). What happens? Explain how this command works, consulting the associated help file if needed.
# Result contains students whose ids are in email_addresses. Those with ids
# in gradebook who are *not* in gradebook are dropped.
right_join(gradebook, email_addresses)
## Joining, by = "student_id"
## # A tibble: 5 × 9
##   student_id name       quiz1 quiz2 quiz3 midterm1 midterm2 final email
##        <dbl> <chr>      <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl> <chr>
## 1     192297 Alice         64    96    68       81       90    99 alice.liddell…
## 2     291857 Bob           58    91    91       75       75    79 microsoftbob@…
## 3     372152 Ethelburga    74    91    70       63       73    96 ethelburga@ly…
## 4     101198 <NA>          NA    NA    NA       NA       NA    NA unclejoe@whit…
## 5     918276 <NA>          NA    NA    NA       NA       NA    NA mzuckerberg@g…
1. Run the command full_join(gradebook, email_addresses). What happens? Explain how this command works, consulting the associated help file if needed.
# Result contains everyone whose id appears in *either* dataset. This
# requires lots of padding out with missing values.
full_join(gradebook, email_addresses)
## Joining, by = "student_id"
## # A tibble: 8 × 9
##   student_id name       quiz1 quiz2 quiz3 midterm1 midterm2 final email
##        <dbl> <chr>      <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl> <chr>
## 1     192297 Alice         64    96    68       81       90    99 alice.liddell…
## 2     291857 Bob           58    91    91       75       75    79 microsoftbob@…
## 3     500286 Charlotte     70    94    71       81       70    74 <NA>
## 4     449192 Dante         57    85    84       83       94    83 <NA>
## 5     372152 Ethelburga    74    91    70       63       73    96 ethelburga@ly…
## 6     627561 Felix         77    86    68       78       83    75 <NA>
## 7     101198 <NA>          NA    NA    NA       NA       NA    NA unclejoe@whit…
## 8     918276 <NA>          NA    NA    NA       NA       NA    NA mzuckerberg@g…
1. Run the command inner_join(gradebook, email_addresses). What happens? Explain how this command works, consulting the associated help file if needed.
# Result contains only those whose id appears in *both* datasets. Everyone
# else is dropped.
inner_join(gradebook, email_addresses)
## Joining, by = "student_id"
## # A tibble: 3 × 9
##   student_id name       quiz1 quiz2 quiz3 midterm1 midterm2 final email
##        <dbl> <chr>      <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl> <chr>
## 1     192297 Alice         64    96    68       81       90    99 alice.liddell…
## 2     291857 Bob           58    91    91       75       75    79 microsoftbob@…
## 3     372152 Ethelburga    74    91    70       63       73    96 ethelburga@ly…
1. Above I used the command left_join(gradebook, email_addresses). How could I have achieved the same thing using the pipe %>%?
gradebook %>%
left_join(email_addresses)
## Joining, by = "student_id"
## # A tibble: 6 × 9
##   student_id name       quiz1 quiz2 quiz3 midterm1 midterm2 final email
##        <dbl> <chr>      <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl> <chr>
## 1     192297 Alice         64    96    68       81       90    99 alice.liddell…
## 2     291857 Bob           58    91    91       75       75    79 microsoftbob@…
## 3     500286 Charlotte     70    94    71       81       70    74 <NA>
## 4     449192 Dante         57    85    84       83       94    83 <NA>
## 5     372152 Ethelburga    74    91    70       63       73    96 ethelburga@ly…
## 6     627561 Felix         77    86    68       78       83    75 <NA>
1. Add a column called name to email_addresses that contains c('Joe', 'Alice', 'Ethelburga', 'Mark', 'Bob'). Then carry out a left join to merge gradebook with email_addresses. What happens? Does the result contain two instances of name? What if you set the argument by = 'student_id'. Explain, consulting the help file as needed.

### 9.6.1 To pipe or not to pipe?

As you may recall from an earlier lesson, %>% provides an alternative way of supplying the first argument to a function. Rather than writing f(x, y) we can instead write x %>% f(y). This is particularly helpful in data cleaning, where we want to repeatedly operate on the same tibble. Rather than writing a sequence of nested function calls f(g(h(dat, arg1), arg2), arg3) we can use a pipeline of the form dat %>% h(arg1) %>% g(arg2) %>% f(arg3). This makes it much easier to tell that h() is applied first and f() is applied last. It also makes it much easier to see at a glance that arg3 is an argument to f() while arg1 is an argument to h(). Even better it removes the need for creating a large number of temporary objects or writing out a large number of assignment statements. There's nothing wrong with this code:

dat <- h(dat, arg1)
dat <- g(dat, arg2)
dat <- f(dat, arg3)

but %>% allows us to do the same thing with a single assignment statement while typing out dat only twice rather than six times:

dat <- dat %>%
h(arg1) %>%
g(arg2) %>%
f(arg3)

So when shouldn't we use the pipe? A common rule of thumb is to limit your pipelines to a maximum of ten steps. Beyond this point things often become too difficult to read. If you find yourself tempted to write an extremely long pipeline, think about whether some of the steps naturally "go together" and consider breaking them up accordingly into two or more pipelines. There are no hard and fast rules here, but don't lose sight of the key reason for using the pipe: making code easier to understand. If the pipe actually makes things more confusing, use a different approach.

### 9.6.2 Supplying an Arbitrary Argument

By default, %>% supplies the first argument to a function: x %>% f(y) is synonymous with f(x,y). But what if we wanted to supply the second argument? We can do this by using the dot (.) as a "placeholder." For example y %>% f(x, .) is synonymous with f(x, y). For example:

x <- c(1, 2, 3, NA, 4, 5)
mean(x, na.rm = TRUE)
## [1] 3
TRUE %>% mean(x, na.rm = .)
## [1] 3

We can also use the dot (.) as a placeholder on the left hand side of a pipeline. This is basically a sneaky way of creating a function. This could come in handy if we want to carry out the same data cleaning steps on more than one dataset. Here's an example in which the sequence of steps is a series of trigonometric functions:

trigonometry_pipeline <- . %>%
sin() %>%
cos() %>%
tan()
x <- c(-pi, 0, pi)
y <- c(-0.5 * pi, 0.5 * pi)
trigonometry_pipeline(x)
## [1] 1.557408 1.557408 1.557408
trigonometry_pipeline(y)
## [1] 0.5998406 0.5998406

### 9.6.3 The many pipes of magrittr

The pipe %>% originated in a package called magrittr, but is now fully-integrated into the tidyverse. If dplyr is loaded in your R session, then you already have access to %>%. But %>% is not the only pipe operator contained in the magrittr package. There is also the assignment pipe %<>%, the exposition pipe %$%, and the tee pipe %T>%. To access these exotic pipes you will need to load magrittr explicitly, even if you've already loaded dplyr library(magrittr) The assignment pipe %<>% is shorthand for a common pattern in which we pipe an object x into a function or sequence of functions and then overwrite it with the result of the pipeline. Rather that writing this: x <- x %>% h() %>% g() %>% f() the assignment pipe %<>% allows us to instead write x %<>% h() %>% g() %>% f() So should you use %<>%? This is a surprisingly controversial question. Hadley Wickham, the driving force behind the tidyverse, says no. In his view, assignment is a sufficiently "special" operation that it's worth being explicit about, even when this requires more typing. I'm on the fence, so you'll have to make up your own mind! The exposition pipe %$% is broadly similar to the base R function with() in that it allows us to refer to elements of a list, or dataframe, without using $. For example, here are three equivalent ways of computing the average quiz score for each student in gradebook without dplyr: (gradebook$quiz1 + gradebook$quiz2 + gradebook$quiz3) / 3
## [1] 76.00000 80.00000 78.33333 75.33333 78.33333 77.00000
with(gradebook, (quiz1 + quiz2 + quiz3) / 3)
## [1] 76.00000 80.00000 78.33333 75.33333 78.33333 77.00000
gradebook %\$%
(quiz1 + quiz2 + quiz3) / 3
## [1] 76.00000 80.00000 78.33333 75.33333 78.33333 77.00000

Last but not least comes the tee pipe %T>%. Think of it like a literal tee pipe, a plumbing fitting that channels water in two directions simultaneously. Consider the following example:

set.seed(1234)
rnorm(200) %>%
matrix(ncol = 2) %T>%
plot() %>%
colSums()

## [1] -15.676174   4.124318

If we had used a %>% instead of a %T>% in the second line this code wouldn't have worked: the function plot() just prints out a plot, it doesn't return a result that could be piped into colSums(). In effect the tee pipe takes the result of rnorm(200) %>% matrix(col = 2), a matrix of normal random draws, and pipes it in two directions at the same time. One arm of the tee is plot(), which displays a scatter plot of the matrix, and the other is colSums(), which computes the column sums of the same matrix.

### 9.6.4 All About that Base: R's "Native" Pipe

As of version 4.1, base R has a "native" pipe operator |>. Like %>%, the native pipe |> supplies the first argument to a function:

x <- 1:5
x %>% mean()
## [1] 3
x |> mean()
## [1] 3

As of R version 4.2, base R also has a placeholder: _ can be used to change the behavior of |> so that it supplies an arbitrary argument to a function. For example:

x <- c(5, 2, NA, 6, 9)
TRUE %>%
mean(x, na.rm = .)
## [1] 5.5
TRUE |>
mean(x, na.rm = _)
## [1] 5.5

To learn more about R's native pipe, see the help file ?pipeOp. You may be wondering: are %>% and . completely interchangeable with |> and _? Almost, but not quite. To quote Dirk Eddelbuettel, R's native pipe |> is "a little faster ... a little simpler and more robust ... [and] a little more restrictive" than %>%. So how are we supposed to know which pipe to use? This is a good question, and it's not one for which I can provide a definitive answer. R's native pipe is so new that it's difficult to predict whether it will catch on and eventually supplant %>% in the tidyverse or if it will end up being an under-used feature of the language. For the moment, I'd suggest sticking with %>% for best results when working with tidyverse packages like dplyr.

### 9.6.5 How does %>% compare to + in ggplot2?

Have you noticed how + in ggplot2 works a bit like the pipe %>% but not quite? If you find this annoying and slightly confusing, you're not the only one! For some years now a package called ggvis has been in the works. The hope is that it will one day replace ggplot2. Its syntax will be similar but with %>% instead of +, removing a common element of confusion. As of now the ggvis project is lying fallow. For updates see https://ggvis.rstudio.com/ and https://github.com/rstudio/ggvis.

### 9.6.6 Exercise

Go back through some of the examples in which we've used %>% in previous lessons and experiment with the new pipes you've learned above. Does any of them break code you've already written? If so, can you figure out why?