forked from tidymodels/yardstick
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.Rmd
123 lines (87 loc) · 3.17 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
---
output: github_document
---
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-"
)
```
```{r load, include = FALSE, message = FALSE, warning = FALSE}
library(yardstick)
library(dplyr)
options(width = 100, digits = 3)
```
# yardstick <img src="man/figures/logo.png" align="right"/>
[](https://travis-ci.org/tidymodels/yardstick)
[](https://codecov.io/github/tidymodels/yardstick?branch=master)
[](http://cran.rstudio.com/package=yardstick)
[](http://cran.rstudio.com/package=yardstick)
[](https://www.tidyverse.org/lifecycle/#maturing)
## Overview
`yardstick` is a package to estimate how well models are working using [tidy data](https://www.jstatsoft.org/article/view/v059i10) principles. See the [package webpage](https://tidymodels.github.io/yardstick/) for more information.
## Installation
To install the package:
```{r install, eval = FALSE}
install.packages("yardstick")
# Development version:
devtools::install_github("tidymodels/yardstick")
```
## Two class metric
For example, suppose you create a classification model and predict on a new data set. You might have data that looks like this:
```{r class-data}
library(yardstick)
library(dplyr)
head(two_class_example)
```
You can use a `dplyr`-like syntax to compute common performance characteristics of the model and get them back in a data frame:
```{r class-metrics}
metrics(two_class_example, truth, predicted)
# or
two_class_example %>%
roc_auc(truth, Class1)
```
## Multiclass metrics
All classification metrics have at least one multiclass extension, with many
of them having multiple ways to calculate multiclass metrics.
```{r}
data("hpc_cv")
hpc_cv <- as_tibble(hpc_cv)
hpc_cv
```
```{r}
# Macro averaged multiclass precision
precision(hpc_cv, obs, pred)
# Micro averaged multiclass precision
precision(hpc_cv, obs, pred, estimator = "micro")
```
## Calculating metrics on resamples
If you have multiple resamples of a model, you can use a metric on a grouped
data frame to calculate the metric across all resamples at once.
This calculates multiclass ROC AUC using the method described in Hand, Till (2001),
and does it across all 10 resamples at once.
```{r}
hpc_cv %>%
group_by(Resample) %>%
roc_auc(obs, VF:L)
```
## Autoplot methods for easy visualization
Curve based methods such as `roc_curve()`, `pr_curve()` and `gain_curve()` all
have `ggplot2::autoplot()` methods that allow for powerful and easy visualization.
```{r roc-curves}
library(ggplot2)
hpc_cv %>%
group_by(Resample) %>%
roc_curve(obs, VF:L) %>%
autoplot()
```
## Quasiquotation
[Quasiquotation](http://rlang.tidyverse.org/reference/quasiquotation.html) can also be used
to supply inputs.
```{r quasi}
# probability columns:
lvl <- levels(two_class_example$truth)
two_class_example %>%
mn_log_loss(truth, !! lvl[1])
```