---
title: "Introduction to prcbench"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Introduction to prcbench}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

The `prcbench` package is a testing workbench for evaluating precision-recall curves, which requires simple three step processes to perform evaluations of libraries that create precision-recall plots.

1. Tool selection by using the [tool interface](#tool_interface)

2. Test data selection/creation by using the [test data interface](#testdata_interface)

    a. [Select pre-defined test data](#testdata_accuracy) for the accuracy evaluation

    b. [Define randomly generated test data](#testdata_runningtime) for the running-time evaluation

3. Run a evaluation function with the selected tools and test data sets

    a. [Accuracy evaluation](#accuracy_evaluation) of precision-recall curves
    
    b. [Running-time evaluation](#runningtime_evaluation) of precision-recall curves


In addition to predifined tools and test data sets, the `prcbench` package provides help functions for users to define their own tools and datasets.

4. [User-defined tool interface](#user_tools)

5. [User-defined test data interface](#user_testdata)

    a. User-defined test data for the accuracy evaluation

    b. User-defined test data for the running-time evaluation

## 1. Tool interface {#tool_interface}
The `prcbench` package provides predefined interfaces for the following five tools that calculate precision-recall curves.

Tool          Language  Link
------------- --------- --------------------------------------------------------
precrec       R         [Tool web site](https://evalclass.github.io/precrec/), [CRAN](https://cran.r-project.org/package=precrec)
ROCR          R         [Tool web site](https://ipa-tys.github.io/ROCR/), [CRAN](https://cran.r-project.org/package=ROCR) 
PRROC         R         [CRAN](https://cran.r-project.org/package=PRROC)
AUCCalculator Java      [Tool web site](http://mark.goadrich.com/programs/AUC/)
PerfMeas      R         [CRAN](https://cran.r-project.org/package=PerfMeas)


### Create a tool set
The `create_toolset` function generates a tool set with a combination of the five tools.
```{r}
library(prcbench)

## A single tool
toolsetA <- create_toolset("ROCR")

## Multiple tools
toolsetB <- create_toolset(c("PerfMeas", "PRROC"))

## Tool sets can be manually combined to a single set
toolsetAB <- c(toolsetA, toolsetB)
```

### Arguments of the `create_toolset` function
The `create_toolset` function takes two additional arguments - `calc_auc` and `store_res`. 

* `calc_auc` decides whether tools calculate AUC score or not (Calculation of AUCs are optional for the running-time evaluation, but not necessary for the evaluation of accurate precision-recall curves)

* `store_res` decides whether tools store the calculated curves or not (actual curves are required for the evaluation of accurate precision-recall curves)


### Use predefined tool sets

The following six tool sets are predefined with a different combination of tools along with default argument values.

Set name  Tools                                          calc\_auc  store\_res
--------- ---------------------------------------------- ---------- -----------
def5      ROCR, AUCCalculator, PerfMeas, PRROC, precrec  TRUE       TRUE
auc5      ROCR, xAUCCalculator, PerfMeas, PRROC, precrec  TRUE       FALSE
crv5      ROCR, AUCCalculator, PerfMeas, PRROC, precrec  FALSE      TRUE
def4      ROCR, AUCCalculator, PerfMeas, precrec         TRUE       TRUE
auc4      ROCR, AUCCalculator, PerfMeas, precrec         TRUE       FALSE
crv4      ROCR, AUCCalculator, PerfMeas, precrec         FALSE      TRUE

```{r}
## Use 'set_names'
toolsetC <- create_toolset(set_names = "auc5")

## Multiple sets are automatically combined to a single set
toolsetD <- create_toolset(set_names = c("auc5", "crv4"))
```

## 2. Test data interface {#testdata_interface}
The `prcbench` package provides two different types of test data sets.

1. `curve`: evaluates the accuracy of precision-recall curves
2. `bench`: measures running times of creating precision-recall curves

The `create_testset` function offers both types of test data by setting the first argument either as "curve" or "bench".

### 2a. Select pre-defined test data for the accuracy evaluation {#testdata_accuracy}
The `create_testset` function takes predefined set names for curve evaluation. These data sets contain pre-calculated precision and recall values. The pre-calculated values must be correct so that they can be compared with the results of specified tools.

The following four test sets are currently available.

name  #scores&labels   #pos labels  #neg labels  expected #points  expected start  expected end
----- ---------------  ------------ ------------ ----------------- --------------- -------------
c1    4                2            2             6                (0, 1)          (1, 0.5)  
c2    4                2            2             6                (0, 0.5)        (1, 0.5)        
c3    4                2            2             6                (0, 0)          (1, 0.5)
c4    8                4            4             9                (0, 1)          (1, 0.5)

```{r}
## C1 test set
testset2A <- create_testset("curve", "c1")

## C2 test set
testset2B <- create_testset("curve", "c2")

## Test data sets can be manually combined to a single set
testset2AB <- c(testset2A, testset2B)

## Multiple sets are automatically combined to a single set
testset2C <- create_testset("curve", c("c1", "c2"))
```

### 2b. Create randomly generated test data for the running-time evaluation {#testdata_runningtime}
The `create_testset` function uses a naming convention for randomly generated data for benchmarking. The format is a prefix ('b' or 'i') followed by the number of dataset. The  prefix 'b' indicates a balanced dataset, whereas 'i' indicates an imbalanced dataset. The number can be used with a suffix 'k' or 'm', indicating respectively 1000 or 1 million.
```{r}
## A balanced data set with 50 positives and 50 negatives
testset1A <- create_testset("bench", "b100")

## An imbalanced data set with 2500 positives and 7500 negatives
testset1B <- create_testset("bench", "i10k")

## Test data sets can be manually combined to a single set
testset1AB <- c(testset1A, testset1B)

## Multiple sets are automatically combined to a single set
testset1C <- create_testset("bench", c("i10", "b10"))
```

## 3. Run a evaluation function with the selected tools and test data sets
The `prcbench` package currently provides two differnt types of peformance evaluation.

1. Accuracy evaluation of precision-recall curves
    
1. Running-time evaluation of precision-recall curves
    
### 3a. Accuracy evaluation of precision-recall curves {#accuracy_evaluation}
The `run_evalcurve` function evaluates precision-recall curves with the following five test cases. The basic idea is that the function returns the full score as long as the points generated by a library matches with the manually calculated recall and precision values.

Test case  Description
---------- ---------------------------------------
fpoint     Check the first point
int_pts    Check the intermediate points
epoint     Check the end point
x_range    Evaluate a range between two recall values
y_range    Evaluate a range between two precision values

#### Evaluation scores
The `run_evalcurve` function calculates the scores of the test cases and summarizes them to a data frame.
```{r}
## Evaluate precision-recall curves for ROCR and precrec with c1 test set
testset <- create_testset("curve", "c1")
toolset <- create_toolset(c("ROCR", "precrec"))
scores <- run_evalcurve(testset, toolset)
scores
```

The result of each test case can be displayed by specifying `data_type` = `all` of the `print` function.
```{r}
## Print all results
print(scores, data_type = "all")
```

#### Visualization of the result
The `autoplot` shows a plot with the result of the `run_evalcurve` function.
```{r, fig.width=7, warning=FALSE, fig.show='hold'}
## ggplot2 is necessary to use autoplot
library(ggplot2)

## Plot base points and the result of precrec on c1, c2, and c3 test sets
testset <- create_testset("curve", c("c1", "c2", "c3"))
toolset <- create_toolset("precrec")
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1)

## Plot the results of PerfMeas and PRROC on c1, c2, and c3 test sets
toolset <- create_toolset(c("PerfMeas", "PRROC"))
scores2 <- run_evalcurve(testset, toolset)
autoplot(scores2, base_plot = FALSE)
```

### 3b. Running-time evaluation of precision-recall curves {#runningtime_evaluation}
The `run_benchmark` function internally calls the `microbenchmark` function provided by the [microbenchmark](https://cran.r-project.org/package=microbenchmark) package. It takes a test set and a tool set and returns the result of `microbenchmark`.
```{r}
## Run microbenchmark for aut5 on b10
testset <- create_testset("bench", "b10")
toolset <- create_toolset(set_names = "auc5")
res <- run_benchmark(testset, toolset)
res
```

## 4. Create a user-defined tool {#user_tools}
In addition to the predefined five tools, users can add new tool interfaces for their own tools to run benchmarking and curve evaluation. The `create_usrtool` function takes a name of the tool and a function for calculating a precision-recall curve.
```{r}
## Create a new tool set for 'xyz'
toolname <- "xyz"
calcfunc <- create_example_func()
toolsetU <- create_usrtool(toolname, calcfunc)

## User-defined tools can be combined with predefined tools
toolsetA <- create_toolset("ROCR")
toolsetU2 <- c(toolsetA, toolsetU)
```

Like the predefined tool sets, user-defined tool sets can be used for both `run_benchmark` and `run_evalcurve`.
```{r, fig.width=7, warning=FALSE, fig.show='hold'}
## Curve evaluation
testset3 <- create_testset("curve", "c2")
scores3 <- run_evalcurve(testset3, toolsetU2)
autoplot(scores3, base_plot = FALSE)
```

### The format of the function for calculating a precision-recall curve
The `create_example_func` function creates an example for the second argument of the `create_usrtool` function. The actual function should also take a `testset` generated by the `create_testset` function and returns a list with three elements - `x`, `y`, and `auc`.
```{r}
## Show an example of the second argument
calcfunc <- create_example_func()
print(calcfunc)
```

The `create_testset` function produces a `testset` as either `TestDataB` or `TestDataC` object. See the help files of the R6 classes - `help(TestDataB)` and `help(TestDataC)` - for the methods that can be used with the precision-recall calculation.

## 5. Create a user-defined test data {#user_testdata}
The `prcbench` package also supports user-defined test data interfaces. The `create_usrdata` function creates two types of test datasets.

1. User-defined test data for the accuracy evaluation

2. User-defined test data for the running-time evaluation
    
### 5a. User-defined test data for the accuracy evaluation
The first argument of the `create_usrdata` function should be "curve" to create a test dataset for the accuracy evaluation. Scores and labels as well as pre-calculated recall and precision values are required. These pre-calculated values are used to compare with the corresponding values predicted by the specified tools.
```{r}
## Create a test dataset 'c5' for benchmarking
testsetC <- create_usrdata("curve",
  scores = c(0.1, 0.2), labels = c(1, 0),
  tsname = "c5", base_x = c(0.0, 1.0),
  base_y = c(0.0, 0.5)
)
```

It can be used in the same way as the predefined test datasets selected by `create_testset`.
```{r, fig.width=7, warning=FALSE, fig.show='hold'}
## Run curve evaluation for ROCR and precrec on a predefined test dataset
toolset2 <- create_toolset(c("ROCR", "precrec"))
scores2 <- run_evalcurve(testsetC, toolset2)
autoplot(scores2, base_plot = FALSE)
```

#### 5b. User-defined test data for the running-time evaluation
The first argument of the `create_usrdata` function should be "bench" to create a test dataset for the running-time evaluation. Scores and labels are also required.
```{r}
## Create a test dataset 'b5' for benchmarking
testsetB <- create_usrdata("bench",
  scores = c(0.1, 0.2), labels = c(1, 0),
  tsname = "b5"
)
```

It can be used in the same way as the test datasets generated by `create_testset`.
```{r}
## Run microbenchmark for ROCR and precrec on a predefined test dataset
toolset <- create_toolset(c("ROCR", "precrec"))
res <- run_benchmark(testsetB, toolset)
res
```

## 6. External links
See our website - [Classifier evaluation with imbalanced datasets](https://classeval.wordpress.com/) -- for useful tips for performance evaluation on binary classifiers. In addition, we have summarized potential pitfalls of ROC plots with imbalanced datasets. See our paper -- [The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432) - for more details.
