Introduction to prcbench

Create a tool set

The create_toolset function generates a tool set with a combination of the five tools.

library(prcbench)

## A single tool
toolsetA <- create_toolset("ROCR")

## Multiple tools
toolsetB <- create_toolset(c("PerfMeas", "PRROC"))

## Tool sets can be manually combined to a single set
toolsetAB <- c(toolsetA, toolsetB)

Arguments of the create_toolset function

The create_toolset function takes two additional arguments - calc_auc and store_res.

• calc_auc decides whether tools calculate AUC score or not (Calculation of AUCs are optional for the running-time evaluation, but not necessary for the evaluation of accurate precision-recall curves)

• store_res decides whether tools store the calculated curves or not (actual curves are required for the evaluation of accurate precision-recall curves)

Use predefined tool sets

The following six tool sets are predefined with a different combination of tools along with default argument values.

## Use 'set_names'
toolsetC <- create_toolset(set_names = "auc5")

## Multiple sets are automatically combined to a single set
toolsetD <- create_toolset(set_names = c("auc5", "crv4"))

2a. Select pre-defined test data for the accuracy evaluation

The create_testset function takes predefined set names for curve evaluation. These data sets contain pre-calculated precision and recall values. The pre-calculated values must be correct so that they can be compared with the results of specified tools.

The following four test sets are currently available.

## C1 test set
testset2A <- create_testset("curve", "c1")

## C2 test set
testset2B <- create_testset("curve", "c2")

## Test data sets can be manually combined to a single set
testset2AB <- c(testset2A, testset2B)

## Multiple sets are automatically combined to a single set
testset2C <- create_testset("curve", c("c1", "c2"))

2b. Create randomly generated test data for the running-time evaluation

The create_testset function uses a naming convention for randomly generated data for benchmarking. The format is a prefix (‘b’ or ‘i’) followed by the number of dataset. The prefix ‘b’ indicates a balanced dataset, whereas ‘i’ indicates an imbalanced dataset. The number can be used with a suffix ‘k’ or ‘m’, indicating respectively 1000 or 1 million.

## A balanced data set with 50 positives and 50 negatives
testset1A <- create_testset("bench", "b100")

## An imbalanced data set with 2500 positives and 7500 negatives
testset1B <- create_testset("bench", "i10k")

## Test data sets can be manually combined to a single set
testset1AB <- c(testset1A, testset1B)

## Multiple sets are automatically combined to a single set
testset1C <- create_testset("bench", c("i10", "b10"))

3a. Accuracy evaluation of precision-recall curves

The run_evalcurve function evaluates precision-recall curves with the following five test cases. The basic idea is that the function returns the full score as long as the points generated by a library matches with the manually calculated recall and precision values.

## Evaluate precision-recall curves for ROCR and precrec with c1 test set
testset <- create_testset("curve", "c1")
toolset <- create_toolset(c("ROCR", "precrec"))
scores <- run_evalcurve(testset, toolset)
scores

##   testset toolset toolname score
## 1      c1    ROCR     ROCR   5/8
## 2      c1 precrec  precrec   8/8

## Print all results
print(scores, data_type = "all")

##    testset toolset toolname testitem testcat success total
## 1       c1    ROCR     ROCR  x_range      Rg       1     1
## 2       c1    ROCR     ROCR  y_range      Rg       1     1
## 3       c1    ROCR     ROCR   fpoint      SE       0     1
## 4       c1    ROCR     ROCR   intpts      Ip       2     4
## 5       c1    ROCR     ROCR   epoint      SE       1     1
## 6       c1 precrec  precrec  x_range      Rg       1     1
## 7       c1 precrec  precrec  y_range      Rg       1     1
## 8       c1 precrec  precrec   fpoint      SE       1     1
## 9       c1 precrec  precrec   intpts      Ip       4     4
## 10      c1 precrec  precrec   epoint      SE       1     1

## ggplot2 is necessary to use autoplot
library(ggplot2)

## Plot base points and the result of precrec on c1, c2, and c3 test sets
testset <- create_testset("curve", c("c1", "c2", "c3"))
toolset <- create_toolset("precrec")
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1)

## Plot the results of PerfMeas and PRROC on c1, c2, and c3 test sets
toolset <- create_toolset(c("PerfMeas", "PRROC"))
scores2 <- run_evalcurve(testset, toolset)
autoplot(scores2, base_plot = FALSE)

3b. Running-time evaluation of precision-recall curves

The run_benchmark function internally calls the microbenchmark function provided by the microbenchmark package. It takes a test set and a tool set and returns the result of microbenchmark.

## Run microbenchmark for aut5 on b10
testset <- create_testset("bench", "b10")
toolset <- create_toolset(set_names = "auc5")
res <- run_benchmark(testset, toolset)
res

##   testset toolset      toolname   min    lq  mean median   uq   max neval
## 1     b10    auc5 AUCCalculator 2.235 2.778 5.372  3.053 3.67 15.12     5
## 2     b10    auc5         PRROC 0.166 0.170 0.200  0.176 0.19  0.30     5
## 3     b10    auc5      PerfMeas 0.067 0.069 0.095  0.071 0.08  0.19     5
## 4     b10    auc5          ROCR 1.792 1.821 1.939  1.845 2.01  2.23     5
## 5     b10    auc5       precrec 4.423 4.476 4.590  4.513 4.55  4.99     5

The format of the function for calculating a precision-recall curve

The create_example_func function creates an example for the second argument of the create_usrtool function. The actual function should also take a testset generated by the create_testset function and returns a list with three elements - x, y, and auc.

## Show an example of the second argument
calcfunc <- create_example_func()
print(calcfunc)

## function (single_testset) 
## {
##     scores <- single_testset$get_scores()
##     list(x = seq(0, 1, 1/length(scores)), y = seq(0, 1, 1/length(scores)), 
##         auc = 0.5)
## }
## <bytecode: 0x55d06b0319d8>
## <environment: 0x55d06bc0cc00>

The create_testset function produces a testset as either TestDataB or TestDataC object. See the help files of the R6 classes - help(TestDataB) and help(TestDataC) - for the methods that can be used with the precision-recall calculation.

5a. User-defined test data for the accuracy evaluation

The first argument of the create_usrdata function should be “curve” to create a test dataset for the accuracy evaluation. Scores and labels as well as pre-calculated recall and precision values are required. These pre-calculated values are used to compare with the corresponding values predicted by the specified tools.

## Create a test dataset 'c5' for benchmarking
testsetC <- create_usrdata("curve",
  scores = c(0.1, 0.2), labels = c(1, 0),
  tsname = "c5", base_x = c(0.0, 1.0),
  base_y = c(0.0, 0.5)
)

It can be used in the same way as the predefined test datasets selected by create_testset.

## Run curve evaluation for ROCR and precrec on a predefined test dataset
toolset2 <- create_toolset(c("ROCR", "precrec"))
scores2 <- run_evalcurve(testsetC, toolset2)
autoplot(scores2, base_plot = FALSE)

## Create a test dataset 'b5' for benchmarking
testsetB <- create_usrdata("bench",
  scores = c(0.1, 0.2), labels = c(1, 0),
  tsname = "b5"
)

## Run microbenchmark for ROCR and precrec on a predefined test dataset
toolset <- create_toolset(c("ROCR", "precrec"))
res <- run_benchmark(testsetB, toolset)
res

##   testset toolset toolname min  lq mean median  uq max neval
## 1      b5    ROCR     ROCR 1.8 1.8  1.9    1.8 1.9 2.2     5
## 2      b5 precrec  precrec 4.5 4.5  4.7    4.6 4.8 5.3     5