Multiple testing for co-primary endpoints
Specification of hypotheses
The R command
?evaluate gives an overview over the function arguments of the evaluate function.
- comparator defines one of the classification rules under consideration to be the primary comparator
- benchmark is a pre-defined accuracy categorize for each subgroup
Together this implies the hypotheses system that is considered, namely
\(H_0: \forall g \forall j: \theta_j^g \leq \theta_0^g\)
In the application of primary interest, diagnostic accuracy studies, this simplifies to \(G=2\) with \(\theta_1 = Se\) and \(\theta_2 =Sp\) indicating sensitivity and specificity of a medical test or classication rule. In this case we aim to reject the global null hypothesis
\(H_0: \forall j: Se_j \leq Se_0 \wedge Sp_j \leq Sp_0\)
Comparison vs. confidence regions
In the following, we highlight the difference between the “co-primary” analysis (comparison regions) and a “full” analysis (confidence regions).
set.seed(1337)
data <- draw_data_roc(n=120, prev=c(0.25, 0.75), m=4,
delta=0.05, e=10, auc=seq(0.90, 0.95, 0.025), rho=c(0.25, 0.25))
head(data)
#> $specificity
#> model1 model2 model3 model4
#> [1,] 1 1 1 1
#> [2,] 0 0 1 1
#> [3,] 1 1 1 1
#> [4,] 1 1 1 1
#> [5,] 1 1 1 1
#> [6,] 1 1 1 1
#> [7,] 1 1 1 1
#> [8,] 1 1 1 1
#> [9,] 1 1 1 1
#> [10,] 1 1 1 1
#> [11,] 1 1 1 1
#> [12,] 1 1 1 1
#> [13,] 1 1 1 1
#> [14,] 1 1 1 1
#> [15,] 1 1 0 0
#> [16,] 0 0 0 0
#> [17,] 0 1 1 1
#> [18,] 1 1 1 1
#> [19,] 1 1 1 1
#> [20,] 1 1 0 0
#> [21,] 1 1 1 1
#> [22,] 0 0 1 1
#> [23,] 1 1 1 1
#> [24,] 1 1 1 1
#> [25,] 1 1 1 1
#> [26,] 1 1 1 1
#> [27,] 0 0 1 1
#> [28,] 1 1 1 1
#> [29,] 1 1 1 1
#> [30,] 1 1 0 0
#> [31,] 1 1 1 1
#> [32,] 1 1 1 1
#> [33,] 1 1 1 1
#> [34,] 1 1 1 1
#> [35,] 0 0 1 1
#> [36,] 1 1 1 1
#> [37,] 0 0 1 1
#> [38,] 1 1 1 1
#> [39,] 1 1 1 1
#> [40,] 1 1 1 1
#> [41,] 1 1 1 1
#> [42,] 1 1 1 1
#> [43,] 1 1 1 1
#> [44,] 0 0 1 1
#> [45,] 1 1 1 1
#> [46,] 1 1 1 1
#> [47,] 1 1 1 1
#> [48,] 1 1 1 1
#> [49,] 1 1 1 1
#> [50,] 0 0 1 1
#> [51,] 1 1 1 1
#> [52,] 1 1 1 1
#> [53,] 1 1 1 1
#> [54,] 1 1 1 1
#> [55,] 1 1 1 1
#> [56,] 1 1 1 1
#> [57,] 1 1 1 1
#> [58,] 1 1 1 1
#> [59,] 1 1 1 1
#> [60,] 1 1 1 1
#> [61,] 1 1 1 1
#> [62,] 0 1 1 1
#> [63,] 1 1 1 1
#> [64,] 1 1 1 1
#> [65,] 0 1 1 1
#> [66,] 1 1 1 1
#> [67,] 0 0 1 1
#> [68,] 1 1 1 1
#> [69,] 1 1 1 1
#> [70,] 1 1 1 1
#> [71,] 1 1 1 1
#> [72,] 1 1 1 1
#> [73,] 0 1 0 0
#> [74,] 1 1 1 1
#> [75,] 1 1 1 1
#> [76,] 0 0 1 1
#> [77,] 1 1 1 1
#> [78,] 0 1 1 1
#> [79,] 1 1 1 1
#> [80,] 1 1 1 1
#> [81,] 1 1 1 1
#> [82,] 0 0 1 1
#> [83,] 0 1 1 1
#> [84,] 1 1 1 1
#> [85,] 1 1 1 1
#> [86,] 1 1 1 1
#> [87,] 1 1 1 1
#> [88,] 1 1 1 1
#> [89,] 1 1 1 1
#> [90,] 1 1 1 1
#>
#> $sensitivity
#> model1 model2 model3 model4
#> [1,] 0 0 1 1
#> [2,] 1 1 1 1
#> [3,] 1 1 1 1
#> [4,] 1 1 0 0
#> [5,] 1 1 1 1
#> [6,] 1 1 1 1
#> [7,] 1 1 1 1
#> [8,] 1 1 1 1
#> [9,] 1 1 1 1
#> [10,] 1 0 1 1
#> [11,] 1 1 1 0
#> [12,] 1 1 1 1
#> [13,] 1 1 0 0
#> [14,] 1 1 0 0
#> [15,] 1 1 1 1
#> [16,] 1 1 1 1
#> [17,] 1 1 1 1
#> [18,] 1 1 0 0
#> [19,] 1 1 0 0
#> [20,] 1 1 1 1
#> [21,] 1 1 1 1
#> [22,] 1 1 1 1
#> [23,] 1 1 1 1
#> [24,] 1 1 1 1
#> [25,] 1 1 1 1
#> [26,] 1 1 1 1
#> [27,] 1 1 1 1
#> [28,] 1 1 1 1
#> [29,] 1 0 0 0
#> [30,] 1 1 1 1## comparison regions
results_comp <- data %>% evaluate(alternative ="greater",
alpha=0.025,
benchmark = c(0.7, 0.8),
analysis = "co-primary",
regu = TRUE,
adj = "maxt")
visualize(results_comp)
## confidence regions
results_conf <- data %>% evaluate(alternative = "greater",
alpha = 0.025,
benchmark = c(0.7, 0.8),
analysis = "full",
regu = TRUE,
adj = "maxt")
visualize(results_conf)
As we can see, the comparison regions are more liberal compared to the confidence regions.