library(gsDesign)
library(gt)
library(dplyr)
This article explores a method of approximating a design using the exact binomial method of Chan and Bohidar (1998) by a time-to-event design using the method of Lachin and Foulkes (1986). This allows use of spending functions to derive boundaries for the exact method. The time-to-event design can not only be used to set boundaries for the Chan and Bohidar (1998) method, but to allow specification of enrollment duration and study duration to determine enrollment rates and sample size required. This vignette also illustrates the concept of super-superiority often used in prevention studies.
We begin with the assumption that we will a require a large sample size due to an endpoint with a small incidence rate. This could apply to a vaccine study or other prevention study with a relatively small number of events expected.
Paralleling the notation of Chan and Bohidar (1998), we assume \(N_C, P_C\) to be binomial sample size and probability of an event for each participant assigned to control; for the experimental treatment group, these are labelled \(N_E, P_E\). Vaccine efficacy is defined as
\[\pi = 1 - P_E/P_C.\]
The parameter \(\pi\) is often labelled as \(VE\) for vaccine efficacy. Taking into account the randomization ratio \(r\) (experimental / control) the approximate probability that any given event is in the experimental group is
\[ \begin{aligned} p &= rP_E/(rP_E+ P_C)\\ &= r/(r + P_C/P_E)\\ &= r/(r + (1-\pi)^{-1}). \end{aligned} \]
This can be inverted to obtain
\[\pi = 1 - \frac{1}{r(1/p-1)}. \]
For our example of interest, we begin with an alternate hypothesis vaccine efficacy \(\pi_1 = 0.7\) and experimental:control randomization ratio \(r=3\). This converts to an alternate hypothesis (approximate) probability that any event is in the experimental group of
pi1 <- .7
ratio <- 3
p1 <- ratio / (ratio + 1 / (1 - pi1))
p1
#> [1] 0.4736842
We use the inversion formula to revert this to \(\pi_1 = 0.7\)
1 - 1 / (ratio * (1 / p1 - 1))
#> [1] 0.7
Letting the null hypothesis vaccine efficacy be \(\pi_0 = 0.3\), our exact binomial null hypothesis probability that an event is in the experimental group is
pi0 <- .3
p0 <- ratio / (ratio + 1 / (1 - pi0))
p0
#> [1] 0.6774194
We also translate several vaccine efficacy values to proportion of events in the experimental group:
ve <- c(.5, .6, .65, .7, .75, .8)
prob_experimental <- ratio / (ratio + 1 / (1 - ve))
tibble(VE = ve, "P(Experimental)" = prob_experimental) %>%
gt() %>%
tab_options(data_row.padding = px(1)) %>%
fmt_number(columns = 2, decimals = 3)
VE | P(Experimental) |
---|---|
0.50 | 0.600 |
0.60 | 0.545 |
0.65 | 0.512 |
0.70 | 0.474 |
0.75 | 0.429 |
0.80 | 0.375 |
Chapter 12 of Jennison and Turnbull (2000) walks through how to design and analyze such a study using a fixed or group sequential design. This may have advantages and disadvantages compared to what is proposed here. However, the time-to-event approximation gives not only proposed bounds, but also sample size and study duration approximations.
For a time-to-event formulation with exponential failure rates \(\lambda_C\) for control and \(\lambda_E\) for experimental group assigned participants, we would define
\[\pi = 1 - \lambda_E / \lambda_C\]
which is 1 minus the hazard ratio often used in time-to-event studies. In the following we examine how closely the time-to-event method using asymptotic distributional assumptions can approximate an appropriate exact binomial design. We will also define a planned number of events at each of \(K\) planned analyses by \(D_k, 1\le k\le K\).
We begin by specifying parameters. The alpha
and
beta
parameters will not be met exactly due to the discrete
group sequential probability calculations performed. Thus, you may need
to adjust these parameters slightly to ensure your final design
operating characteristics are within a targeted range. The current
version includes only designs that use non-binding futility bounds or no
futility bounds. The design is generated by first using asymptotic
theory for a time-to-event design with specified spending functions.
This design is then adapted to a design using the exact binomial method
of Chan and Bohidar (1998). The
randomization ratio (experimental/control) was assumed to be 3:1 as in
the Logunov et al. (2021) trial.
alpha <- 0.023 # Type I error; this was adjusted from .025 to ensure Type I error control
beta <- 0.09 # Type II error (1 - power); this was reduced from .1 to .09 to ensure power
k <- 3 # number of analyses in group sequential design
timing <- c(.45, .7) # Relative timing of interim analyses compared to final
sfu <- sfHSD # Efficacy bound spending function (Hwang-Shih-DeCani)
sfupar <- -4 # Parameter for efficacy spending function
sfl <- sfLDOF # Futility bound spending function (O'Brien-Fleming-like here)
sflpar <- 1 # Futility bound spending function parameter; this is the standard O'Brien-Fleming
timename <- "Month" # Time unit
failRate <- .002 # Exponential failure rate
dropoutRate <- .0001 # Exponential dropout rate
enrollDuration <- 8 # Enrollment duration
trialDuration <- 24 # Planned trial duration
VE1 <- .7 # Alternate hypothesis vaccine efficacy
VE0 <- .3 # Null hypothesis vaccine efficacy
ratio <- 3 # Experimental/Control enrollment ratio
test.type <- 4 # 1 for one-sided, 4 for non-binding futility
Now we generate the design. If resulting alpha and beta do not satisfy requirements, adjust parameters above until a satisfactory result is obtained.
# Derive Group Sequential Design
# This determines final sample size
x <- gsSurv(
k = k, test.type = test.type, alpha = alpha, beta = beta, timing = timing,
sfu = sfu, sfupar = sfupar, sfl = sfl, sflpar = sflpar,
lambdaC = failRate, eta = dropoutRate,
# Translate vaccine efficacy to HR
hr = 1 - VE1, hr0 = 1 - VE0,
R = enrollDuration, T = trialDuration,
minfup = trialDuration - enrollDuration, ratio = ratio
)
Now we convert this to a design with integer event counts at
analyses. This is achieved by rounding interim analysis event counts
from the above design and rounding up the final analysis event count.
This will result in a slight change in event fractions at interim
analyses as well as a slight change from the targeted 90% power. We now
explain the rationale behind the spending function choices. Recall that
the hazard ratio (HR) is 1 minus the VE. The ~HR at bound
represents the approximate hazard ratio required to cross a bound. Thus,
small HR’s at the interim analyses along with small cumulative efficacy
spending suggest crossing an interim efficacy bound would provide a
result strong enough to potentially justify the new treatment. The
hazard ratio of ~0.7 (VE ~ 0.3) for the interim 1 futility bound mean
that the efficacy trend would be no better than the null hypothesis if
the futility bound were crossed. The second analysis futility bound with
approximate VE of 0.5 would be worth discussion with a data monitoring
committee as well as other planners for the trial; a custom spending
function could be used to set both the first and second interim bounds
to desired levels.
xx <- toInteger(x)
gsBoundSummary(xx,
tdigits = 1, logdelta = TRUE, deltaname = "HR", Nname = "Events",
exclude = c("B-value", "CP", "CP H1", "PP")
) %>%
gt() %>%
tab_header(
title = "Initial group sequential approximation",
subtitle = "Integer event counts at analyses"
) %>%
tab_options(data_row.padding = px(1))
Initial group sequential approximation | |||
Integer event counts at analyses | |||
Analysis | Value | Efficacy | Futility |
---|---|---|---|
IA 1: 44% | Z | 2.8613 | 0.0106 |
Events: 3832 | p (1-sided) | 0.0021 | 0.4958 |
Events: 32 | ~HR at bound | 0.2177 | 0.6970 |
Month: 12.8 | Spending | 0.0021 | 0.0110 |
P(Cross) if HR=0.7 | 0.0021 | 0.5042 | |
P(Cross) if HR=0.3 | 0.2878 | 0.0110 | |
IA 2: 69% | Z | 2.5509 | 1.1093 |
Events: 3832 | p (1-sided) | 0.0054 | 0.1336 |
Events: 51 | ~HR at bound | 0.3043 | 0.4873 |
Month: 17.9 | Spending | 0.0044 | 0.0309 |
P(Cross) if HR=0.7 | 0.0065 | 0.8700 | |
P(Cross) if HR=0.3 | 0.6372 | 0.0419 | |
Final | Z | 2.0393 | 2.0393 |
Events: 3832 | p (1-sided) | 0.0207 | 0.0207 |
Events: 72 | ~HR at bound | 0.4018 | 0.4018 |
Month: 24.1 | Spending | 0.0165 | 0.0481 |
P(Cross) if HR=0.7 | 0.0214 | 0.9786 | |
P(Cross) if HR=0.3 | 0.9110 | 0.0890 |
A textual summary for the design is:
cat(summary(xx, timeunit = "months"))
Asymmetric two-sided group sequential design with non-binding futility bound, 3 analyses, time-to-event outcome with sample size 3832 and 72 events required, 91 percent power, 2.3 percent (1-sided) Type I error to detect a hazard ratio of 0.3 with a null hypothesis hazard ratio of 0.7. Enrollment and total study durations are assumed to be 8 and 24.1 months, respectively. Efficacy bounds derived using a Hwang-Shih-DeCani spending function with gamma = -4. Futility bounds derived using a Lan-DeMets O’Brien-Fleming approximation spending function with none = 1.
We now convert this to an exact binomial design. The bound counts are
described in this initial table displayed. N
is the total
event count, a
the maximum number of events in the
experimental group to cross the efficacy bound. For example, if 13 or
fewer of 32 events at interim 1 are in the experimental group the
efficacy bound has been crossed. The futility bound is in
b
; at the first interim, if 22 or more of 32 total events
are in the experimental group then the futility bound would be crossed
and the alternate hypothesis could be rejected. The second and third
tables below give probabilities of crossing the upper (futility) and
lower (efficacy) bounds under the null (Theta
=
0.6774) and alternate (
Theta` = 0.4737) hypotheses,
respectively; these calculations are done under the exact binomial
distribution assumptions. Total Type I error of 0.0187 (third table,
next to last row) is below the targeted 0.025. Total power of 0.9145
(third table, last row) is above the targeted 90%.
xb <- toBinomialExact(x)
# Function to print summary table for VE design
veTable <- function(xbDesign, tteDesign, ve) {
# Analysis, N, Time, Total Cases, Success(Cases, VE, VE lower CI, Spend), Futility (Cases, VE, Spend), Type I Error, Power table
ratio <- tteDesign$ratio
prob_experimental <- ratio / (ratio + 1 / (1 - ve))
power_table <- gsBinomialExact(
k = xbDesign$k, theta = prob_experimental,
n.I = xbDesign$n.I, a = xbDesign$lower$bound,
b = xbDesign$upper$bound
)$lower$prob
# Cumulative sum within rows
power_table <- apply(power_table, 2, cumsum)
colnames(power_table) <- paste(ve * 100, "%", sep = "")
out_tab <- tibble(
Analysis = 1:tteDesign$k,
Time = tteDesign$T,
N = as.vector(round(tteDesign$eNC + tteDesign$eNE)),
Cases = xbDesign$n.I,
Success = xb$lower$bound,
Futility = xb$upper$bound,
ve_efficacy = 1 - 1 / (ratio * (xbDesign$n.I / xbDesign$lower$bound - 1)), # Efficacy bound
ve_futility = 1 - 1 / (ratio * (xbDesign$n.I / xbDesign$upper$bound - 1)), # Futility bound
alpha = as.vector(cumsum(
gsBinomialExact(
k = k, theta = xbDesign$theta[1], n.I = xbDesign$n.I,
a = xbDesign$lower$bound, b = xbDesign$n.I + 1
)$lower$prob
)),
beta = as.vector(cumsum(xbDesign$upper$prob[, 2]))
)
out_tab <- cbind(out_tab, power_table)
}
veTable(xb, x, ve) %>%
gt() %>%
fmt_number(columns = 2, decimals = 1) %>%
fmt_number(columns = c(7:8, 11:16), decimals = 2) %>%
fmt_number(columns = 9:10, decimals = 4) %>%
tab_spanner(label = "Experimental Cases at Bound", columns = 5:6, id = "cases") %>%
tab_spanner(label = "Power by Vaccine Efficacy", columns = 11:16, id = "power") %>%
tab_spanner(label = "Error Spending", columns = 9:10, id = "spend") %>%
tab_spanner(label = "Vaccine Efficacy at Bound", columns = 7:8, id = "vebound") %>%
cols_label(
ve_efficacy = "Efficacy",
ve_futility = "Futility"
) %>%
tab_footnote(
footnote = "Cumulative spending at each analysis",
locations = cells_column_spanners(spanners = "spend")
) %>%
tab_footnote(
footnote = "Experimental case counts to cross between success and futility counts do not stop trial",
locations = cells_column_spanners(spanners = "cases")
) %>%
tab_footnote(
footnote = "Exact vaccine efficacy required to cross bound",
locations = cells_column_spanners(spanners = "vebound")
) %>%
tab_footnote(
footnote = "Cumulative power at each analysis by underlying vaccine efficacy",
locations = cells_column_spanners(spanners = "power")
) %>%
tab_footnote(
footnote = "Efficacy spending ignores non-binding futility bound",
location = cells_column_labels(columns = alpha)
) %>%
tab_header("Design Bounds and Operating Characteristics")
Design Bounds and Operating Characteristics | |||||||||||||||
Analysis | Time | N | Cases | Experimental Cases at Bound1 | Vaccine Efficacy at Bound2 | Error Spending3 | Power by Vaccine Efficacy4 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Success | Futility | Efficacy | Futility | alpha5 | beta | 50% | 60% | 65% | 70% | 75% | 80% | ||||
1 | 12.9 | 3831 | 32 | 13 | 23 | 0.77 | 0.15 | 0.0014 | 0.0044 | 0.02 | 0.08 | 0.15 | 0.28 | 0.47 | 0.71 |
2 | 17.9 | 3831 | 50 | 24 | 31 | 0.69 | 0.46 | 0.0038 | 0.0278 | 0.06 | 0.23 | 0.39 | 0.60 | 0.82 | 0.95 |
3 | 24.0 | 3831 | 72 | 40 | 41 | 0.58 | 0.56 | 0.0214 | 0.0720 | 0.25 | 0.60 | 0.79 | 0.93 | 0.99 | 1.00 |
1 Experimental case counts to cross between success and futility counts do not stop trial | |||||||||||||||
2 Exact vaccine efficacy required to cross bound | |||||||||||||||
3 Cumulative spending at each analysis | |||||||||||||||
4 Cumulative power at each analysis by underlying vaccine efficacy | |||||||||||||||
5 Efficacy spending ignores non-binding futility bound |
Next, we look at \(\alpha\)- and \(\beta\)-spending for the time-to-event design to compare to the exact bound restricted by the discrete possible counts at bounds. For both \(\alpha\)- and \(\beta\)-spending, the exact binomial design has no more spending at each analysis than allowed by each spending function.
# Cumulative beta-spending at lower bounds
cumsum(xx$lower$spend) # Asymptotic design
#> [1] 0.01098749 0.04190350 0.09000000
# Cumulative alpha-spending at efficacy bounds
cumsum(gsProbability(k = xx$k, n.I = xx$n.I, a = rep(-20, xx$k), b = xx$upper$bound, theta = 0)$upper$prob)
#> [1] 0.002109848 0.006472510 0.023000001
We have provided an extended example to show that a Chan and Bohidar (1998) exact binomial using spending function bounds can be derived in a two-step process that delivers sample size and bounds by 1) deriving a related time-to-event design using asymptotic methods and then 2) converting to an exact binomial design. Adjustments were made to target Type I and Type II error probabilities in the asymptotic approximation to ensure the exact binomial Type I and Type II error rates were achieved. The method seems a reasonable and straightforward approach to develop a complete design.