1 Purpose

This vignette is used to introduce futility and it’s two versions: binding and non-binding.

2 Futility boundaries

In a sequential meta-analysis it is possible to stop early based on evidence for benefit of treatment or harm of treatment. But it might also be of interest to stop the sequential meta-analysis early for evidence of futility; when the probability of rejecting the null hypothesis is sufficiently low [1]. Futility boundaries are computed in RTSA to be able to stop a sequential meta-analysis for futility.

The futility boundaries in RTSA allows to stop for futility while not compromising the control of type-I-errors. There are two kinds of futility boundaries to choose from: binding and non-binding futility boundaries [2]. We will in this section introduce these two kinds of futility boundaries. After the introduction will we compare the two. The original TSA software only included non-binding futility boundaries [3].

2.1 Binding futility

We provide two examples of binding futility before explaining what is means to have binding futility in detail. See Figure 1 for one- and two-sided designs with no futility or with binding futility.

library(RTSA)
bound_none <- boundaries(timing = c(0.5,0.75, 1), alpha  = 0.025, beta = 0.1, 
                    side = 1, futility = "none", es_alpha = "esOF")
bound_bind <- boundaries(timing = c(0.5,0.75, 1), alpha  = 0.025, beta = 0.1, 
                    side = 1, futility = "binding", es_alpha = "esOF",
                    es_beta = "esOF")
bound_none2 <- boundaries(timing = c(0.5,0.75, 1), alpha  = 0.05, beta = 0.1, 
                    side = 2, futility = "none", es_alpha = "esOF")
bound_bind2 <- boundaries(timing = c(0.5,0.75, 1), alpha  = 0.05, beta = 0.1, 
                    side = 2, futility = "binding", es_alpha = "esOF",
                    es_beta = "esOF")

## 
## Attaching package: 'gridExtra'

## The following object is masked from 'package:dplyr':
## 
##     combine

Figure 2.1: Boundaries for benefit and harm (red lines) and boundaries for futility (blue lines) are shown on the four plots.

To control for type-I-errors, it is assumed that one will stop the sequential meta-analysis if any of the alpha boundaries are crossed. When using binding futility boundaries, it will also be assumed that the meta-analysis stops when entering the futility area. If the meta-analysis does not stop when reaching the futility area but continues will the hypothesis testing of the null hypothesis be too optimistic (more than the set level of type-I-error). In the examples above, we have a too optimistic test. We can see this based on the boundaries printed below. The first print is the one-sided test with no futility. Here the boundaries are larger than 1.96 due to the multiple testing.

bound_none

## Boundaries for a 1-sided design with a type-I-error of 0.025 and type-II-error of 0.1.
## Futility is set to: none. Alpha-spending function: esOF.
## Beta-spending function: .
## 
## Timing and boundaries:
##  SMA_Timing Upper
##       0.509 2.963
##       0.764 2.359
##       1.018 2.014

The boundaries of our design with the binding futility will however have some boundaries less than 1.96. It is not true for all designs with binding futility to have boundaries below 1.96. The boundaries depends on the choice of spending function and timing of the interim analyses. But the example here shows that under binding futility we can encounter a boundary less than 1.96 (here 1.93). The reason is that if we have binding futility, the chance of a type-I-error becomes smaller as we will stop for futility. If the null hypothesis is true, the futility boundaries will catch it with a specific probability and hence reducing the risk of a false-positive, a type-I-error. This affects the boundaries as shown below.

bound_bind

## Boundaries for a 1-sided design with a type-I-error of 0.025 and type-II-error of 0.1.
## Futility is set to: binding. Alpha-spending function: esOF.
## Beta-spending function: esOF.
## 
## Timing and boundaries:
##  SMA_Timing Upper FutLower
##       0.527 2.963    0.299
##       0.790 2.359    1.251
##       1.053 1.963    1.963

So in this specific design, if we continue the meta-analysis without respecting the futility stopping boundaries, we will on average have a type-I-error larger than 5%. This means that scenarios should as the one on the left plot in Figure 2 must not occur under binding futility if one wants to state that the type-I-error is controlled.

Figure 2.2: Boundaries for benefit and harm (red lines) and boundaries for futility (blue lines) are shown on the 2 plots.

2.2 Non-binding futility

To allow for crossing of the futility boundary without deciding to stop and also without compromising the type-I-error, one can use non-binding futility boundaries. We look into two none-binding designs:

bound_nbind <- boundaries(timing = c(0.5,0.75, 1), alpha  = 0.025, beta = 0.1, 
                    side = 1, futility = "non-binding", es_alpha = "esOF",
                    es_beta = "esOF")
bound_nbind2 <- boundaries(timing = c(0.5,0.75, 1), alpha  = 0.05, beta = 0.1, 
                    side = 2, futility = "non-binding", es_alpha = "esOF",
                    es_beta = "esOF")

The boundaries are shown on Figure 3.

Figure 2.3: Boundaries for benefit and harm (red lines) and boundaries for futility (blue lines) are shown on the four plots.

To control for type-I-errors, it is assumed that one will stop the sequential meta-analysis if any of the alpha boundaries are crossed. When using non-binding futility boundaries, futility boundaries will be calculated, but it will not be assumed that the meta-analysis stops when entering the futility area. If the meta-analysis does not stop when reaching the futility area but continues, then the result will be an over-powered meta-analysis with a controlled type-I-error. The boundaries of our design with the non-binding futility will never have boundaries less than 1.96. It is true for all designs with non-binding futility to have boundaries above 1.96 when testing on a 2.5% level for one-sided or a 5% level for two-sided. The reason is that the same boundaries are used for stopping for benefit and for harm as in the design with no futility. The print of the non-binding design below shows identical upper boundaries to the design with no futility.

bound_nbind

## Boundaries for a 1-sided design with a type-I-error of 0.025 and type-II-error of 0.1.
## Futility is set to: non-binding. Alpha-spending function: esOF.
## Beta-spending function: esOF.
## 
## Timing and boundaries:
##  SMA_Timing Upper FutLower
##       0.541 2.963    0.332
##       0.812 2.359    1.292
##       1.083 2.014    2.014

If one decides to stop the meta-analysis when reaching the non-binding futility area, we still would have controlled for type-I-error in this design, however the test will be too conservative.

2.3 Comparison

Before making recommendations about which kind of futility to use, if used at all, it is fruitful to compare the two. Several points are worth considering:

The sample size requirement is larger for non-binding futility designs compared to binding futility designs.
The control of type-I-errors in binding futility is only secured if the meta-analysis will stop when crossing a futility boundary.
Only stopping for benefit or harm in a non-binding futility meta-analysis design will cause the null hypothesis test to be too conservative.

Hence, the choice of selecting whether the futility boundaries should be binding or non-binding depends on the application. The design of the sequential meta-analysis is affected by the choice:

The stopping behavior: If one will stop the meta-analysis when reaching the futility area, binding futility should be chosen over non-binding futility.
The power of the design: More participants are required under non-binding futility to achieve a given level of power. Maybe it is too unrealistic to reach the level of participants required in a non-binding futility design.

For many designs, it will not be realistic to assume that the meta-analysis will always stop for futility. More trials might be underway and the meta-analysis will be updated. In most sequential meta-analyses will the obvious choice of futility boundaries be the non-binding futility. But this is only to be considered if one will ever use the futility boundaries for anything. The design without futility ensures a precise level of type-I-error on average at the right level of power if the meta-analysis is prospective and has the same amount of control as in a sequential trial. Further, the designs with no futility boundaries require less participants for the initial design. However, if choosing to use the futility boundaries and actually stop when the boundaries are reached, then in average the number of participants will be less due to early stopping.

3 References

Yen Chang, Tianhao Song, Jane Monaco & Anastasia Ivanova (2020) Futility stopping in clinical trials, optimality and practical considerations, Journal of Biopharmaceutical Statistics, 30:6, 1050-1059, DOI: 10.1080/10543406.2020.1818253
Power: Conditional, Unconditional, and Predictive. In: Statistical Monitoring of Clinical Trials. (2006), Statistics for Biology and Health. Springer, New York, NY. https://doi.org/10.1007/978-0-387-44970-8_3
Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C (2011). User manual for trial sequential analysis (TSA). Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen, Denmark. URL https://ctu.dk/wp-content/uploads/2021/03/2017-10-10-TSA-Manual-ENG_ER.pdf.