Safe tests, as we will show below, do conserve the type I error rate
under optional continuation. Optional continuation implies gathering
more samples than was planned for because, for instance, (1) more
funding came available and the experimenter wants to learn more, (2) the
evidence looked promising, (3) a reviewer or editor urged the
experimenter to collect more data, or (4) other researchers attempt to
replicate the first finding.
A natural way to deal with the first three cases is by computing an
e-value over the combined data set. This is permitted if the data come
from the same population, and if the E-variable used is a test martingale, which is the case for
the problem at hand.
Replication attempts, however, are typically based on samples from a
different population. One way to deal with this is by multiplying the
e-value computed from the original study with the e-value computed from
the replication attempt. In this situation, the e-value formula for the
replication study could also be redesigned through the
function, for example when more information on nuisance parameters or
effect size has become available for designing a more powerful test.
We show that both procedures are safe, that is, they do not lead to
exceedance of the tolerable type I error rate, as was the case with
classical p-values.
i. Optional continuation by extending the experiment does
not result in safe tests exceeding the tolerable \(\alpha\)-level
In this subsection, we show that only continuing studies for which
the e-value \(\leq 1/ \alpha\) does
not lead to an over-rejection of the null. This is because the
sampling distribution of e-values under the null slowly drifts towards
smaller values as the number of samples increases.
Again, we consider the situation in which we only continue studies
for which the original e-values did not lead to a null rejection. For
the first batch of e-values, we use the simulation study ran in the
previous section, and we recall that under optional stopping we get
dataBatch1 <- list(dataGroup1=simResultDeltaTrueIsZero$safeSim$dataGroup1,
dataGroup2=simResultDeltaTrueIsZero$safeSim$dataGroup2)
eValuesBatch1 <- simResultDeltaTrueIsZero$safeSim$eValues
sum(eValuesBatch1 > 1/alpha)
#> [1] 13
thus, 13 false null rejections out of 500 experiments.
The follow-up batches of data will be of the same size as the
original, thus, 54, 54, and will also be generated under the null. The
slow drift to lower e-values is visualised by two histograms. The blue
histogram represents the sampling distribution of e-values of the
original simulation study that did not resulted in a null rejection. The
red histogram represents the sampling distribution of e-values computed
over the two batches of data combined. To ease visualisation, we plot
the histogram of the logarithm of e-values; a negative log e-value
implies that the e-value is smaller than one, whereas a positive log
e-values corresponds to e-values larger than one. For this we run the
following code:
selectivelyContinueDeltaTrueIsZero <-
selectivelyContinueTTestCombineData(oldValues=eValuesBatch1,
designObj=designObj,
alternative="greater",
oldData=dataBatch1,
deltaTrue=0,
seed=2, paired=TRUE,
muGlobal=120, sigmaTrue=15,
moreMainText="Batch 1-2")
Note that compared to blue histogram, the red histogram is shifted to
the left, thus, the sampling distribution of e-values computed over the
two batches combined concentrates on smaller values. In particular, most
of the mass remains under the threshold value of \(1/\alpha\), which is represented by the
vertical grey line \(\log(1/\alpha) \approx
3.00\). This shift to the left is caused by the increase in
sample sizes from n1=n2=54 to n1=n2=108. The commands
eValuesBatch1To2 <- selectivelyContinueDeltaTrueIsZero$newValues
sum(eValuesBatch1To2 > 1/alpha)
#> [1] 0
length(eValuesBatch1To2)
#> [1] 487
show that 0 out of the 487 of the selectively continued experiments
(0%) now lead to a false null rejection due to optional continuation.
Hence, after the second batch of data the total number of total number
of false null rejections is 13 out of a total of a 500 original
experiments, thus, 2.6%. Similar, a third batch won’t increase the
collective false rejection rate either.
eValuesBatch1To2 <- selectivelyContinueDeltaTrueIsZero$newValue
dataBatch1To2 <- selectivelyContinueDeltaTrueIsZero$combinedData
selectivelyContinueDeltaTrueIsZero <-
selectivelyContinueTTestCombineData(oldValues=eValuesBatch1To2,
designObj=designObj,
alternative="greater",
oldData=dataBatch1To2,
deltaTrue=0,
seed=3, paired=TRUE,
muGlobal=120, sigmaTrue=15,
moreMainText=paste("Batch: 1 to", 3))
sum(selectivelyContinueDeltaTrueIsZero$newValues > 1/alpha)
#> [1] 1
Another batch yields e-values so small that it leads to an
underflow.
When the effect is present optional continuation results in safe
tests correctly rejecting the null
The slow drift of the sampling distribution of e-values to smaller
values is replaced by a fast drift to large values whenever there is an
effect. We again consider the situation in which we continue studies for
which the first batch of e-values did not lead to a null rejection. For
this we consider the case with deltaTrue smaller than deltaMin.
simResultDeltaTrueLessThanDeltaMin <- simulate(object=designObj, nSim=1000L,
seed=1, deltaTrue=0.3,
muGlobal=120, sigmaTrue=15, pb=FALSE)
simResultDeltaTrueLessThanDeltaMin
#>
#> Simulations for Safe Paired Sample T-Test
#>
#> Based on nSim = and if the true effect size is
#> deltaTrue = 0.3
#> then the safe test optimised to detect an effect size of at least:
#> deltaMin = 0.4242641
#> with tolerable type I error rate of
#> alpha = 0.05 and power: 1-beta = 0.8
#> For experiments with planned sample sizes:
#> n1Plan = 54 and n2Plan = 54
#>
#> Is estimated to have a null rejection rate of
#> powerAtNPlan = 0.354
#> at the planned sample sizes.
#>
#> Is estimated to have a null rejection rate of
#> powerOptioStop = 0.502
#> under optional stopping, and the average stopping time is:
#> n1Mean = 43.126
The follow-up batch of data will again be of the same sizes, thus,
54, 54, and be generated with the same deltaTrue smaller than deltaMin,
as in the first batch.
As a first batch of e-values, we use the simulation study ran in the
previous section when deltaTrue equals deltaMin, and we recall that
under optional stopping we get
dataBatch1 <- list(
dataGroup1=simResultDeltaTrueLessThanDeltaMin$safeSim$dataGroup1,
dataGroup2=simResultDeltaTrueLessThanDeltaMin$safeSim$dataGroup2
)
eValuesBatch1 <- simResultDeltaTrueLessThanDeltaMin$safeSim$eValues
sum(eValuesBatch1 > 1/alpha)
#> [1] 251
251 correct null rejections, since this simulation is based on data
generated under alternative with deltaTrue > 0.
The following code selectively continues the 249 experiments which
did not lead to a null rejection:
selectivelyContinueDeltaTrueLessThanDeltaMin <-
selectivelyContinueTTestCombineData(oldValues=eValuesBatch1,
designObj=designObj,
alternative="greater",
oldData=dataBatch1,
deltaTrue=deltaMin,
seed=2, paired=TRUE, muGlobal=120,
sigmaTrue=15)
The plot shows that after the second batch of data that the sampling
distribution of e-values now concentrates on larger values, as is
apparent from the blue histogram shifting to the red histogram on the
right. Note that most of the red histogram’s mass is on the right-hand
side of the grey vertical line that represents the \(\alpha\) threshold (e.g., \(\log(1/\alpha) \approx 3\). The
continuation of the 249 experiments with \(E
< 1/\alpha=20\) led to
eValuesBatch1To2 <- selectivelyContinueDeltaTrueLessThanDeltaMin$newValues
sum(eValuesBatch1To2 > 1/alpha)
#> [1] 177
an additional 177 correct null rejections (71.0843373% of 249
experiments). This brings up the total number of null rejections to 428
out of the 500 experiments. In this case, a null rejection is correct,
since the data were generated with a true effect larger than zero,
though smaller than the minimal clinically relevant effect size.
ii. Optional continuation through replication studies
It is not always appropriate to compute one e-value over a combine
data set, in particular for replication attempts where the original
experiment is performed on a different population. In that case, one can
still easily do safe inference by multiplying the e-values computed over
each data set separately. This procedure also conserves the \(\alpha\)-level, as we show below.
In all scenarios the simulation results of the optional stopping
studies are used as original experiments. The data from these simulated
experiments were all generated with a global population mean (e.g.,
baseline blood pressure) that was set to \(\mu_{g}=120\), a population standard
deviation of \(\sigma=15\), and a
deltaTrue, which depending on the absence or presence of the effect was
zero, or equal to deltaMin, respectively.
Type I error is guaranteed when multiplying e-values
We continue the e-values from the optional stopping simulation
study
eValuesOri <- simResultDeltaTrueIsZero$safeSim$eValues
The code below multiplies these original e-values with e-values
observed in a replication attempt with a larger sample size. As in the
original study the data are generated under the null. Suppose that for
the replication attempt we now administer the same drug to a clinical
group that has a lower overall baseline blood pressure of \(\mu_{g}=90\) mmHg and standard deviation of
\(\sigma=6\).
# Needs to be larger than 1/designObj$n1Plan to have at least two samples
# in the replication attempt
someConstant <- 1.2
repData <- generateNormalData(nPlan=c(ceiling(someConstant*designObj$nPlan[1]),
ceiling(someConstant*designObj$nPlan[2])),
deltaTrue=0, nSim=nSim,
muGlobal=90, sigmaTrue=6,
paired=TRUE, seed=2)
eValuesRep <- vector("numeric", length=nSim)
for (i in seq_along(eValuesRep)) {
eValuesRep[i] <- safeTTest(x=repData$dataGroup1[i, ],
y=repData$dataGroup2[i, ],
designObj=designObj,
alternative="greater", paired=TRUE)$eValue
}
eValuesMultiply <- eValuesOri*eValuesRep
mean(eValuesMultiply > 1/alpha)
#> [1] 0.004
This shows that the type I error (0.4% < \(\alpha=5\)%) is still controlled for, even
if the replication attempt is done on a different population. In fact,
the \(\alpha\)-level is controlled for
regardless of the values of the nuisance parameters (e.g., \(\mu_{g}\) and \(\sigma\)), or the sample sizes of the
replication attempt as long as there are more than 2 data points in any
study.
Multiplying e-values under the alternative
As original experiments we now take the e-values from the optional
stopping simulation study with deltaTrue equal to deltaMin:
eValuesOri <- simResultDeltaTrueIsDeltaMin$safeSim$eValues
The code below multiplies these original e-values with e-values based
on replication data, which as in the original study are generated under
deltaTrue equal to deltaMin, but with different nuisance parameters,
e.g., \(\mu_{g}=110\) and \(\sigma=50\), thus, much more spread out
than in the original studies. Again the replication attempt is assumed
to have a larger sample size.
someConstant <- 1.2
repData <- generateNormalData(nPlan=c(ceiling(someConstant*designObj$nPlan[1]),
ceiling(someConstant*designObj$nPlan[2])),
deltaTrue=deltaMin, nSim=nSim,
muGlobal=110, sigmaTrue=50,
paired=TRUE, seed=2)
eValuesRep <- vector("numeric", length=nSim)
for (i in seq_along(eValuesRep)) {
eValuesRep[i] <- safeTTest(x=repData$dataGroup1[i, ],
y=repData$dataGroup2[i, ],
designObj=designObj,
alternative="greater", paired=TRUE)$eValue
}
eValuesMulti <- eValuesOri*eValuesRep
mean(eValuesMulti > 1/alpha)
#> [1] 0.934
This led to 467 null rejections out of the 500 experiments, which is
the correct result as the effect is present in both the original and
replication studies.
