Validation
The first step of the validation is to remove the real \(Q\) text. This is the actual forensic sample to analyse and it must be therefore removed when validating the analysis. The validation set is therefore made up of only the \(K\) and \(R\) datasets
This dataset can now be re-divided into ‘fake’ \(Q\) texts and ‘fake’ \(K\) texts. Each text in this corpus is
labelled as ‘unknown’ or ‘known’ so two new disjoint datasets,
validation.Q and validation.K can be created
by selecting the texts based on this label.
validation.Q <- corpus_subset(validation, grepl("^unknown", docnames(validation)))
validation.K <- corpus_subset(validation, grepl("^known", docnames(validation)))This is not the only way in which a validation analysis can be conducted. For example, one could adopt a leave-one-out approach by taking each single text and treat it as a \(Q\) and then run an authorship analysis method for each one of them. Alternatively, a completely different dataset that is similar to the case data could be used. This simpler approach is more suitable for this small example.
Authorship analysis
The analysis that is being validated is the same analysis that will be applied to the \(Q\) text. Therefore, a choice of method has to be made depending on the right choice to analyse \(Q\). In this example, the scenario simulated is a verification: was the unknown \(Q\) text written by the \(K\) author, Kw? For this reason, the method chosen is one of the most successful authorship verification methods available today, the Impostors Method (Koppel and Winter 2014), and in particular one of its latest variants called the Rank-Based Impostors Method (Potha and Stamatatos 2017, 2020).
This analysis can be run in idiolect using the function
impostors() and then selecting the default parameter for
the algorithm argument, “RBI”. The main argument of
this function are the q.data, which is the set of \(Q\) texts to test, and the k.data,
which is the set of \(K\) texts from
one or more authors that are going to be tested, and finally the set of
impostors data, cand.imps. For this example, the impostors data
is \(R\) set but generally the
recommendation is to use another dataset when possible.
The impostors() function accepts more than one author in
k.data and it also accepts the same dataset as input for both
k.data and cand.imps. When the same dataset is used,
impostors() will test each author in k.data and
use the texts written by other authors as impostors.
In contrast to other authorship analysis functions like
delta() and ngram_tracing(),
impostors() does not offer additional parameters to modify
the vectorisation process because all the Impostors Method algorithms
already have a well-specified default setting. If the user wants to
change that they should then vectorise the corpus separately using
vectorize() and then use the dfm as the input of
impostors().
The RBI variant of the method also requires setting a parameter called \(k\), which is the number of most similar impostors texts to sample from the wider set of impostors. The recommended setting is \(k=100\) or \(k=300\) but for simplicity this is set to \(k=50\) in this example.
Because an analysis using the Impostors Method can have long run times, this function can also be parallelised using more than one core.
The output of impostors() is a data frame showing the
results of comparing each \(K\) author
with each \(Q\) text. The variable
target is TRUE if the comparison is a same-author one or FALSE
if it is a different-author one. The variable score contains
the Impostors score, which is a value that ranges from 0 to 1. Other
authorship analysis functions return the same data frame type with the
same columns. The variable score therefore represents different
quantities depending on the analysis function used (e.g. for
delta(), this is the \(\Delta\) coefficient, and so on).
res[1:10,]
#> K Q target score
#> 1 Kw unknown [Kh Mail_2].txt FALSE 0.404
#> 2 Kw unknown [Lc Mail_1].txt FALSE 0.243
#> 3 Kw unknown [Ld Mail_4].txt FALSE 0.752
#> 4 Kw unknown [Lt Mail_2].txt FALSE 0.215
#> 5 Kw unknown [Lk Mail_4].txt FALSE 0.306
#> 6 Kw unknown [Lb Mail_3].txt FALSE 0.975
#> 7 Kw unknown [La Mail_3].txt FALSE 0.262
#> 8 Kw unknown [Mf Mail_1].txt FALSE 0.933
#> 9 Kw unknown [Ml Mail_3].txt FALSE 0.846
#> 10 Kh unknown [Kh Mail_2].txt TRUE 0.555In order to assess the results of this validation analysis, the
function performance() can be used to return a series of
performance metrics. This function can take one or two result data
frames as input. If two are provided, then one is used as training and
the other one as test. If only one data frame is provided, then the
performance metrics are calculated using a leave-one-out approach.
The procedure followed by this function is to held out one text (if
leave-one-out, otherwise the test dataset in its entirety) and then use
the rest of the data (or the training dataset) as a calibration dataset
to calculate a Log-Likelihood Ratio (\(LLR\)). This analysis is done using the
calibrate_LLR() function, which fits a logistic regression
model to calibrate the score into a \(LLR\) Ishihara
(2021) using the ROC library (Leeuwen 2015).
The output of the function is the following
p <- performance(res)
#> | | | 0% | |= | 1% | |== | 2% | |== | 3% | |=== | 4% | |==== | 6% | |===== | 7% | |====== | 8% | |====== | 9% | |======= | 10% | |======== | 11% | |========= | 12% | |========= | 13% | |========== | 15% | |=========== | 16% | |============ | 17% | |============= | 18% | |============= | 19% | |============== | 20% | |=============== | 21% | |================ | 22% | |================= | 24% | |================= | 25% | |================== | 26% | |=================== | 27% | |==================== | 28% | |==================== | 29% | |===================== | 30% | |====================== | 31% | |======================= | 33% | |======================== | 34% | |======================== | 35% | |========================= | 36% | |========================== | 37% | |=========================== | 38% | |============================ | 39% | |============================ | 40% | |============================= | 42% | |============================== | 43% | |=============================== | 44% | |=============================== | 45% | |================================ | 46% | |================================= | 47% | |================================== | 48% | |=================================== | 49% | |=================================== | 51% | |==================================== | 52% | |===================================== | 53% | |====================================== | 54% | |======================================= | 55% | |======================================= | 56% | |======================================== | 57% | |========================================= | 58% | |========================================== | 60% | |========================================== | 61% | |=========================================== | 62% | |============================================ | 63% | |============================================= | 64% | |============================================== | 65% | |============================================== | 66% | |=============================================== | 67% | |================================================ | 69% | |================================================= | 70% | |================================================== | 71% | |================================================== | 72% | |=================================================== | 73% | |==================================================== | 74% | |===================================================== | 75% | |===================================================== | 76% | |====================================================== | 78% | |======================================================= | 79% | |======================================================== | 80% | |========================================================= | 81% | |========================================================= | 82% | |========================================================== | 83% | |=========================================================== | 84% | |============================================================ | 85% | |============================================================= | 87% | |============================================================= | 88% | |============================================================== | 89% | |=============================================================== | 90% | |================================================================ | 91% | |================================================================ | 92% | |================================================================= | 93% | |================================================================== | 94% | |=================================================================== | 96% | |==================================================================== | 97% | |==================================================================== | 98% | |===================================================================== | 99% | |======================================================================| 100%
p$evaluation
#> Cllr Cllr_min EER Mean TRUE LLR Mean FALSE LLR TRUE trials
#> 1 0.8043863 0.6155326 19.04762 0.413398 -0.3864078 11
#> FALSE trials AUC Balanced Accuracy Precision Recall F1 TP FN
#> 1 83 0.829904 0.845679 0.3333333 0.8888889 0.4848485 8 1
#> FP TN
#> 1 16 65The \(C_{llr}\) and \(C_{llr}^{min}\) coefficients are used to evaluate the performance of the \(LLR\) (Ramos et al. 2013). These coefficients estimate the cost of the \(LLR\), where a value of 1 indicates no information in the \(LLR\) and a lower coefficient \(C_{llr}<1\) suggests that there is information in the \(LLR\), with lower values of \(C_{llr}\) suggesting better performance. The other binary classification metrics returned, such as Precision, Recall, and F1, are all calculated using \(LLR > 0\) as the threshold for a TRUE (or same-author in this case) classification.
In the present example, a \(C_{llr}=\) 0.804 suggests that there is enough information in the \(LLR\) to be able to proceed with the actual forensic analysis. The \(C_{llr}^{min}\), which is the component of \(C_{llr}\) measuring the amount of discrimination, is even lower, which means that there is a substantial difference in the two distributions. This is confirmed by the Area Under the Curve value of 0.83. Because of the large disparity between the TRUE and FALSE test cases, the values of Precision and F1 are misleading. The Balanced Accuracy value of 0.846, however, again suggests a substantial amount of discrimination at \(LLR=0\).
The results of the analysis can also be plotted using a density plot
for each of the two distributions, TRUE and FALSE. This can be done
using the density_plot() function
This plot shows the values of the score on the horizontal axis and the density for TRUE (red) vs. FALSE (blue) on the vertical axis.
These findings are evidence that the method is validated for this dataset and it is now possible to analyse the \(Q\) text and use these results to calibrate the \(LLR\) for \(Q\).
