Simulation based calibration for OncoBayes2

Tue Apr 7 15:52:05 2020

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2020-03-18 19:56:42 UTC
## git hash: 67ccabb401cc5d0313ea80fee24ebdfc8a23e5e9
## MD5:      2c9b58f279483f6a3d7f20ba9a5ceb28

The MD5 hash of the calibration data file presented here must match the above listed MD5:

##                    calibration.rds 
## "2c9b58f279483f6a3d7f20ba9a5ceb28"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Model 1: Single-agent logistic regression

Component intercept/slopes

Means

Standard deviations (tau’s)

Component intercept/slopes: group estimates

Group estimates components

Model 2: Double combination, fully exchangeable

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation

Component intercept/slopes: group estimates

Group estimates components

Group estimates interaction(s)

Model 3: Double combination, EXchangeable/NonEXchangeable model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation (tau)

Component intercept/slopes: group estimates

Group estimates components

Group estimates interaction(s)

Model 4: Triple combination, EX/NEX model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (means from exchangeable part)

Mean

Standard deviation (tau)

Component intercept/slopes: group estimates

Group estimates components

Group estimates interaction(s)

\(\chi^2\) Statistic, Model 1: Single-agent logistic regression

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 28.621 31 0.589
beta_group[A,I(log(drug_A/1)),log_slope] 39.040 31 0.152
beta_group[B,I(log(drug_A/1)),intercept] 27.840 31 0.629
beta_group[B,I(log(drug_A/1)),log_slope] 19.430 31 0.947
beta_group[C,I(log(drug_A/1)),intercept] 23.226 31 0.841
beta_group[C,I(log(drug_A/1)),log_slope] 32.672 31 0.385
mu_log_beta[I(log(drug_A/1)),intercept] 32.646 31 0.386
mu_log_beta[I(log(drug_A/1)),log_slope] 40.237 31 0.124
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 33.760 31 0.335
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 39.373 31 0.144

\(\chi^2\) Statistic, Model 2: Double combination, fully exchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 42.221 31 0.086
beta_group[A,I(log(drug_A/1)),log_slope] 45.626 31 0.044
beta_group[A,I(log(drug_B/1)),intercept] 23.405 31 0.834
beta_group[A,I(log(drug_B/1)),log_slope] 43.354 31 0.069
beta_group[B,I(log(drug_A/1)),intercept] 32.326 31 0.401
beta_group[B,I(log(drug_A/1)),log_slope] 18.733 31 0.959
beta_group[B,I(log(drug_B/1)),intercept] 23.821 31 0.818
beta_group[B,I(log(drug_B/1)),log_slope] 46.630 31 0.035
beta_group[C,I(log(drug_A/1)),intercept] 27.290 31 0.658
beta_group[C,I(log(drug_A/1)),log_slope] 48.198 31 0.025
beta_group[C,I(log(drug_B/1)),intercept] 38.157 31 0.176
beta_group[C,I(log(drug_B/1)),log_slope] 39.270 31 0.146
eta_group[A,I(drug_A/1 * drug_B/1)] 15.904 31 0.989
eta_group[B,I(drug_A/1 * drug_B/1)] 25.261 31 0.756
eta_group[C,I(drug_A/1 * drug_B/1)] 27.770 31 0.633
mu_eta[I(drug_A/1 * drug_B/1)] 29.824 31 0.526
mu_log_beta[I(log(drug_A/1)),intercept] 39.283 31 0.146
mu_log_beta[I(log(drug_A/1)),log_slope] 42.176 31 0.087
mu_log_beta[I(log(drug_B/1)),intercept] 33.613 31 0.342
mu_log_beta[I(log(drug_B/1)),log_slope] 45.466 31 0.045
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 17.728 31 0.973
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 33.709 31 0.338
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 31.354 31 0.448
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 26.374 31 0.703
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 31.264 31 0.453

\(\chi^2\) Statistic, Model 3: Double combination, EXchangeable/NonEXchangeable model

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 46.227 31 0.039
beta_group[A,I(log(drug_A/1)),log_slope] 23.840 31 0.817
beta_group[A,I(log(drug_B/1)),intercept] 25.766 31 0.732
beta_group[A,I(log(drug_B/1)),log_slope] 32.915 31 0.373
beta_group[B,I(log(drug_A/1)),intercept] 31.680 31 0.432
beta_group[B,I(log(drug_A/1)),log_slope] 40.000 31 0.129
beta_group[B,I(log(drug_B/1)),intercept] 29.261 31 0.556
beta_group[B,I(log(drug_B/1)),log_slope] 38.016 31 0.180
beta_group[C,I(log(drug_A/1)),intercept] 26.374 31 0.703
beta_group[C,I(log(drug_A/1)),log_slope] 33.990 31 0.326
beta_group[C,I(log(drug_B/1)),intercept] 33.171 31 0.362
beta_group[C,I(log(drug_B/1)),log_slope] 16.870 31 0.982
eta_group[A,I(drug_A/1 * drug_B/1)] 26.080 31 0.717
eta_group[B,I(drug_A/1 * drug_B/1)] 30.253 31 0.504
eta_group[C,I(drug_A/1 * drug_B/1)] 31.328 31 0.450
mu_eta[I(drug_A/1 * drug_B/1)] 29.747 31 0.530
mu_log_beta[I(log(drug_A/1)),intercept] 24.998 31 0.768
mu_log_beta[I(log(drug_A/1)),log_slope] 20.019 31 0.935
mu_log_beta[I(log(drug_B/1)),intercept] 30.515 31 0.491
mu_log_beta[I(log(drug_B/1)),log_slope] 48.358 31 0.024
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 25.824 31 0.730
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 37.664 31 0.191
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 33.766 31 0.335
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 21.574 31 0.896
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 32.557 31 0.390

\(\chi^2\) Statistic, Model 4: Triple combination, EX/NEX model

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 36.038 31 0.245
beta_group[A,I(log(drug_A/1)),log_slope] 26.458 31 0.699
beta_group[A,I(log(drug_B/1)),intercept] 22.490 31 0.867
beta_group[A,I(log(drug_B/1)),log_slope] 24.512 31 0.789
beta_group[A,I(log(drug_C/1)),intercept] 24.998 31 0.768
beta_group[A,I(log(drug_C/1)),log_slope] 20.653 31 0.921
beta_group[B,I(log(drug_A/1)),intercept] 39.661 31 0.137
beta_group[B,I(log(drug_A/1)),log_slope] 24.275 31 0.799
beta_group[B,I(log(drug_B/1)),intercept] 37.786 31 0.187
beta_group[B,I(log(drug_B/1)),log_slope] 35.450 31 0.266
beta_group[B,I(log(drug_C/1)),intercept] 36.256 31 0.237
beta_group[B,I(log(drug_C/1)),log_slope] 39.917 31 0.131
beta_group[C,I(log(drug_A/1)),intercept] 39.354 31 0.144
beta_group[C,I(log(drug_A/1)),log_slope] 28.954 31 0.572
beta_group[C,I(log(drug_B/1)),intercept] 37.805 31 0.186
beta_group[C,I(log(drug_B/1)),log_slope] 27.706 31 0.636
beta_group[C,I(log(drug_C/1)),intercept] 32.858 31 0.376
beta_group[C,I(log(drug_C/1)),log_slope] 37.395 31 0.199
eta_group[A,I(drug_A/1 * drug_B/1 * drug_C/1)] 28.621 31 0.589
eta_group[A,I(drug_A/1 * drug_B/1)] 19.130 31 0.953
eta_group[A,I(drug_A/1 * drug_C/1)] 35.949 31 0.248
eta_group[A,I(drug_B/1 * drug_C/1)] 26.138 31 0.715
eta_group[B,I(drug_A/1 * drug_B/1 * drug_C/1)] 32.710 31 0.383
eta_group[B,I(drug_A/1 * drug_B/1)] 26.080 31 0.717
eta_group[B,I(drug_A/1 * drug_C/1)] 43.533 31 0.067
eta_group[B,I(drug_B/1 * drug_C/1)] 30.080 31 0.513
eta_group[C,I(drug_A/1 * drug_B/1 * drug_C/1)] 33.453 31 0.349
eta_group[C,I(drug_A/1 * drug_B/1)] 36.550 31 0.227
eta_group[C,I(drug_A/1 * drug_C/1)] 18.048 31 0.969
eta_group[C,I(drug_B/1 * drug_C/1)] 27.456 31 0.649
mu_eta[I(drug_A/1 * drug_B/1 * drug_C/1)] 34.368 31 0.310
mu_eta[I(drug_A/1 * drug_B/1)] 31.974 31 0.418
mu_eta[I(drug_A/1 * drug_C/1)] 30.675 31 0.483
mu_eta[I(drug_B/1 * drug_C/1)] 33.888 31 0.330
mu_log_beta[I(log(drug_A/1)),intercept] 24.045 31 0.809
mu_log_beta[I(log(drug_A/1)),log_slope] 26.208 31 0.711
mu_log_beta[I(log(drug_B/1)),intercept] 31.629 31 0.435
mu_log_beta[I(log(drug_B/1)),log_slope] 33.786 31 0.334
mu_log_beta[I(log(drug_C/1)),intercept] 37.632 31 0.192
mu_log_beta[I(log(drug_C/1)),log_slope] 34.566 31 0.301
tau_eta[STRAT,I(drug_A/1 * drug_B/1 * drug_C/1)] 37.267 31 0.203
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] 18.490 31 0.963
tau_eta[STRAT,I(drug_A/1 * drug_C/1)] 31.078 31 0.462
tau_eta[STRAT,I(drug_B/1 * drug_C/1)] 25.696 31 0.736
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] 31.846 31 0.424
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] 23.699 31 0.823
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] 29.850 31 0.525
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] 29.267 31 0.555
tau_log_beta[STRAT,I(log(drug_C/1)),intercept] 31.053 31 0.464
tau_log_beta[STRAT,I(log(drug_C/1)),log_slope] 25.101 31 0.763

Session Info

## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
## 
## locale:
## [1] C
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] mvtnorm_1.0-11     RBesT_1.4-0        tibble_2.1.3      
##  [4] rstan_2.19.2       StanHeaders_2.19.0 abind_1.4-5       
##  [7] Formula_1.2-3      checkmate_1.9.4    OncoBayes2_0.6-4  
## [10] testthat_2.2.1     Rcpp_1.0.2         devtools_2.2.1    
## [13] usethis_1.5.1      ggplot2_3.2.1      broom_0.5.2       
## [16] tidyr_1.0.0        dplyr_0.8.3        assertthat_0.2.1  
## [19] knitr_1.25         rmarkdown_1.16    
## 
## loaded via a namespace (and not attached):
##  [1] lattice_0.20-38    prettyunits_1.0.2  ps_1.3.0          
##  [4] zeallot_0.1.0      rprojroot_1.3-2    digest_0.6.21     
##  [7] plyr_1.8.4         R6_2.4.0           ggridges_0.5.1    
## [10] backports_1.1.5    stats4_3.6.1       evaluate_0.14     
## [13] highr_0.8          pillar_1.4.2       rlang_0.4.0       
## [16] lazyeval_0.2.2     rstudioapi_0.10    callr_3.3.2       
## [19] labeling_0.3       desc_1.2.0         stringr_1.4.0     
## [22] loo_2.1.0          munsell_0.5.0      compiler_3.6.1    
## [25] xfun_0.10          pkgconfig_2.0.3    pkgbuild_1.0.6    
## [28] rstantools_2.0.0   htmltools_0.4.0    tidyselect_0.2.5  
## [31] gridExtra_2.3      codetools_0.2-16   matrixStats_0.55.0
## [34] crayon_1.3.4       withr_2.1.2        grid_3.6.1        
## [37] nlme_3.1-141       gtable_0.3.0       lifecycle_0.1.0   
## [40] magrittr_1.5       scales_1.0.0       cli_1.1.0         
## [43] stringi_1.4.3      fs_1.3.1           remotes_2.1.0     
## [46] ellipsis_0.3.0     generics_0.0.2     vctrs_0.2.0       
## [49] glue_1.3.1         purrr_0.3.3        processx_3.4.1    
## [52] pkgload_1.0.2      parallel_3.6.1     yaml_2.2.0        
## [55] inline_0.3.15      colorspace_1.4-1   sessioninfo_1.1.1 
## [58] bayesplot_1.7.0    memoise_1.1.0