This report documents the results of a simulation based calibration (SBC) run for OncoBayes2
. TODO
The calibration data presented here has been generated at and with the OncoBayes
git version as:
## Created: 2020-03-18 19:56:42 UTC
## git hash: 67ccabb401cc5d0313ea80fee24ebdfc8a23e5e9
## MD5: 2c9b58f279483f6a3d7f20ba9a5ceb28
The MD5 hash of the calibration data file presented here must match the above listed MD5:
## calibration.rds
## "2c9b58f279483f6a3d7f20ba9a5ceb28"
Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
The fake data simulation function returns … TODO. Please refer to the sbc_tools.R
and make_reference_rankhist.R
R programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 28.621 | 31 | 0.589 |
beta_group[A,I(log(drug_A/1)),log_slope] | 39.040 | 31 | 0.152 |
beta_group[B,I(log(drug_A/1)),intercept] | 27.840 | 31 | 0.629 |
beta_group[B,I(log(drug_A/1)),log_slope] | 19.430 | 31 | 0.947 |
beta_group[C,I(log(drug_A/1)),intercept] | 23.226 | 31 | 0.841 |
beta_group[C,I(log(drug_A/1)),log_slope] | 32.672 | 31 | 0.385 |
mu_log_beta[I(log(drug_A/1)),intercept] | 32.646 | 31 | 0.386 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 40.237 | 31 | 0.124 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 33.760 | 31 | 0.335 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 39.373 | 31 | 0.144 |
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 42.221 | 31 | 0.086 |
beta_group[A,I(log(drug_A/1)),log_slope] | 45.626 | 31 | 0.044 |
beta_group[A,I(log(drug_B/1)),intercept] | 23.405 | 31 | 0.834 |
beta_group[A,I(log(drug_B/1)),log_slope] | 43.354 | 31 | 0.069 |
beta_group[B,I(log(drug_A/1)),intercept] | 32.326 | 31 | 0.401 |
beta_group[B,I(log(drug_A/1)),log_slope] | 18.733 | 31 | 0.959 |
beta_group[B,I(log(drug_B/1)),intercept] | 23.821 | 31 | 0.818 |
beta_group[B,I(log(drug_B/1)),log_slope] | 46.630 | 31 | 0.035 |
beta_group[C,I(log(drug_A/1)),intercept] | 27.290 | 31 | 0.658 |
beta_group[C,I(log(drug_A/1)),log_slope] | 48.198 | 31 | 0.025 |
beta_group[C,I(log(drug_B/1)),intercept] | 38.157 | 31 | 0.176 |
beta_group[C,I(log(drug_B/1)),log_slope] | 39.270 | 31 | 0.146 |
eta_group[A,I(drug_A/1 * drug_B/1)] | 15.904 | 31 | 0.989 |
eta_group[B,I(drug_A/1 * drug_B/1)] | 25.261 | 31 | 0.756 |
eta_group[C,I(drug_A/1 * drug_B/1)] | 27.770 | 31 | 0.633 |
mu_eta[I(drug_A/1 * drug_B/1)] | 29.824 | 31 | 0.526 |
mu_log_beta[I(log(drug_A/1)),intercept] | 39.283 | 31 | 0.146 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 42.176 | 31 | 0.087 |
mu_log_beta[I(log(drug_B/1)),intercept] | 33.613 | 31 | 0.342 |
mu_log_beta[I(log(drug_B/1)),log_slope] | 45.466 | 31 | 0.045 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] | 17.728 | 31 | 0.973 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 33.709 | 31 | 0.338 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 31.354 | 31 | 0.448 |
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] | 26.374 | 31 | 0.703 |
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] | 31.264 | 31 | 0.453 |
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 46.227 | 31 | 0.039 |
beta_group[A,I(log(drug_A/1)),log_slope] | 23.840 | 31 | 0.817 |
beta_group[A,I(log(drug_B/1)),intercept] | 25.766 | 31 | 0.732 |
beta_group[A,I(log(drug_B/1)),log_slope] | 32.915 | 31 | 0.373 |
beta_group[B,I(log(drug_A/1)),intercept] | 31.680 | 31 | 0.432 |
beta_group[B,I(log(drug_A/1)),log_slope] | 40.000 | 31 | 0.129 |
beta_group[B,I(log(drug_B/1)),intercept] | 29.261 | 31 | 0.556 |
beta_group[B,I(log(drug_B/1)),log_slope] | 38.016 | 31 | 0.180 |
beta_group[C,I(log(drug_A/1)),intercept] | 26.374 | 31 | 0.703 |
beta_group[C,I(log(drug_A/1)),log_slope] | 33.990 | 31 | 0.326 |
beta_group[C,I(log(drug_B/1)),intercept] | 33.171 | 31 | 0.362 |
beta_group[C,I(log(drug_B/1)),log_slope] | 16.870 | 31 | 0.982 |
eta_group[A,I(drug_A/1 * drug_B/1)] | 26.080 | 31 | 0.717 |
eta_group[B,I(drug_A/1 * drug_B/1)] | 30.253 | 31 | 0.504 |
eta_group[C,I(drug_A/1 * drug_B/1)] | 31.328 | 31 | 0.450 |
mu_eta[I(drug_A/1 * drug_B/1)] | 29.747 | 31 | 0.530 |
mu_log_beta[I(log(drug_A/1)),intercept] | 24.998 | 31 | 0.768 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 20.019 | 31 | 0.935 |
mu_log_beta[I(log(drug_B/1)),intercept] | 30.515 | 31 | 0.491 |
mu_log_beta[I(log(drug_B/1)),log_slope] | 48.358 | 31 | 0.024 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] | 25.824 | 31 | 0.730 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 37.664 | 31 | 0.191 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 33.766 | 31 | 0.335 |
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] | 21.574 | 31 | 0.896 |
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] | 32.557 | 31 | 0.390 |
param | statistic | df | p.value |
---|---|---|---|
beta_group[A,I(log(drug_A/1)),intercept] | 36.038 | 31 | 0.245 |
beta_group[A,I(log(drug_A/1)),log_slope] | 26.458 | 31 | 0.699 |
beta_group[A,I(log(drug_B/1)),intercept] | 22.490 | 31 | 0.867 |
beta_group[A,I(log(drug_B/1)),log_slope] | 24.512 | 31 | 0.789 |
beta_group[A,I(log(drug_C/1)),intercept] | 24.998 | 31 | 0.768 |
beta_group[A,I(log(drug_C/1)),log_slope] | 20.653 | 31 | 0.921 |
beta_group[B,I(log(drug_A/1)),intercept] | 39.661 | 31 | 0.137 |
beta_group[B,I(log(drug_A/1)),log_slope] | 24.275 | 31 | 0.799 |
beta_group[B,I(log(drug_B/1)),intercept] | 37.786 | 31 | 0.187 |
beta_group[B,I(log(drug_B/1)),log_slope] | 35.450 | 31 | 0.266 |
beta_group[B,I(log(drug_C/1)),intercept] | 36.256 | 31 | 0.237 |
beta_group[B,I(log(drug_C/1)),log_slope] | 39.917 | 31 | 0.131 |
beta_group[C,I(log(drug_A/1)),intercept] | 39.354 | 31 | 0.144 |
beta_group[C,I(log(drug_A/1)),log_slope] | 28.954 | 31 | 0.572 |
beta_group[C,I(log(drug_B/1)),intercept] | 37.805 | 31 | 0.186 |
beta_group[C,I(log(drug_B/1)),log_slope] | 27.706 | 31 | 0.636 |
beta_group[C,I(log(drug_C/1)),intercept] | 32.858 | 31 | 0.376 |
beta_group[C,I(log(drug_C/1)),log_slope] | 37.395 | 31 | 0.199 |
eta_group[A,I(drug_A/1 * drug_B/1 * drug_C/1)] | 28.621 | 31 | 0.589 |
eta_group[A,I(drug_A/1 * drug_B/1)] | 19.130 | 31 | 0.953 |
eta_group[A,I(drug_A/1 * drug_C/1)] | 35.949 | 31 | 0.248 |
eta_group[A,I(drug_B/1 * drug_C/1)] | 26.138 | 31 | 0.715 |
eta_group[B,I(drug_A/1 * drug_B/1 * drug_C/1)] | 32.710 | 31 | 0.383 |
eta_group[B,I(drug_A/1 * drug_B/1)] | 26.080 | 31 | 0.717 |
eta_group[B,I(drug_A/1 * drug_C/1)] | 43.533 | 31 | 0.067 |
eta_group[B,I(drug_B/1 * drug_C/1)] | 30.080 | 31 | 0.513 |
eta_group[C,I(drug_A/1 * drug_B/1 * drug_C/1)] | 33.453 | 31 | 0.349 |
eta_group[C,I(drug_A/1 * drug_B/1)] | 36.550 | 31 | 0.227 |
eta_group[C,I(drug_A/1 * drug_C/1)] | 18.048 | 31 | 0.969 |
eta_group[C,I(drug_B/1 * drug_C/1)] | 27.456 | 31 | 0.649 |
mu_eta[I(drug_A/1 * drug_B/1 * drug_C/1)] | 34.368 | 31 | 0.310 |
mu_eta[I(drug_A/1 * drug_B/1)] | 31.974 | 31 | 0.418 |
mu_eta[I(drug_A/1 * drug_C/1)] | 30.675 | 31 | 0.483 |
mu_eta[I(drug_B/1 * drug_C/1)] | 33.888 | 31 | 0.330 |
mu_log_beta[I(log(drug_A/1)),intercept] | 24.045 | 31 | 0.809 |
mu_log_beta[I(log(drug_A/1)),log_slope] | 26.208 | 31 | 0.711 |
mu_log_beta[I(log(drug_B/1)),intercept] | 31.629 | 31 | 0.435 |
mu_log_beta[I(log(drug_B/1)),log_slope] | 33.786 | 31 | 0.334 |
mu_log_beta[I(log(drug_C/1)),intercept] | 37.632 | 31 | 0.192 |
mu_log_beta[I(log(drug_C/1)),log_slope] | 34.566 | 31 | 0.301 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1 * drug_C/1)] | 37.267 | 31 | 0.203 |
tau_eta[STRAT,I(drug_A/1 * drug_B/1)] | 18.490 | 31 | 0.963 |
tau_eta[STRAT,I(drug_A/1 * drug_C/1)] | 31.078 | 31 | 0.462 |
tau_eta[STRAT,I(drug_B/1 * drug_C/1)] | 25.696 | 31 | 0.736 |
tau_log_beta[STRAT,I(log(drug_A/1)),intercept] | 31.846 | 31 | 0.424 |
tau_log_beta[STRAT,I(log(drug_A/1)),log_slope] | 23.699 | 31 | 0.823 |
tau_log_beta[STRAT,I(log(drug_B/1)),intercept] | 29.850 | 31 | 0.525 |
tau_log_beta[STRAT,I(log(drug_B/1)),log_slope] | 29.267 | 31 | 0.555 |
tau_log_beta[STRAT,I(log(drug_C/1)),intercept] | 31.053 | 31 | 0.464 |
tau_log_beta[STRAT,I(log(drug_C/1)),log_slope] | 25.101 | 31 | 0.763 |
## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
##
## Matrix products: default
## BLAS: /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
##
## locale:
## [1] C
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] mvtnorm_1.0-11 RBesT_1.4-0 tibble_2.1.3
## [4] rstan_2.19.2 StanHeaders_2.19.0 abind_1.4-5
## [7] Formula_1.2-3 checkmate_1.9.4 OncoBayes2_0.6-4
## [10] testthat_2.2.1 Rcpp_1.0.2 devtools_2.2.1
## [13] usethis_1.5.1 ggplot2_3.2.1 broom_0.5.2
## [16] tidyr_1.0.0 dplyr_0.8.3 assertthat_0.2.1
## [19] knitr_1.25 rmarkdown_1.16
##
## loaded via a namespace (and not attached):
## [1] lattice_0.20-38 prettyunits_1.0.2 ps_1.3.0
## [4] zeallot_0.1.0 rprojroot_1.3-2 digest_0.6.21
## [7] plyr_1.8.4 R6_2.4.0 ggridges_0.5.1
## [10] backports_1.1.5 stats4_3.6.1 evaluate_0.14
## [13] highr_0.8 pillar_1.4.2 rlang_0.4.0
## [16] lazyeval_0.2.2 rstudioapi_0.10 callr_3.3.2
## [19] labeling_0.3 desc_1.2.0 stringr_1.4.0
## [22] loo_2.1.0 munsell_0.5.0 compiler_3.6.1
## [25] xfun_0.10 pkgconfig_2.0.3 pkgbuild_1.0.6
## [28] rstantools_2.0.0 htmltools_0.4.0 tidyselect_0.2.5
## [31] gridExtra_2.3 codetools_0.2-16 matrixStats_0.55.0
## [34] crayon_1.3.4 withr_2.1.2 grid_3.6.1
## [37] nlme_3.1-141 gtable_0.3.0 lifecycle_0.1.0
## [40] magrittr_1.5 scales_1.0.0 cli_1.1.0
## [43] stringi_1.4.3 fs_1.3.1 remotes_2.1.0
## [46] ellipsis_0.3.0 generics_0.0.2 vctrs_0.2.0
## [49] glue_1.3.1 purrr_0.3.3 processx_3.4.1
## [52] pkgload_1.0.2 parallel_3.6.1 yaml_2.2.0
## [55] inline_0.3.15 colorspace_1.4-1 sessioninfo_1.1.1
## [58] bayesplot_1.7.0 memoise_1.1.0