This report documents the results of a simulation based calibration (SBC) run for OncoBayes2
. TODO
The calibration data presented here has been generated at and with the OncoBayes
git version as:
## Created: 2019-08-26 19:49:00 UTC
## git hash: 2535d9cf22794728d1eaf0e7c1a41aa5704a727e
## MD5: fbdf05162120bac28069c8b597ee113a
The MD5 hash of the calibration data file presented here must match the above listed MD5:
## calibration.rds
## "fbdf05162120bac28069c8b597ee113a"
Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
The fake data simulation function returns … TODO. Please refer to the sbc_tools.R
and make_reference_rankhist.R
R programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
param | statistic | df | p.value |
---|---|---|---|
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 34.586 | 31 | 0.301 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 26.778 | 31 | 0.683 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 32.595 | 31 | 0.388 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 41.024 | 31 | 0.108 |
param | statistic | df | p.value |
---|---|---|---|
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 27.206 | 31 | 0.662 |
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 37.101 | 31 | 0.208 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 40.006 | 31 | 0.129 |
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] | 39.002 | 31 | 0.153 |
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] | 45.638 | 31 | 0.044 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 22.266 | 31 | 0.874 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 30.067 | 31 | 0.514 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 40.947 | 31 | 0.109 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] | 26.016 | 31 | 0.721 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] | 19.949 | 31 | 0.937 |
param | statistic | df | p.value |
---|---|---|---|
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 27.514 | 31 | 0.646 |
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 32.218 | 31 | 0.406 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 23.686 | 31 | 0.823 |
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] | 23.168 | 31 | 0.843 |
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] | 32.358 | 31 | 0.400 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 26.829 | 31 | 0.681 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 27.603 | 31 | 0.642 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 21.882 | 31 | 0.887 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] | 25.120 | 31 | 0.762 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] | 15.437 | 31 | 0.991 |
param | statistic | df | p.value |
---|---|---|---|
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 37.152 | 31 | 0.207 |
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 42.534 | 31 | 0.081 |
mu_eta[I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] | 30.630 | 31 | 0.485 |
mu_eta[I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 48.102 | 31 | 0.026 |
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] | 21.510 | 31 | 0.898 |
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] | 31.130 | 31 | 0.460 |
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] | 16.160 | 31 | 0.987 |
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] | 26.515 | 31 | 0.696 |
mu_log_beta[I(log(DosesAdm3/dref[3])),intercept] | 27.738 | 31 | 0.635 |
mu_log_beta[I(log(DosesAdm3/dref[3])),log_slope] | 19.859 | 31 | 0.939 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 25.318 | 31 | 0.753 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] | 26.656 | 31 | 0.689 |
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] | 34.061 | 31 | 0.322 |
tau_eta[1,I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] | 33.754 | 31 | 0.336 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] | 29.267 | 31 | 0.555 |
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] | 28.390 | 31 | 0.601 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] | 27.782 | 31 | 0.632 |
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] | 24.467 | 31 | 0.791 |
tau_log_beta[1,I(log(DosesAdm3/dref[3])),intercept] | 18.381 | 31 | 0.964 |
tau_log_beta[1,I(log(DosesAdm3/dref[3])),log_slope] | 27.085 | 31 | 0.668 |
## R version 3.5.3 (2019-03-11)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
##
## Matrix products: default
## BLAS: /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
##
## locale:
## [1] C
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] abind_1.4-5 Formula_1.2-3 checkmate_1.8.5
## [4] mvtnorm_1.0-8 RBesT_1.4-0 rstan_2.18.2
## [7] StanHeaders_2.18.0-1 OncoBayes2_0.4-4 testthat_2.0.1
## [10] Rcpp_1.0.1 usethis_1.4.0 devtools_2.0.1
## [13] ggplot2_3.2.0 broom_0.5.1 tidyr_0.8.2
## [16] dplyr_0.8.3 assertthat_0.2.1 knitr_1.22
## [19] rmarkdown_1.11
##
## loaded via a namespace (and not attached):
## [1] lattice_0.20-38 prettyunits_1.0.2 ps_1.2.1
## [4] rprojroot_1.3-2 digest_0.6.18 R6_2.4.0
## [7] plyr_1.8.4 ggridges_0.5.1 backports_1.1.3
## [10] stats4_3.5.3 evaluate_0.13 highr_0.8
## [13] pillar_1.4.2 rlang_0.4.0 lazyeval_0.2.1
## [16] rstudioapi_0.10 callr_3.1.0 labeling_0.3
## [19] desc_1.2.0 stringr_1.4.0 loo_2.0.0
## [22] munsell_0.5.0 compiler_3.5.3 xfun_0.6
## [25] pkgconfig_2.0.2 pkgbuild_1.0.2 rstantools_1.5.1
## [28] htmltools_0.3.6 tidyselect_0.2.5 tibble_2.1.3
## [31] gridExtra_2.3 codetools_0.2-16 matrixStats_0.54.0
## [34] crayon_1.3.4 withr_2.1.2 grid_3.5.3
## [37] nlme_3.1-137 gtable_0.2.0 magrittr_1.5
## [40] scales_1.0.0 cli_1.0.1 stringi_1.4.3
## [43] fs_1.2.6 remotes_2.0.2 generics_0.0.2
## [46] glue_1.3.1 purrr_0.2.5 processx_3.2.1
## [49] pkgload_1.0.2 parallel_3.5.3 yaml_2.2.0
## [52] inline_0.3.15 colorspace_1.3-2 sessioninfo_1.1.1
## [55] bayesplot_1.6.0 memoise_1.1.0