Simulation based calibration for OncoBayes2

Wed Aug 28 15:53:06 2019

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2019-08-26 19:49:00 UTC
## git hash: 2535d9cf22794728d1eaf0e7c1a41aa5704a727e
## MD5:      fbdf05162120bac28069c8b597ee113a

The MD5 hash of the calibration data file presented here must match the above listed MD5:

##                    calibration.rds 
## "fbdf05162120bac28069c8b597ee113a"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Model 1: Single-agent logistic regression

Component intercept/slopes

Means

Standard deviations (tau’s)

Model 2: Double combination, fully exchangeable

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation

Model 3: Double combination, EXchangeable/NonEXchangeable model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (from exchangeable part)

Mean

Standard deviation (tau)

Model 4: Triple combination, EX/NEX model

Component intercept/slopes: exchangeable mixture component

Means

Standard deviations (tau’s)

Interaction parameters (means from exchangeable part)

Mean

Standard deviation (tau)

\(\chi^2\) Statistic, Model 1: Single-agent logistic regression

param statistic df p.value
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 34.586 31 0.301
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 26.778 31 0.683
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 32.595 31 0.388
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 41.024 31 0.108

\(\chi^2\) Statistic, Model 2: Double combination, fully exchangeable

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 27.206 31 0.662
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 37.101 31 0.208
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 40.006 31 0.129
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 39.002 31 0.153
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 45.638 31 0.044
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 22.266 31 0.874
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 30.067 31 0.514
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 40.947 31 0.109
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 26.016 31 0.721
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 19.949 31 0.937

\(\chi^2\) Statistic, Model 3: Double combination, EXchangeable/NonEXchangeable model

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 27.514 31 0.646
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 32.218 31 0.406
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 23.686 31 0.823
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 23.168 31 0.843
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 32.358 31 0.400
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 26.829 31 0.681
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 27.603 31 0.642
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 21.882 31 0.887
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 25.120 31 0.762
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 15.437 31 0.991

\(\chi^2\) Statistic, Model 4: Triple combination, EX/NEX model

param statistic df p.value
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] 37.152 31 0.207
mu_eta[I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 42.534 31 0.081
mu_eta[I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] 30.630 31 0.485
mu_eta[I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] 48.102 31 0.026
mu_log_beta[I(log(DosesAdm1/dref[1])),intercept] 21.510 31 0.898
mu_log_beta[I(log(DosesAdm1/dref[1])),log_slope] 31.130 31 0.460
mu_log_beta[I(log(DosesAdm2/dref[2])),intercept] 16.160 31 0.987
mu_log_beta[I(log(DosesAdm2/dref[2])),log_slope] 26.515 31 0.696
mu_log_beta[I(log(DosesAdm3/dref[3])),intercept] 27.738 31 0.635
mu_log_beta[I(log(DosesAdm3/dref[3])),log_slope] 19.859 31 0.939
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2] * DosesAdm3/dref[3])] 25.318 31 0.753
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm2/dref[2])] 26.656 31 0.689
tau_eta[1,I(DosesAdm1/dref[1] * DosesAdm3/dref[3])] 34.061 31 0.322
tau_eta[1,I(DosesAdm2/dref[2] * DosesAdm3/dref[3])] 33.754 31 0.336
tau_log_beta[1,I(log(DosesAdm1/dref[1])),intercept] 29.267 31 0.555
tau_log_beta[1,I(log(DosesAdm1/dref[1])),log_slope] 28.390 31 0.601
tau_log_beta[1,I(log(DosesAdm2/dref[2])),intercept] 27.782 31 0.632
tau_log_beta[1,I(log(DosesAdm2/dref[2])),log_slope] 24.467 31 0.791
tau_log_beta[1,I(log(DosesAdm3/dref[3])),intercept] 18.381 31 0.964
tau_log_beta[1,I(log(DosesAdm3/dref[3])),log_slope] 27.085 31 0.668

Session Info

## R version 3.5.3 (2019-03-11)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
## 
## Matrix products: default
## BLAS: /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
## 
## locale:
## [1] C
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] abind_1.4-5          Formula_1.2-3        checkmate_1.8.5     
##  [4] mvtnorm_1.0-8        RBesT_1.4-0          rstan_2.18.2        
##  [7] StanHeaders_2.18.0-1 OncoBayes2_0.4-4     testthat_2.0.1      
## [10] Rcpp_1.0.1           usethis_1.4.0        devtools_2.0.1      
## [13] ggplot2_3.2.0        broom_0.5.1          tidyr_0.8.2         
## [16] dplyr_0.8.3          assertthat_0.2.1     knitr_1.22          
## [19] rmarkdown_1.11      
## 
## loaded via a namespace (and not attached):
##  [1] lattice_0.20-38    prettyunits_1.0.2  ps_1.2.1          
##  [4] rprojroot_1.3-2    digest_0.6.18      R6_2.4.0          
##  [7] plyr_1.8.4         ggridges_0.5.1     backports_1.1.3   
## [10] stats4_3.5.3       evaluate_0.13      highr_0.8         
## [13] pillar_1.4.2       rlang_0.4.0        lazyeval_0.2.1    
## [16] rstudioapi_0.10    callr_3.1.0        labeling_0.3      
## [19] desc_1.2.0         stringr_1.4.0      loo_2.0.0         
## [22] munsell_0.5.0      compiler_3.5.3     xfun_0.6          
## [25] pkgconfig_2.0.2    pkgbuild_1.0.2     rstantools_1.5.1  
## [28] htmltools_0.3.6    tidyselect_0.2.5   tibble_2.1.3      
## [31] gridExtra_2.3      codetools_0.2-16   matrixStats_0.54.0
## [34] crayon_1.3.4       withr_2.1.2        grid_3.5.3        
## [37] nlme_3.1-137       gtable_0.2.0       magrittr_1.5      
## [40] scales_1.0.0       cli_1.0.1          stringi_1.4.3     
## [43] fs_1.2.6           remotes_2.0.2      generics_0.0.2    
## [46] glue_1.3.1         purrr_0.2.5        processx_3.2.1    
## [49] pkgload_1.0.2      parallel_3.5.3     yaml_2.2.0        
## [52] inline_0.3.15      colorspace_1.3-2   sessioninfo_1.1.1 
## [55] bayesplot_1.6.0    memoise_1.1.0