Designing Monte Carlo simulations can be a fun and rewarding experience. Whether you are interested in evaluating the performance of a new optimizer, re-evaluating previous research claims (like the ANOVA is ‘robust’ to violations of normality), determine power rates for an upcoming research proposal, or simply to appease a strange thought in your head about a new statistical idea you heard about, Monte Carlo simulations can be incredibly rewarding and are extremely important to the statistically oriented. However, organizing simulations can be a challenge, and all to often people resort to the dreaded for “loop-nesting”, for-ever resulting in confusing, error prone, and simulation specific code. The package SimDesign is one attempt to fix these and other issue that arise when designing Monte Carlo experiments.
Generally speaking, Monte Carlo simulations can be broken into three major operations:
Each operation above represents the essential components of the SimDesign package. The design component is represented by a data.frame object containing the simulation conditions to be investigated, while generate, analyse, and summarise represent user defined functions which comprise the three steps in the simulation. Each of these components are constructed and passed to the runSimulation() function where the simulation steps are evaluated, ultimately returning a data.frame object containing the simulation results.
After loading the SimDesign package, we begin by defining the required user-constructed functions. To expedite this process, a call to SimFunctions() will create a template to be filled in, where all the necessary functional arguments have been pre-assigned. The documentation of each argument can be found in the respective R help files, however there organization is very simple conceptually.
To begin, the following code should be copied and saved to an external source (i.e., text) file.
library(SimDesign)
SimFunctions()#-------------------------------------------------------------------
library(SimDesign)
Design <- expand.grid(condition1 = NA,
condition2 = NA)
#-------------------------------------------------------------------
Generate <- function(condition, fixed_objects = NULL) {
dat <- data.frame()
dat
}
Analyse <- function(condition, dat, fixed_objects = NULL) {
ret <- c(stat1 = NaN, stat2 = NaN)
ret
}
Summarise <- function(condition, results, fixed_objects = NULL) {
ret <- c(bias = NaN, RMSE = NaN)
ret
}
#-------------------------------------------------------------------
results <- runSimulation(design=Design, replications=1000, generate=Generate,
analyse=Analyse, summarise=Summarise)Alternatively, if you are lazy or just don’t like copy-and-pasting, SimFunctions() can write the output to a file by providing a filename argument. The following creates a file (mysim.R) containing the simulation design/execution and required user-defined functions.
SimFunctions('mysim')Alternatively, for larger simulations you may want to use two files, and if you’d prefer to have helpful comments included then these can be achieved with the singlefile and comments arguments, respectively.
SimFunctions('mysim', singlefile = FALSE, comments = TRUE)Personally, I find keeping the design and functions separate when writing real-world simulations, though there are some other reasons to keep them separate. For example, when debugging code (either through the edit arguments or by explicitly using browser() calls) GUIs such as Rstudio are usually better at tracking the debugged functions. As a good amount of your time will initially be spent debugging, it’s good to make this as painless as possible. Second, it’s easier and more fluid to simply source() (keyboard shortcut ctrl + shift + s) the file containing functions and not worry that you might accidentally start running your simulation; though this is a matter of preference. Finally, the structure may be more readable, especially when you return sometime in the future after you’ve long forgotten what you’ve done. In the design file you can describe the simulation study more thoroughly with comments and generally outline the specifics of how the simulation is to be run, while the functions file simply contains the underlying mechanics and cogs required to run the simulation machine.
As a toy example, let’s consider how the following question can be investigated with SimDesign:
Question: How does trimming affect recovering the mean of a distribution? Investigate this using different sample size with Gaussian and \(\chi^2\) distributions. Also demonstrate the effect of using the median to recover the mean.
First, define the condition combinations that should be investigated. In this case we wish to study 4 different sample sizes, and use a symmetric and skewed distribution. The use of expand.grid() is extremely helpful here to create a completely crossed-design for each combination (there are 8 in total).
Design <- expand.grid(sample_size = c(30, 60, 120, 240),
distribution = c('norm', 'chi'))
Design## sample_size distribution
## 1 30 norm
## 2 60 norm
## 3 120 norm
## 4 240 norm
## 5 30 chi
## 6 60 chi
## 7 120 chi
## 8 240 chiEach row in Design represents a unique condition to be studied in the simulation. In this case, the first condition to be studied comes from row 1, where \(N=30\) and the distribution should be normal.
We first start by defining the generate component. The only argument accepted by this function is condition, which will always be a single row from the Design data.frame object and will be of class data.frame. Conditions are run sequentially from row 1 to the last row in Design. It is also possible to pass a fixed_objects object to the function for including fixed sets of population parameters and other conditions, however for this simple simulation this input is not required.
Generate <- function(condition, fixed_objects = NULL) {
N <- condition$sample_size
dist <- condition$distribution
if(dist == 'norm'){
dat <- rnorm(N, mean = 3)
} else if(dist == 'chi'){
dat <- rchisq(N, df = 3)
}
dat
}As we can see from above, Generate() will return a numeric vector of length \(N\) containing the data to be analysed each with a population mean of 3 (because a \(\chi^2\) distribution has a mean equal to its df). Next, we define the analyse component to analyse said data:
Analyse <- function(condition, dat, fixed_objects = NULL) {
M0 <- mean(dat)
M1 <- mean(dat, trim = .1)
M2 <- mean(dat, trim = .2)
med <- median(dat)
ret <- c(mean_no_trim=M0, mean_trim.1=M1, mean_trim.2=M2, median=med)
ret
}This function accepts the data previously returned from Generate() (dat), the condition vector previously mentioned.
At this point we may conceptually think of the first two functions as being run \(R\) times to obtain \(R\) sets of results. In other words, if we wanted the number of replications to be 100, the first two functions would be independently run (at least) 100 times, the results from Analyse() would be stored, and we would then need to summarise these 100 elements into meaningful meta statistics to describe their empirical properties. This is where computing meta-statistics such as bias, root mean-square error, detection rates, and so on are of primary importance. Unsurprisingly, this is the purpose of the summarise component:
Summarise <- function(condition, results, fixed_objects = NULL) {
obs_bias <- bias(results, parameter = 3)
obs_RMSE <- RMSE(results, parameter = 3)
ret <- c(bias=obs_bias, RMSE=obs_RMSE, RE=RE(obs_RMSE))
ret
}Again, condition is the same as was defined before, while results is a matrix containing all the results from Analyse(), where each row represents the result returned from each respective replication and the number of columns is equal to the length of a single vector returned by Analyse().
That sounds much more complicated than it is — all you really need to know for this simulation is that an \(R\) x 4 matrix called results is available to build a suitable summary. Because the results is a matrix, apply() is useful to apply a function over each respective row. The bias and RMSE are obtained for each respective statistic, and the overall result is returned as a vector.
Stopping for a moment and thinking now, each condition will be paired with a unique vector returned from Summarise(). Therefore, you might be thinking that the result returned from the simulation will be in a rectangular form, such as in a matrix or data.frame. Well, you’d be right — good on you for thinking.
The last stage of the SimDesign work-flow is to pass the four defined elements to the runSimulation() function which, unsurprisingly, runs the simulation. There are numerous options available in the function, and these should be investigated by reading the help(runSimulation) HTML file. Options for performing simulations in parallel, storing/resuming temporary results, debugging functions, and so on are available. Below we simply request that each condition be run 1000 times on a single processor, and finally store the results to an object called results.
results <- runSimulation(Design, replications = 1000, generate=Generate,
analyse=Analyse, summarise=Summarise)##
Design row: 1/8; Started: Sun Oct 29 19:05:26 2017; Total elapsed time: 0.00s
##
Design row: 2/8; Started: Sun Oct 29 19:05:26 2017; Total elapsed time: 0.29s
##
Design row: 3/8; Started: Sun Oct 29 19:05:26 2017; Total elapsed time: 0.59s
##
Design row: 4/8; Started: Sun Oct 29 19:05:27 2017; Total elapsed time: 0.88s
##
Design row: 5/8; Started: Sun Oct 29 19:05:27 2017; Total elapsed time: 1.20s
##
Design row: 6/8; Started: Sun Oct 29 19:05:27 2017; Total elapsed time: 1.53s
##
Design row: 7/8; Started: Sun Oct 29 19:05:28 2017; Total elapsed time: 1.82s
##
Design row: 8/8; Started: Sun Oct 29 19:05:28 2017; Total elapsed time: 2.12sresults## sample_size distribution bias.mean_no_trim bias.mean_trim.1
## 1 30 norm 0.00082 -0.00018
## 2 60 norm 0.00182 0.00086
## 3 120 norm 0.00618 0.00521
## 4 240 norm -0.00041 -0.00071
## 5 30 chi 0.00978 -0.30932
## 6 60 chi 0.00832 -0.32727
## 7 120 chi -0.00017 -0.34301
## 8 240 chi -0.00207 -0.34972
## bias.mean_trim.2 bias.median RMSE.mean_no_trim RMSE.mean_trim.1
## 1 0.0015 0.00507 0.179 0.184
## 2 0.0022 0.00224 0.129 0.132
## 3 0.0050 0.00334 0.085 0.088
## 4 -0.0008 -0.00056 0.065 0.066
## 5 -0.4558 -0.60211 0.434 0.515
## 6 -0.4720 -0.62121 0.323 0.447
## 7 -0.4871 -0.62316 0.225 0.403
## 8 -0.4932 -0.63021 0.153 0.380
## RMSE.mean_trim.2 RMSE.median RE.mean_no_trim RE.mean_trim.1
## 1 0.192 0.22 1 1.1
## 2 0.136 0.16 1 1.1
## 3 0.092 0.11 1 1.1
## 4 0.068 0.08 1 1.0
## 5 0.619 0.77 1 1.4
## 6 0.563 0.71 1 1.9
## 7 0.530 0.67 1 3.2
## 8 0.515 0.65 1 6.2
## RE.mean_trim.2 RE.median REPLICATIONS SIM_TIME COMPLETED
## 1 1.2 1.5 1000 0.29s Sun Oct 29 19:05:26 2017
## 2 1.1 1.5 1000 0.30s Sun Oct 29 19:05:26 2017
## 3 1.2 1.6 1000 0.29s Sun Oct 29 19:05:27 2017
## 4 1.1 1.5 1000 0.32s Sun Oct 29 19:05:27 2017
## 5 2.0 3.2 1000 0.33s Sun Oct 29 19:05:27 2017
## 6 3.0 4.8 1000 0.29s Sun Oct 29 19:05:28 2017
## 7 5.5 8.7 1000 0.30s Sun Oct 29 19:05:28 2017
## 8 11.4 18.3 1000 0.33s Sun Oct 29 19:05:28 2017As can be seen from the printed results, each result from the Summarise() function has been paired with the respective conditions, statistics have been properly named, and two additional columns have been appended to the results: REPLICATIONS, which indicates how many time the conditions were performed, and SIM_TIME indicating the time (in seconds) it took to completely finish the respective conditions. A call to View() in the R console may also be a nice way to sift through the results object.
In this case visually inspecting the simulation table is enough to understand what is occurring, though for other Monte Carlo simulations use of ANOVAs, marginalized tables, and graphics should be used to capture the essentially phenomenon in the results. Monte Carlo simulations are just like collecting data for experiments, so be an analyst and present your data as though it were data collected from the real world.
In this particular simulation, it is readily clear that using the un-adjusted mean will adequately recover the population mean with little bias. The precision also seems to increase as sample sizes increase, which is indicated by the decreasing RMSE statistics. Generally, trimming causes less efficiency in the estimates, where greater amounts of trimming result in even less efficiency, and using the median as a proxy to estimate the mean is the least effective method. This can be seen rather clearly in the following table, which prints the relative efficiency of the estimators:
REs <- results[,grepl('RE\\.', colnames(results))]
data.frame(Design, REs)## sample_size distribution RE.mean_no_trim RE.mean_trim.1 RE.mean_trim.2
## 1 30 norm 1 1.1 1.2
## 2 60 norm 1 1.1 1.1
## 3 120 norm 1 1.1 1.2
## 4 240 norm 1 1.0 1.1
## 5 30 chi 1 1.4 2.0
## 6 60 chi 1 1.9 3.0
## 7 120 chi 1 3.2 5.5
## 8 240 chi 1 6.2 11.4
## RE.median
## 1 1.5
## 2 1.5
## 3 1.6
## 4 1.5
## 5 3.2
## 6 4.8
## 7 8.7
## 8 18.3Finally, when the \(\chi^2\) distribution was investigated only the un-adjusted mean accurately portrayed the population mean. This isn’t surprising, because the trimmed mean is after all making inferences about the population trimmed mean, and the median is making inferences about, well, the median. Only when the distributions under investigation are symmetric will the statistics be able to make the same inferences about the mean of the population.
The following is a conceptual breakdown of what runSimulation() is actually doing behind the scenes. Here we demonstrate the results from the first condition (row 1 of Design) to show what each function returns.
A single replication in a Monte Carlo simulation results in the following objects:
(condition <- Design[1, ])## sample_size distribution
## 1 30 normdat <- Generate(condition)
dat## [1] 2.37 3.18 2.16 4.60 3.33 2.18 3.49 3.74 3.58 2.69 4.51 3.39 2.38 0.79
## [15] 4.12 2.96 2.98 3.94 3.82 3.59 3.92 3.78 3.07 1.01 3.62 2.94 2.84 1.53
## [29] 2.52 3.42res <- Analyse(condition, dat)
res## mean_no_trim mean_trim.1 mean_trim.2 median
## 3.1 3.2 3.2 3.3We can see here that Generate() returns a numeric vector which is accepted by Analyse(). The Analyse() function then completes the analysis portion using the generated data, and returns a named vector with the observed parameter estimates. Of course, this is only a single replication and therefore is not really meaningful in the grand scheme of things, so it must be repeated a number of times.
# repeat 1000x
results <- matrix(0, 1000, 4)
colnames(results) <- names(res)
for(i in 1:1000){
dat <- Generate(condition)
res <- Analyse(condition, dat)
results[i, ] <- res
}
head(results)## mean_no_trim mean_trim.1 mean_trim.2 median
## [1,] 3.1 3.1 3.1 2.9
## [2,] 3.1 3.1 3.1 3.1
## [3,] 3.1 3.1 3.0 2.8
## [4,] 2.7 2.6 2.6 2.7
## [5,] 3.2 3.2 3.2 3.0
## [6,] 3.1 3.1 3.0 3.1The matrix stored in results contains 1000 parameter estimates returned from each statistic. After this is obtained we can move on to summarising the output through the Summarise() function to obtain average estimates, their associated sampling error, their efficiency, and so on.
Summarise(condition, results) ## bias.mean_no_trim bias.mean_trim.1 bias.mean_trim.2 bias.median
## -0.0011 -0.0031 -0.0035 -0.0037
## RMSE.mean_no_trim RMSE.mean_trim.1 RMSE.mean_trim.2 RMSE.median
## 0.1739 0.1777 0.1859 0.2146
## RE.mean_no_trim RE.mean_trim.1 RE.mean_trim.2 RE.median
## 1.0000 1.0442 1.1425 1.5225This scheme is then repeated for each row in the Design object until the entire simulation study is complete.
Of course, runSimulation() does much more than this conceptual outline which is why it exists. Namely, errors and warnings are controlled and tracked, data is re-drawn when needed, parallel processing is supported, debugging is easier with the edit input, temporary and full results can be saved to external files, the simulation state can be saved/restored, contains build-in saftey features, and more. The point, however, is that you as the user do not need to be bogged down with the nitty-gritty details of setting up the simulation work-flow/features, and instead can focus all your time on the important generate-analyse-summarise steps needed to get your results.
To access further examples and instructions feel free to visit the package wiki on Github