Model Comparison and Norovirus Fit

Model Overview

This app fits different versions of an SIR model to Norovirus infection data.

Model Diagram

The diagram illustrates the model.

Model Diagram

Model Equations

Implementing the models as continuous-time, deterministic systems leads to the following set of ordinary differential equations:

\[ \begin{aligned} \dot S & = - nS - bSI \\ \dot I & = nS + bSI - gI \\ \dot R & = gI \\ \end{aligned} \]

Model Variants

• We explore the following 3 models/hypotheses:
  • The outbreak involved only person-to-person transmission, i.e. n=0.
  • The outbreak involved person-to-person transmission and an outside source of infection (e.g. contaminated food) that lead to constant infections at a fixed rate n>0.
  • The outbreak involved person-to-person transmission and an outside source of infection which was present for some period of time during the outbreak, i.e. n>0 between some time t₁ and t₂ and 0 otherwise.

Data source

The data being used in this app is daily new cases of norovirus for an outbreak at a school camp. See `help(‘norodata’) for more details.

Model comparison

There are different ways to evaluate how well a model fits to data, and to compare between different models. This app shows the approach of using Akaike’s “An Information Criterion” (AIC), or more precisely, we’ll use the one corrected for small sample size, AICc . If we fit by minimizing the sum of squares (SSR), as we do here, the formula for the AICc is:

\[ AICc = N \log(\frac{SSR}{N})+2(K+1)+\frac{2(K+1)(K+2)}{N-K} \] where N is the number of data points, SSR is the sum of squares residual at the final fit, and K is the number of parameters being fit. A lower value means a better model. One nice feature of the AIC is that one can compare as many models as one wants without having issues with p-values and correcting for multiple comparison, and the models do not need to be nested (i.e. each smaller model being contained inside the larger models). That said, AIC has its drawbacks. Nowadays, if one has enough data available, the best approach is to evaluate model performance by a method like cross-validation (hastie11?).

For evaluation of models with different AIC, there is fortunately no arbitrary, “magic” value (like p=0.05). A rule of thumb is that if models differ by AIC of more than 2, the one with the smaller one is considered statistically better supported (don’t use the word ‘significant’ since that is usually associated with a p-value<0.05, which we don’t have here). I tend to be more conservative and want AIC differences to be at least >10 before I’m willing to favor a given model. Also, I think that visual inspection of the fits is useful. If one model has a lower AIC, but the fit doesn’t look that convincing biologically (e.g. very steep increases or decreases in some quantity), I’d be careful drawing very strong conclusions.

Note that the absolute value of the AIC is unimportant and varies from dataset to dataset. Only relative differences matter. And it goes without saying that we can only compare models that are fit to exactly the same data.

Also note that you can only compare between the models you build and evaluate. It is impossible to implement every possible reasonable model, some simplifications always need to be made. Thus, always keep in mind that if you use mechanistic models, like the ones we explore here, to discriminate between potential mechanisms, your conclusion always comes with the caveat based on the way we implemented the mechanisms/processes in our model we conclude that model/mechanism N aligns better with the data than the others.

Task 1

Take a look at the inputs and outputs for the app. It is similar to the Fitting influenza data app (which you should do before this one). Each model variant fits parameters b and g. Model variant 2 also fits a common source infection rate n. Model variant 3 additionally fits times t₁ and t₂ at which infection from the common source starts and stops (i.e., is larger than 0). The best fit estimates are shown under the figure, together with the SSR and AICc. For simplicity, the lower and upper bounds for the on/off times t1 and t2 are set inside the simulator function to the beginning and end of the data time-series. As a result, they can’t be adjusted through the user interface. To find out more about the data, see Further Resources or help(norodata). You can play around a bit with the app before starting with the next task.

Record

• Nothing

Task 2

The setting from which the data comes had a total of 288 susceptible individuals. Use that as starting value for the susceptibles. Assume 1 initial infected, no recovered. We’ll start by fitting model variant 1, which fits parameters b and g. You could set b and g to pretty much any starting value and try to run a large number of iterations with the 3 different solvers until you get a good fit. As you learned in the flu fitting app, that doesn’t always work, starting values are often important. To find good starting values, you can manually change starting values for b and g, run a single iteration and visually compare model and data. If model and data are reasonable close, you could run the optimizer for more iterations. In theory there is a single best fit. However, as discussed in the flu fitting app, optimizers can sometimes get stuck on good, but not best, fits. So it might be for certain settings you don’t end up with a good fit. Some trial and error is usually required. The best fit for this model is (as far as I know) one with an SSR = 1218. Try various combinations of solver type, iterations and starting values until you reach this SSR. Note that for this and the following tasks, you might need to push iterations into the thousands. Record the best fit value for the rate of recovery. Also make a note of the AICc, you’ll need it later.

Record

• Estimate for best-fit value of parameter g (recovery rate)

Task 3

Now switch to model variant 2, which also fits parameter n. Play around with different starting values for the parameters, different optimizers and different numbers of iterations and see what the best fit is you can find. You’ll likely notice that getting a good fit is trickier now that you added one more parameter to be estimated. That’s a general occurence, more parameters make it harder to obtain estimates. The best fit I was able to find is identical to model 1, i.e., the optimizer sets the rate of infection from an external source to n=0. This, of course, gives the same SSR. However, since we have more parameters now, the AICc is larger. Based on comparison of the AICc values for the 2 models we explored so far, we would conclude that model 1 (no additional external source of infection) is a more consistent or parsimonious description of the data.

Record

• AIC for best fit

Task 4

Now switch to model variant 3, which also fits parameters t₁ and t₂. See what best fit you can find. Note that the starting value for t₁ must be 8 days (the start of the data) or greater, otherwise you’ll get an error. I was able, with 10,000 iterations and good starting conditions, to get SSR values <20. My lowest was around 17, you might be able to find an even better fit (lower SSR). Try until you find one with an SSR of 20 or less. Sometimes using the best-fit results from a run as new starting values for a different run while switching between optimizers can help with the getting stuck problem. An SSR of <20 is clearly lower than that for model 1 and 2. AICc will give us an indication of this gain in better fit justifies adding the additional 3 parameters (n, t₁ and t₂). Look at the AICc for your best fit (one for which SSR<20). Is the AICc value lower than that for model 1 and 2? If yes, it means model 3 is favored, otherwise model 1 is favored (we already ruled out model 2).

Record

• Based on AICc, model 3 is favored (TRUE/FALSE)