A-quick-tour-of-StMoE

Introduction

StMoE (Skew-t Mixture-of-Experts) provides a flexible and robust modelling framework for heterogenous data with possibly skewed, heavy-tailed distributions and corrupted by atypical observations. StMoE consists of a mixture of K skew-t expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:

Model estimation/learning is performed by a dedicated expectation conditional maximization (ECM) algorithm by maximizing the observed data log-likelihood. We provide simulated examples to illustrate the use of the model in model-based clustering of heterogeneous regression data and in fitting non-linear regression functions.

It was written in R Markdown, using the knitr package for production.

See help(package="meteorits") for further details and references provided by citation("meteorits").

Application to a simulated dataset

Generate sample

n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
sigmak <- c(0.5, 0.5) # Standard deviations of the experts
lambdak <- c(3, 5) # Skewness parameters of the experts
nuk <- c(5, 7) # Degrees of freedom of the experts network t densities
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)

# Generate sample of size n
sample <- sampleUnivStMoE(alphak = alphak, betak = betak, sigmak = sigmak, 
                          lambdak = lambdak, nuk = nuk, x = x)
y <- sample$y

Set up StMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

stmoe <- emStMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
                 threshold, verbose, verbose_IRLS)
## EM - StMoE: Iteration: 1 | log-likelihood: -360.509785387114
## EM - StMoE: Iteration: 2 | log-likelihood: -345.89233924966
## EM - StMoE: Iteration: 3 | log-likelihood: -335.763644585029
## EM - StMoE: Iteration: 4 | log-likelihood: -329.693266262759
## EM - StMoE: Iteration: 5 | log-likelihood: -325.723643093316
## EM - StMoE: Iteration: 6 | log-likelihood: -322.876747985193
## EM - StMoE: Iteration: 7 | log-likelihood: -320.573638336893
## EM - StMoE: Iteration: 8 | log-likelihood: -318.469179770404
## EM - StMoE: Iteration: 9 | log-likelihood: -316.357499688118
## EM - StMoE: Iteration: 10 | log-likelihood: -314.121282656448
## EM - StMoE: Iteration: 11 | log-likelihood: -311.70645585543
## EM - StMoE: Iteration: 12 | log-likelihood: -309.119406183489
## EM - StMoE: Iteration: 13 | log-likelihood: -306.414506924394
## EM - StMoE: Iteration: 14 | log-likelihood: -303.648921573317
## EM - StMoE: Iteration: 15 | log-likelihood: -300.871094933106
## EM - StMoE: Iteration: 16 | log-likelihood: -298.12290461117
## EM - StMoE: Iteration: 17 | log-likelihood: -295.440413687371
## EM - StMoE: Iteration: 18 | log-likelihood: -292.85807105632
## EM - StMoE: Iteration: 19 | log-likelihood: -290.4018382511
## EM - StMoE: Iteration: 20 | log-likelihood: -288.087220640608
## EM - StMoE: Iteration: 21 | log-likelihood: -285.915401586999
## EM - StMoE: Iteration: 22 | log-likelihood: -283.902123662212
## EM - StMoE: Iteration: 23 | log-likelihood: -282.045478146665
## EM - StMoE: Iteration: 24 | log-likelihood: -280.341365485725
## EM - StMoE: Iteration: 25 | log-likelihood: -278.784895535853
## EM - StMoE: Iteration: 26 | log-likelihood: -277.368298714391
## EM - StMoE: Iteration: 27 | log-likelihood: -276.080629864303
## EM - StMoE: Iteration: 28 | log-likelihood: -274.905849520119
## EM - StMoE: Iteration: 29 | log-likelihood: -273.836624622767
## EM - StMoE: Iteration: 30 | log-likelihood: -272.864373159931
## EM - StMoE: Iteration: 31 | log-likelihood: -271.980427297331
## EM - StMoE: Iteration: 32 | log-likelihood: -271.180516732136
## EM - StMoE: Iteration: 33 | log-likelihood: -270.456388802176
## EM - StMoE: Iteration: 34 | log-likelihood: -269.800503092153
## EM - StMoE: Iteration: 35 | log-likelihood: -269.204729840099
## EM - StMoE: Iteration: 36 | log-likelihood: -268.662887759472
## EM - StMoE: Iteration: 37 | log-likelihood: -268.16933039602
## EM - StMoE: Iteration: 38 | log-likelihood: -267.719570191348
## EM - StMoE: Iteration: 39 | log-likelihood: -267.31117627337
## EM - StMoE: Iteration: 40 | log-likelihood: -266.940959723609
## EM - StMoE: Iteration: 41 | log-likelihood: -266.60546042297
## EM - StMoE: Iteration: 42 | log-likelihood: -266.301250441547
## EM - StMoE: Iteration: 43 | log-likelihood: -266.024192198332
## EM - StMoE: Iteration: 44 | log-likelihood: -265.770810269999
## EM - StMoE: Iteration: 45 | log-likelihood: -265.538953234543
## EM - StMoE: Iteration: 46 | log-likelihood: -265.326560060158
## EM - StMoE: Iteration: 47 | log-likelihood: -265.131591858709
## EM - StMoE: Iteration: 48 | log-likelihood: -264.952024581947
## EM - StMoE: Iteration: 49 | log-likelihood: -264.78611639483
## EM - StMoE: Iteration: 50 | log-likelihood: -264.632488017487
## EM - StMoE: Iteration: 51 | log-likelihood: -264.490497199337
## EM - StMoE: Iteration: 52 | log-likelihood: -264.358933275829
## EM - StMoE: Iteration: 53 | log-likelihood: -264.236564857992
## EM - StMoE: Iteration: 54 | log-likelihood: -264.122635568598
## EM - StMoE: Iteration: 55 | log-likelihood: -264.016360124777
## EM - StMoE: Iteration: 56 | log-likelihood: -263.917067209161
## EM - StMoE: Iteration: 57 | log-likelihood: -263.824278279225
## EM - StMoE: Iteration: 58 | log-likelihood: -263.737847241062
## EM - StMoE: Iteration: 59 | log-likelihood: -263.657199042171
## EM - StMoE: Iteration: 60 | log-likelihood: -263.581911205964
## EM - StMoE: Iteration: 61 | log-likelihood: -263.511490137066
## EM - StMoE: Iteration: 62 | log-likelihood: -263.44536937813
## EM - StMoE: Iteration: 63 | log-likelihood: -263.383225802908
## EM - StMoE: Iteration: 64 | log-likelihood: -263.324736224837
## EM - StMoE: Iteration: 65 | log-likelihood: -263.269745913256
## EM - StMoE: Iteration: 66 | log-likelihood: -263.217949394869
## EM - StMoE: Iteration: 67 | log-likelihood: -263.169088385566
## EM - StMoE: Iteration: 68 | log-likelihood: -263.122940505852
## EM - StMoE: Iteration: 69 | log-likelihood: -263.079311398604
## EM - StMoE: Iteration: 70 | log-likelihood: -263.038029097437
## EM - StMoE: Iteration: 71 | log-likelihood: -262.998939908293
## EM - StMoE: Iteration: 72 | log-likelihood: -262.961816485371
## EM - StMoE: Iteration: 73 | log-likelihood: -262.926637136731
## EM - StMoE: Iteration: 74 | log-likelihood: -262.893327860247
## EM - StMoE: Iteration: 75 | log-likelihood: -262.861750342001
## EM - StMoE: Iteration: 76 | log-likelihood: -262.831784751771
## EM - StMoE: Iteration: 77 | log-likelihood: -262.803455558676
## EM - StMoE: Iteration: 78 | log-likelihood: -262.776749490598
## EM - StMoE: Iteration: 79 | log-likelihood: -262.751530202972
## EM - StMoE: Iteration: 80 | log-likelihood: -262.727678988675
## EM - StMoE: Iteration: 81 | log-likelihood: -262.705091621633
## EM - StMoE: Iteration: 82 | log-likelihood: -262.683675886015
## EM - StMoE: Iteration: 83 | log-likelihood: -262.663349622465
## EM - StMoE: Iteration: 84 | log-likelihood: -262.644039168417
## EM - StMoE: Iteration: 85 | log-likelihood: -262.625678101714
## EM - StMoE: Iteration: 86 | log-likelihood: -262.6082062198
## EM - StMoE: Iteration: 87 | log-likelihood: -262.591568703449
## EM - StMoE: Iteration: 88 | log-likelihood: -262.575715426209
## EM - StMoE: Iteration: 89 | log-likelihood: -262.560600379801
## EM - StMoE: Iteration: 90 | log-likelihood: -262.546181192461
## EM - StMoE: Iteration: 91 | log-likelihood: -262.532418722308
## EM - StMoE: Iteration: 92 | log-likelihood: -262.519276711689
## EM - StMoE: Iteration: 93 | log-likelihood: -262.506721491404
## EM - StMoE: Iteration: 94 | log-likelihood: -262.494721726008
## EM - StMoE: Iteration: 95 | log-likelihood: -262.483248193127
## EM - StMoE: Iteration: 96 | log-likelihood: -262.472273591148
## EM - StMoE: Iteration: 97 | log-likelihood: -262.461772370683
## EM - StMoE: Iteration: 98 | log-likelihood: -262.45172058609
## EM - StMoE: Iteration: 99 | log-likelihood: -262.442095764019
## EM - StMoE: Iteration: 100 | log-likelihood: -262.432876786458
## EM - StMoE: Iteration: 101 | log-likelihood: -262.424043786234
## EM - StMoE: Iteration: 102 | log-likelihood: -262.415634361485
## EM - StMoE: Iteration: 103 | log-likelihood: -262.407566188135
## EM - StMoE: Iteration: 104 | log-likelihood: -262.399824071728
## EM - StMoE: Iteration: 105 | log-likelihood: -262.392393461813
## EM - StMoE: Iteration: 106 | log-likelihood: -262.385260450693
## EM - StMoE: Iteration: 107 | log-likelihood: -262.378390372579
## EM - StMoE: Iteration: 108 | log-likelihood: -262.371802712591
## EM - StMoE: Iteration: 109 | log-likelihood: -262.365478324097
## EM - StMoE: Iteration: 110 | log-likelihood: -262.35940513916
## EM - StMoE: Iteration: 111 | log-likelihood: -262.353571735936
## EM - StMoE: Iteration: 112 | log-likelihood: -262.347967294587
## EM - StMoE: Iteration: 113 | log-likelihood: -262.342581557779
## EM - StMoE: Iteration: 114 | log-likelihood: -262.337404794965
## EM - StMoE: Iteration: 115 | log-likelihood: -262.332427769862
## EM - StMoE: Iteration: 116 | log-likelihood: -262.327641710653
## EM - StMoE: Iteration: 117 | log-likelihood: -262.323038282561
## EM - StMoE: Iteration: 118 | log-likelihood: -262.318609562507
## EM - StMoE: Iteration: 119 | log-likelihood: -262.314348015621
## EM - StMoE: Iteration: 120 | log-likelihood: -262.310246473403
## EM - StMoE: Iteration: 121 | log-likelihood: -262.306298113385
## EM - StMoE: Iteration: 122 | log-likelihood: -262.302496440148
## EM - StMoE: Iteration: 123 | log-likelihood: -262.298835267573
## EM - StMoE: Iteration: 124 | log-likelihood: -262.295308702235
## EM - StMoE: Iteration: 125 | log-likelihood: -262.291911127821
## EM - StMoE: Iteration: 126 | log-likelihood: -262.288637190529
## EM - StMoE: Iteration: 127 | log-likelihood: -262.285481785335
## EM - StMoE: Iteration: 128 | log-likelihood: -262.282440043095
## EM - StMoE: Iteration: 129 | log-likelihood: -262.279507318398
## EM - StMoE: Iteration: 130 | log-likelihood: -262.27667917814
## EM - StMoE: Iteration: 131 | log-likelihood: -262.273951390746
## EM - StMoE: Iteration: 132 | log-likelihood: -262.271319916013
## EM - StMoE: Iteration: 133 | log-likelihood: -262.268780890033

Summary

stmoe$summary()
## ------------------------------------------
## Fitted Skew t Mixture-of-Experts model
## ------------------------------------------
## 
## StMoE model with K = 2 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -262.2688 12 -274.2688 -299.5564 -299.5535
## 
## Clustering table (Number of observations in each expert):
## 
##   1   2 
## 249 251 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)
## 1   -0.02272258 -0.02494224
## X^1  2.52030178 -2.58584153
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##      0.2965375     0.5003761

Plots

Mean curve

stmoe$plot(what = "meancurve")

Confidence regions

stmoe$plot(what = "confregions")

Clusters

stmoe$plot(what = "clusters")

Log-likelihood

stmoe$plot(what = "loglikelihood")

Application to real dataset

Load data

library(MASS)
data("mcycle")
x <- mcycle$times
y <- mcycle$accel

Set up StMoE model parameters

K <- 4 # Number of regressors/experts
p <- 2 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

stmoe <- emStMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
                 threshold, verbose, verbose_IRLS)
## EM - StMoE: Iteration: 1 | log-likelihood: -596.703140667053
## EM - StMoE: Iteration: 2 | log-likelihood: -591.632790111225
## EM - StMoE: Iteration: 3 | log-likelihood: -586.786442552368
## EM - StMoE: Iteration: 4 | log-likelihood: -582.700075930783
## EM - StMoE: Iteration: 5 | log-likelihood: -574.870137160854
## EM - StMoE: Iteration: 6 | log-likelihood: -568.727109036854
## EM - StMoE: Iteration: 7 | log-likelihood: -564.709403127665
## EM - StMoE: Iteration: 8 | log-likelihood: -562.765954346884
## EM - StMoE: Iteration: 9 | log-likelihood: -561.706121191199
## EM - StMoE: Iteration: 10 | log-likelihood: -560.961677952306
## EM - StMoE: Iteration: 11 | log-likelihood: -560.607655590676
## EM - StMoE: Iteration: 12 | log-likelihood: -560.440339710116
## EM - StMoE: Iteration: 13 | log-likelihood: -560.321203993309
## EM - StMoE: Iteration: 14 | log-likelihood: -560.21444740852
## EM - StMoE: Iteration: 15 | log-likelihood: -560.109576386548
## EM - StMoE: Iteration: 16 | log-likelihood: -560.001982792494
## EM - StMoE: Iteration: 17 | log-likelihood: -559.888959143282
## EM - StMoE: Iteration: 18 | log-likelihood: -559.769353742445
## EM - StMoE: Iteration: 19 | log-likelihood: -559.644282845424
## EM - StMoE: Iteration: 20 | log-likelihood: -559.520710831432
## EM - StMoE: Iteration: 21 | log-likelihood: -559.414555541112
## EM - StMoE: Iteration: 22 | log-likelihood: -559.342171749609
## EM - StMoE: Iteration: 23 | log-likelihood: -559.296769956063
## EM - StMoE: Iteration: 24 | log-likelihood: -559.25968724734
## EM - StMoE: Iteration: 25 | log-likelihood: -559.220788055369
## EM - StMoE: Iteration: 26 | log-likelihood: -559.17634035457
## EM - StMoE: Iteration: 27 | log-likelihood: -559.124630473172
## EM - StMoE: Iteration: 28 | log-likelihood: -559.064923183722
## EM - StMoE: Iteration: 29 | log-likelihood: -558.997478269702
## EM - StMoE: Iteration: 30 | log-likelihood: -558.922855290055
## EM - StMoE: Iteration: 31 | log-likelihood: -558.841628375018
## EM - StMoE: Iteration: 32 | log-likelihood: -558.754039725864
## EM - StMoE: Iteration: 33 | log-likelihood: -558.659517097186
## EM - StMoE: Iteration: 34 | log-likelihood: -558.558920590589
## EM - StMoE: Iteration: 35 | log-likelihood: -558.454042595269
## EM - StMoE: Iteration: 36 | log-likelihood: -558.347345356857
## EM - StMoE: Iteration: 37 | log-likelihood: -558.240409624742
## EM - StMoE: Iteration: 38 | log-likelihood: -558.133472925921
## EM - StMoE: Iteration: 39 | log-likelihood: -558.026363741142
## EM - StMoE: Iteration: 40 | log-likelihood: -557.92039510123
## EM - StMoE: Iteration: 41 | log-likelihood: -557.817170063815
## EM - StMoE: Iteration: 42 | log-likelihood: -557.718631534061
## EM - StMoE: Iteration: 43 | log-likelihood: -557.626341897324
## EM - StMoE: Iteration: 44 | log-likelihood: -557.54138647031
## EM - StMoE: Iteration: 45 | log-likelihood: -557.464724712915
## EM - StMoE: Iteration: 46 | log-likelihood: -557.396337062975
## EM - StMoE: Iteration: 47 | log-likelihood: -557.33598838839
## EM - StMoE: Iteration: 48 | log-likelihood: -557.284567853769
## EM - StMoE: Iteration: 49 | log-likelihood: -557.240817690723
## EM - StMoE: Iteration: 50 | log-likelihood: -557.205176662084
## EM - StMoE: Iteration: 51 | log-likelihood: -557.175432379439
## EM - StMoE: Iteration: 52 | log-likelihood: -557.150270927437
## EM - StMoE: Iteration: 53 | log-likelihood: -557.128913023965
## EM - StMoE: Iteration: 54 | log-likelihood: -557.110779660512
## EM - StMoE: Iteration: 55 | log-likelihood: -557.095454474865
## EM - StMoE: Iteration: 56 | log-likelihood: -557.08256150635
## EM - StMoE: Iteration: 57 | log-likelihood: -557.071761659815
## EM - StMoE: Iteration: 58 | log-likelihood: -557.062761177142
## EM - StMoE: Iteration: 59 | log-likelihood: -557.055360748357
## EM - StMoE: Iteration: 60 | log-likelihood: -557.049279921637
## EM - StMoE: Iteration: 61 | log-likelihood: -557.044384153839

Summary

stmoe$summary()
## ------------------------------------------
## Fitted Skew t Mixture-of-Experts model
## ------------------------------------------
## 
## StMoE model with K = 4 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -557.0444 30 -587.0444 -630.3996 -630.3953
## 
## Clustering table (Number of observations in each expert):
## 
##  1  2  3  4 
## 28 37 31 37 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)  Beta(k = 3) Beta(k = 4)
## 1   -3.55474768 1254.749181 -2090.708043 294.1900080
## X^1  0.89227302 -136.012526   131.893262 -12.2265788
## X^2 -0.08259914    3.317713    -2.032427   0.1254723
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2) Sigma2(k = 3) Sigma2(k = 4)
##       14.25574      896.8983      335.8382       564.349

Plots

Mean curve

stmoe$plot(what = "meancurve")

Confidence regions

stmoe$plot(what = "confregions")

Clusters

stmoe$plot(what = "clusters")

Log-likelihood

stmoe$plot(what = "loglikelihood")