StMoE (Skew-t Mixture-of-Experts) provides a flexible and robust modelling framework for heterogenous data with possibly skewed, heavy-tailed distributions and corrupted by atypical observations. StMoE consists of a mixture of K skew-t expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:
alpha
’s of the softmax net.beta
’s, scale parameters sigma
’s, the skewness parameters lambda
’s and the degree of freedom parameters nu
’s. StMoE thus generalises mixtures of (normal, skew-normal, t, and skew-t) distributions and mixtures of regressions with these distributions. For example, when \(q=0\), we retrieve mixtures of (skew-t, t-, skew-normal, or normal) regressions, and when both \(p=0\) and \(q=0\), it is a mixture of (skew-t, t-, skew-normal, or normal) distributions. It also reduces to the standard (normal, skew-normal, t, and skew-t) distribution when we only use a single expert (\(K=1\)).Model estimation/learning is performed by a dedicated expectation conditional maximization (ECM) algorithm by maximizing the observed data log-likelihood. We provide simulated examples to illustrate the use of the model in model-based clustering of heterogeneous regression data and in fitting non-linear regression functions.
It was written in R Markdown, using the knitr package for production.
See help(package="meteorits")
for further details and references provided by citation("meteorits")
.
n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
sigmak <- c(0.5, 0.5) # Standard deviations of the experts
lambdak <- c(3, 5) # Skewness parameters of the experts
nuk <- c(5, 7) # Degrees of freedom of the experts network t densities
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)
# Generate sample of size n
sample <- sampleUnivStMoE(alphak = alphak, betak = betak, sigmak = sigmak,
lambdak = lambdak, nuk = nuk, x = x)
y <- sample$y
stmoe <- emStMoE(X = x, Y = y, K, p, q, n_tries, max_iter,
threshold, verbose, verbose_IRLS)
## EM - StMoE: Iteration: 1 | log-likelihood: -360.509785387114
## EM - StMoE: Iteration: 2 | log-likelihood: -345.89233924966
## EM - StMoE: Iteration: 3 | log-likelihood: -335.763644585029
## EM - StMoE: Iteration: 4 | log-likelihood: -329.693266262759
## EM - StMoE: Iteration: 5 | log-likelihood: -325.723643093316
## EM - StMoE: Iteration: 6 | log-likelihood: -322.876747985193
## EM - StMoE: Iteration: 7 | log-likelihood: -320.573638336893
## EM - StMoE: Iteration: 8 | log-likelihood: -318.469179770404
## EM - StMoE: Iteration: 9 | log-likelihood: -316.357499688118
## EM - StMoE: Iteration: 10 | log-likelihood: -314.121282656448
## EM - StMoE: Iteration: 11 | log-likelihood: -311.70645585543
## EM - StMoE: Iteration: 12 | log-likelihood: -309.119406183489
## EM - StMoE: Iteration: 13 | log-likelihood: -306.414506924394
## EM - StMoE: Iteration: 14 | log-likelihood: -303.648921573317
## EM - StMoE: Iteration: 15 | log-likelihood: -300.871094933106
## EM - StMoE: Iteration: 16 | log-likelihood: -298.12290461117
## EM - StMoE: Iteration: 17 | log-likelihood: -295.440413687371
## EM - StMoE: Iteration: 18 | log-likelihood: -292.85807105632
## EM - StMoE: Iteration: 19 | log-likelihood: -290.4018382511
## EM - StMoE: Iteration: 20 | log-likelihood: -288.087220640608
## EM - StMoE: Iteration: 21 | log-likelihood: -285.915401586999
## EM - StMoE: Iteration: 22 | log-likelihood: -283.902123662212
## EM - StMoE: Iteration: 23 | log-likelihood: -282.045478146665
## EM - StMoE: Iteration: 24 | log-likelihood: -280.341365485725
## EM - StMoE: Iteration: 25 | log-likelihood: -278.784895535853
## EM - StMoE: Iteration: 26 | log-likelihood: -277.368298714391
## EM - StMoE: Iteration: 27 | log-likelihood: -276.080629864303
## EM - StMoE: Iteration: 28 | log-likelihood: -274.905849520119
## EM - StMoE: Iteration: 29 | log-likelihood: -273.836624622767
## EM - StMoE: Iteration: 30 | log-likelihood: -272.864373159931
## EM - StMoE: Iteration: 31 | log-likelihood: -271.980427297331
## EM - StMoE: Iteration: 32 | log-likelihood: -271.180516732136
## EM - StMoE: Iteration: 33 | log-likelihood: -270.456388802176
## EM - StMoE: Iteration: 34 | log-likelihood: -269.800503092153
## EM - StMoE: Iteration: 35 | log-likelihood: -269.204729840099
## EM - StMoE: Iteration: 36 | log-likelihood: -268.662887759472
## EM - StMoE: Iteration: 37 | log-likelihood: -268.16933039602
## EM - StMoE: Iteration: 38 | log-likelihood: -267.719570191348
## EM - StMoE: Iteration: 39 | log-likelihood: -267.31117627337
## EM - StMoE: Iteration: 40 | log-likelihood: -266.940959723609
## EM - StMoE: Iteration: 41 | log-likelihood: -266.60546042297
## EM - StMoE: Iteration: 42 | log-likelihood: -266.301250441547
## EM - StMoE: Iteration: 43 | log-likelihood: -266.024192198332
## EM - StMoE: Iteration: 44 | log-likelihood: -265.770810269999
## EM - StMoE: Iteration: 45 | log-likelihood: -265.538953234543
## EM - StMoE: Iteration: 46 | log-likelihood: -265.326560060158
## EM - StMoE: Iteration: 47 | log-likelihood: -265.131591858709
## EM - StMoE: Iteration: 48 | log-likelihood: -264.952024581947
## EM - StMoE: Iteration: 49 | log-likelihood: -264.78611639483
## EM - StMoE: Iteration: 50 | log-likelihood: -264.632488017487
## EM - StMoE: Iteration: 51 | log-likelihood: -264.490497199337
## EM - StMoE: Iteration: 52 | log-likelihood: -264.358933275829
## EM - StMoE: Iteration: 53 | log-likelihood: -264.236564857992
## EM - StMoE: Iteration: 54 | log-likelihood: -264.122635568598
## EM - StMoE: Iteration: 55 | log-likelihood: -264.016360124777
## EM - StMoE: Iteration: 56 | log-likelihood: -263.917067209161
## EM - StMoE: Iteration: 57 | log-likelihood: -263.824278279225
## EM - StMoE: Iteration: 58 | log-likelihood: -263.737847241062
## EM - StMoE: Iteration: 59 | log-likelihood: -263.657199042171
## EM - StMoE: Iteration: 60 | log-likelihood: -263.581911205964
## EM - StMoE: Iteration: 61 | log-likelihood: -263.511490137066
## EM - StMoE: Iteration: 62 | log-likelihood: -263.44536937813
## EM - StMoE: Iteration: 63 | log-likelihood: -263.383225802908
## EM - StMoE: Iteration: 64 | log-likelihood: -263.324736224837
## EM - StMoE: Iteration: 65 | log-likelihood: -263.269745913256
## EM - StMoE: Iteration: 66 | log-likelihood: -263.217949394869
## EM - StMoE: Iteration: 67 | log-likelihood: -263.169088385566
## EM - StMoE: Iteration: 68 | log-likelihood: -263.122940505852
## EM - StMoE: Iteration: 69 | log-likelihood: -263.079311398604
## EM - StMoE: Iteration: 70 | log-likelihood: -263.038029097437
## EM - StMoE: Iteration: 71 | log-likelihood: -262.998939908293
## EM - StMoE: Iteration: 72 | log-likelihood: -262.961816485371
## EM - StMoE: Iteration: 73 | log-likelihood: -262.926637136731
## EM - StMoE: Iteration: 74 | log-likelihood: -262.893327860247
## EM - StMoE: Iteration: 75 | log-likelihood: -262.861750342001
## EM - StMoE: Iteration: 76 | log-likelihood: -262.831784751771
## EM - StMoE: Iteration: 77 | log-likelihood: -262.803455558676
## EM - StMoE: Iteration: 78 | log-likelihood: -262.776749490598
## EM - StMoE: Iteration: 79 | log-likelihood: -262.751530202972
## EM - StMoE: Iteration: 80 | log-likelihood: -262.727678988675
## EM - StMoE: Iteration: 81 | log-likelihood: -262.705091621633
## EM - StMoE: Iteration: 82 | log-likelihood: -262.683675886015
## EM - StMoE: Iteration: 83 | log-likelihood: -262.663349622465
## EM - StMoE: Iteration: 84 | log-likelihood: -262.644039168417
## EM - StMoE: Iteration: 85 | log-likelihood: -262.625678101714
## EM - StMoE: Iteration: 86 | log-likelihood: -262.6082062198
## EM - StMoE: Iteration: 87 | log-likelihood: -262.591568703449
## EM - StMoE: Iteration: 88 | log-likelihood: -262.575715426209
## EM - StMoE: Iteration: 89 | log-likelihood: -262.560600379801
## EM - StMoE: Iteration: 90 | log-likelihood: -262.546181192461
## EM - StMoE: Iteration: 91 | log-likelihood: -262.532418722308
## EM - StMoE: Iteration: 92 | log-likelihood: -262.519276711689
## EM - StMoE: Iteration: 93 | log-likelihood: -262.506721491404
## EM - StMoE: Iteration: 94 | log-likelihood: -262.494721726008
## EM - StMoE: Iteration: 95 | log-likelihood: -262.483248193127
## EM - StMoE: Iteration: 96 | log-likelihood: -262.472273591148
## EM - StMoE: Iteration: 97 | log-likelihood: -262.461772370683
## EM - StMoE: Iteration: 98 | log-likelihood: -262.45172058609
## EM - StMoE: Iteration: 99 | log-likelihood: -262.442095764019
## EM - StMoE: Iteration: 100 | log-likelihood: -262.432876786458
## EM - StMoE: Iteration: 101 | log-likelihood: -262.424043786234
## EM - StMoE: Iteration: 102 | log-likelihood: -262.415634361485
## EM - StMoE: Iteration: 103 | log-likelihood: -262.407566188135
## EM - StMoE: Iteration: 104 | log-likelihood: -262.399824071728
## EM - StMoE: Iteration: 105 | log-likelihood: -262.392393461813
## EM - StMoE: Iteration: 106 | log-likelihood: -262.385260450693
## EM - StMoE: Iteration: 107 | log-likelihood: -262.378390372579
## EM - StMoE: Iteration: 108 | log-likelihood: -262.371802712591
## EM - StMoE: Iteration: 109 | log-likelihood: -262.365478324097
## EM - StMoE: Iteration: 110 | log-likelihood: -262.35940513916
## EM - StMoE: Iteration: 111 | log-likelihood: -262.353571735936
## EM - StMoE: Iteration: 112 | log-likelihood: -262.347967294587
## EM - StMoE: Iteration: 113 | log-likelihood: -262.342581557779
## EM - StMoE: Iteration: 114 | log-likelihood: -262.337404794965
## EM - StMoE: Iteration: 115 | log-likelihood: -262.332427769862
## EM - StMoE: Iteration: 116 | log-likelihood: -262.327641710653
## EM - StMoE: Iteration: 117 | log-likelihood: -262.323038282561
## EM - StMoE: Iteration: 118 | log-likelihood: -262.318609562507
## EM - StMoE: Iteration: 119 | log-likelihood: -262.314348015621
## EM - StMoE: Iteration: 120 | log-likelihood: -262.310246473403
## EM - StMoE: Iteration: 121 | log-likelihood: -262.306298113385
## EM - StMoE: Iteration: 122 | log-likelihood: -262.302496440148
## EM - StMoE: Iteration: 123 | log-likelihood: -262.298835267573
## EM - StMoE: Iteration: 124 | log-likelihood: -262.295308702235
## EM - StMoE: Iteration: 125 | log-likelihood: -262.291911127821
## EM - StMoE: Iteration: 126 | log-likelihood: -262.288637190529
## EM - StMoE: Iteration: 127 | log-likelihood: -262.285481785335
## EM - StMoE: Iteration: 128 | log-likelihood: -262.282440043095
## EM - StMoE: Iteration: 129 | log-likelihood: -262.279507318398
## EM - StMoE: Iteration: 130 | log-likelihood: -262.27667917814
## EM - StMoE: Iteration: 131 | log-likelihood: -262.273951390746
## EM - StMoE: Iteration: 132 | log-likelihood: -262.271319916013
## EM - StMoE: Iteration: 133 | log-likelihood: -262.268780890033
stmoe$summary()
## ------------------------------------------
## Fitted Skew t Mixture-of-Experts model
## ------------------------------------------
##
## StMoE model with K = 2 experts:
##
## log-likelihood df AIC BIC ICL
## -262.2688 12 -274.2688 -299.5564 -299.5535
##
## Clustering table (Number of observations in each expert):
##
## 1 2
## 249 251
##
## Regression coefficients:
##
## Beta(k = 1) Beta(k = 2)
## 1 -0.02272258 -0.02494224
## X^1 2.52030178 -2.58584153
##
## Variances:
##
## Sigma2(k = 1) Sigma2(k = 2)
## 0.2965375 0.5003761
stmoe <- emStMoE(X = x, Y = y, K, p, q, n_tries, max_iter,
threshold, verbose, verbose_IRLS)
## EM - StMoE: Iteration: 1 | log-likelihood: -596.703140667053
## EM - StMoE: Iteration: 2 | log-likelihood: -591.632790111225
## EM - StMoE: Iteration: 3 | log-likelihood: -586.786442552368
## EM - StMoE: Iteration: 4 | log-likelihood: -582.700075930783
## EM - StMoE: Iteration: 5 | log-likelihood: -574.870137160854
## EM - StMoE: Iteration: 6 | log-likelihood: -568.727109036854
## EM - StMoE: Iteration: 7 | log-likelihood: -564.709403127665
## EM - StMoE: Iteration: 8 | log-likelihood: -562.765954346884
## EM - StMoE: Iteration: 9 | log-likelihood: -561.706121191199
## EM - StMoE: Iteration: 10 | log-likelihood: -560.961677952306
## EM - StMoE: Iteration: 11 | log-likelihood: -560.607655590676
## EM - StMoE: Iteration: 12 | log-likelihood: -560.440339710116
## EM - StMoE: Iteration: 13 | log-likelihood: -560.321203993309
## EM - StMoE: Iteration: 14 | log-likelihood: -560.21444740852
## EM - StMoE: Iteration: 15 | log-likelihood: -560.109576386548
## EM - StMoE: Iteration: 16 | log-likelihood: -560.001982792494
## EM - StMoE: Iteration: 17 | log-likelihood: -559.888959143282
## EM - StMoE: Iteration: 18 | log-likelihood: -559.769353742445
## EM - StMoE: Iteration: 19 | log-likelihood: -559.644282845424
## EM - StMoE: Iteration: 20 | log-likelihood: -559.520710831432
## EM - StMoE: Iteration: 21 | log-likelihood: -559.414555541112
## EM - StMoE: Iteration: 22 | log-likelihood: -559.342171749609
## EM - StMoE: Iteration: 23 | log-likelihood: -559.296769956063
## EM - StMoE: Iteration: 24 | log-likelihood: -559.25968724734
## EM - StMoE: Iteration: 25 | log-likelihood: -559.220788055369
## EM - StMoE: Iteration: 26 | log-likelihood: -559.17634035457
## EM - StMoE: Iteration: 27 | log-likelihood: -559.124630473172
## EM - StMoE: Iteration: 28 | log-likelihood: -559.064923183722
## EM - StMoE: Iteration: 29 | log-likelihood: -558.997478269702
## EM - StMoE: Iteration: 30 | log-likelihood: -558.922855290055
## EM - StMoE: Iteration: 31 | log-likelihood: -558.841628375018
## EM - StMoE: Iteration: 32 | log-likelihood: -558.754039725864
## EM - StMoE: Iteration: 33 | log-likelihood: -558.659517097186
## EM - StMoE: Iteration: 34 | log-likelihood: -558.558920590589
## EM - StMoE: Iteration: 35 | log-likelihood: -558.454042595269
## EM - StMoE: Iteration: 36 | log-likelihood: -558.347345356857
## EM - StMoE: Iteration: 37 | log-likelihood: -558.240409624742
## EM - StMoE: Iteration: 38 | log-likelihood: -558.133472925921
## EM - StMoE: Iteration: 39 | log-likelihood: -558.026363741142
## EM - StMoE: Iteration: 40 | log-likelihood: -557.92039510123
## EM - StMoE: Iteration: 41 | log-likelihood: -557.817170063815
## EM - StMoE: Iteration: 42 | log-likelihood: -557.718631534061
## EM - StMoE: Iteration: 43 | log-likelihood: -557.626341897324
## EM - StMoE: Iteration: 44 | log-likelihood: -557.54138647031
## EM - StMoE: Iteration: 45 | log-likelihood: -557.464724712915
## EM - StMoE: Iteration: 46 | log-likelihood: -557.396337062975
## EM - StMoE: Iteration: 47 | log-likelihood: -557.33598838839
## EM - StMoE: Iteration: 48 | log-likelihood: -557.284567853769
## EM - StMoE: Iteration: 49 | log-likelihood: -557.240817690723
## EM - StMoE: Iteration: 50 | log-likelihood: -557.205176662084
## EM - StMoE: Iteration: 51 | log-likelihood: -557.175432379439
## EM - StMoE: Iteration: 52 | log-likelihood: -557.150270927437
## EM - StMoE: Iteration: 53 | log-likelihood: -557.128913023965
## EM - StMoE: Iteration: 54 | log-likelihood: -557.110779660512
## EM - StMoE: Iteration: 55 | log-likelihood: -557.095454474865
## EM - StMoE: Iteration: 56 | log-likelihood: -557.08256150635
## EM - StMoE: Iteration: 57 | log-likelihood: -557.071761659815
## EM - StMoE: Iteration: 58 | log-likelihood: -557.062761177142
## EM - StMoE: Iteration: 59 | log-likelihood: -557.055360748357
## EM - StMoE: Iteration: 60 | log-likelihood: -557.049279921637
## EM - StMoE: Iteration: 61 | log-likelihood: -557.044384153839
stmoe$summary()
## ------------------------------------------
## Fitted Skew t Mixture-of-Experts model
## ------------------------------------------
##
## StMoE model with K = 4 experts:
##
## log-likelihood df AIC BIC ICL
## -557.0444 30 -587.0444 -630.3996 -630.3953
##
## Clustering table (Number of observations in each expert):
##
## 1 2 3 4
## 28 37 31 37
##
## Regression coefficients:
##
## Beta(k = 1) Beta(k = 2) Beta(k = 3) Beta(k = 4)
## 1 -3.55474768 1254.749181 -2090.708043 294.1900080
## X^1 0.89227302 -136.012526 131.893262 -12.2265788
## X^2 -0.08259914 3.317713 -2.032427 0.1254723
##
## Variances:
##
## Sigma2(k = 1) Sigma2(k = 2) Sigma2(k = 3) Sigma2(k = 4)
## 14.25574 896.8983 335.8382 564.349