NMoE (Normal Mixtures-of-Experts) provides a flexible modelling framework for heterogenous data with Gaussian distributions. NMoE consists of a mixture of K Normal expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:
alpha
’s of the softmax net.beta
’s and variances sigma2
’s.It was written in R Markdown, using the knitr package for production.
See help(package="meteorits")
for further details and references provided by citation("meteorits")
.
n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
sigmak <- c(1, 1) # Standard deviations of the experts
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)
# Generate sample of size n
sample <- sampleUnivNMoE(alphak = alphak, betak = betak, sigmak = sigmak, x = x)
y <- sample$y
nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter,
threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: -859.802294546189
## EM NMoE: Iteration: 2 | log-likelihood: -858.566551943307
## EM NMoE: Iteration: 3 | log-likelihood: -857.903799829465
## EM NMoE: Iteration: 4 | log-likelihood: -856.866888674095
## EM NMoE: Iteration: 5 | log-likelihood: -854.673341505402
## EM NMoE: Iteration: 6 | log-likelihood: -850.052507997069
## EM NMoE: Iteration: 7 | log-likelihood: -841.059782716473
## EM NMoE: Iteration: 8 | log-likelihood: -825.844666022858
## EM NMoE: Iteration: 9 | log-likelihood: -804.895811336778
## EM NMoE: Iteration: 10 | log-likelihood: -782.769232830412
## EM NMoE: Iteration: 11 | log-likelihood: -765.28112146296
## EM NMoE: Iteration: 12 | log-likelihood: -754.389013457001
## EM NMoE: Iteration: 13 | log-likelihood: -748.337452629062
## EM NMoE: Iteration: 14 | log-likelihood: -745.156904197516
## EM NMoE: Iteration: 15 | log-likelihood: -743.561746401266
## EM NMoE: Iteration: 16 | log-likelihood: -742.763294370501
## EM NMoE: Iteration: 17 | log-likelihood: -742.340802230305
## EM NMoE: Iteration: 18 | log-likelihood: -742.09362900551
## EM NMoE: Iteration: 19 | log-likelihood: -741.931204794879
## EM NMoE: Iteration: 20 | log-likelihood: -741.81296475435
## EM NMoE: Iteration: 21 | log-likelihood: -741.720158775935
## EM NMoE: Iteration: 22 | log-likelihood: -741.643557378
## EM NMoE: Iteration: 23 | log-likelihood: -741.578229900185
## EM NMoE: Iteration: 24 | log-likelihood: -741.521305676763
## EM NMoE: Iteration: 25 | log-likelihood: -741.470976134551
## EM NMoE: Iteration: 26 | log-likelihood: -741.426022507401
## EM NMoE: Iteration: 27 | log-likelihood: -741.385576619693
## EM NMoE: Iteration: 28 | log-likelihood: -741.34898972453
## EM NMoE: Iteration: 29 | log-likelihood: -741.315759152619
## EM NMoE: Iteration: 30 | log-likelihood: -741.285482682241
## EM NMoE: Iteration: 31 | log-likelihood: -741.257830199368
## EM NMoE: Iteration: 32 | log-likelihood: -741.23252520399
## EM NMoE: Iteration: 33 | log-likelihood: -741.209332309915
## EM NMoE: Iteration: 34 | log-likelihood: -741.188048504828
## EM NMoE: Iteration: 35 | log-likelihood: -741.168496840121
## EM NMoE: Iteration: 36 | log-likelihood: -741.150521738608
## EM NMoE: Iteration: 37 | log-likelihood: -741.133985413031
## EM NMoE: Iteration: 38 | log-likelihood: -741.118765070637
## EM NMoE: Iteration: 39 | log-likelihood: -741.104750690308
## EM NMoE: Iteration: 40 | log-likelihood: -741.091843227739
## EM NMoE: Iteration: 41 | log-likelihood: -741.079953147972
## EM NMoE: Iteration: 42 | log-likelihood: -741.068999213089
## EM NMoE: Iteration: 43 | log-likelihood: -741.058907471897
## EM NMoE: Iteration: 44 | log-likelihood: -741.049610411511
## EM NMoE: Iteration: 45 | log-likelihood: -741.041046239994
## EM NMoE: Iteration: 46 | log-likelihood: -741.033158275919
## EM NMoE: Iteration: 47 | log-likelihood: -741.025894425731
nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
##
## NMoE model with K = 2 experts:
##
## log-likelihood df AIC BIC ICL
## -741.0259 8 -749.0259 -765.8843 -794.8574
##
## Clustering table (Number of observations in each expert):
##
## 1 2
## 215 285
##
## Regression coefficients:
##
## Beta(k = 1) Beta(k = 2)
## 1 -0.2860525 0.3402424
## X^1 -2.0261643 3.0165605
##
## Variances:
##
## Sigma2(k = 1) Sigma2(k = 2)
## 1.048244 1.071049
nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter,
threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: 49.297018175833
## EM NMoE: Iteration: 2 | log-likelihood: 51.1368888425327
## EM NMoE: Iteration: 3 | log-likelihood: 55.4912505009536
## EM NMoE: Iteration: 4 | log-likelihood: 63.0861878419702
## EM NMoE: Iteration: 5 | log-likelihood: 69.9617963035736
## EM NMoE: Iteration: 6 | log-likelihood: 73.3599689123057
## EM NMoE: Iteration: 7 | log-likelihood: 75.0875990552462
## EM NMoE: Iteration: 8 | log-likelihood: 76.5122053961316
## EM NMoE: Iteration: 9 | log-likelihood: 78.0229240013591
## EM NMoE: Iteration: 10 | log-likelihood: 79.7862795995033
## EM NMoE: Iteration: 11 | log-likelihood: 82.0353583290607
## EM NMoE: Iteration: 12 | log-likelihood: 85.1676860227326
## EM NMoE: Iteration: 13 | log-likelihood: 89.4913067503075
## EM NMoE: Iteration: 14 | log-likelihood: 93.6548162896807
## EM NMoE: Iteration: 15 | log-likelihood: 95.5131628082516
## EM NMoE: Iteration: 16 | log-likelihood: 96.0751118133032
## EM NMoE: Iteration: 17 | log-likelihood: 96.2790099302208
## EM NMoE: Iteration: 18 | log-likelihood: 96.3842738811018
## EM NMoE: Iteration: 19 | log-likelihood: 96.4626746927347
## EM NMoE: Iteration: 20 | log-likelihood: 96.5365440422305
## EM NMoE: Iteration: 21 | log-likelihood: 96.6139816153684
## EM NMoE: Iteration: 22 | log-likelihood: 96.6985299537183
## EM NMoE: Iteration: 23 | log-likelihood: 96.7919023489221
## EM NMoE: Iteration: 24 | log-likelihood: 96.8946589209055
## EM NMoE: Iteration: 25 | log-likelihood: 97.0062621533563
## EM NMoE: Iteration: 26 | log-likelihood: 97.12502679902
## EM NMoE: Iteration: 27 | log-likelihood: 97.2482174534687
## EM NMoE: Iteration: 28 | log-likelihood: 97.3724183099739
## EM NMoE: Iteration: 29 | log-likelihood: 97.4941856387689
## EM NMoE: Iteration: 30 | log-likelihood: 97.6108680064574
## EM NMoE: Iteration: 31 | log-likelihood: 97.7213488687057
## EM NMoE: Iteration: 32 | log-likelihood: 97.8264521152446
## EM NMoE: Iteration: 33 | log-likelihood: 97.9288302574943
## EM NMoE: Iteration: 34 | log-likelihood: 98.0323173063805
## EM NMoE: Iteration: 35 | log-likelihood: 98.1409534626323
## EM NMoE: Iteration: 36 | log-likelihood: 98.2580063176062
## EM NMoE: Iteration: 37 | log-likelihood: 98.3853375502151
## EM NMoE: Iteration: 38 | log-likelihood: 98.5233682051375
## EM NMoE: Iteration: 39 | log-likelihood: 98.6716465223696
## EM NMoE: Iteration: 40 | log-likelihood: 98.829684002568
## EM NMoE: Iteration: 41 | log-likelihood: 98.9977024084379
## EM NMoE: Iteration: 42 | log-likelihood: 99.1770456571134
## EM NMoE: Iteration: 43 | log-likelihood: 99.3703602328962
## EM NMoE: Iteration: 44 | log-likelihood: 99.5818170193955
## EM NMoE: Iteration: 45 | log-likelihood: 99.8176597707485
## EM NMoE: Iteration: 46 | log-likelihood: 100.087270945232
## EM NMoE: Iteration: 47 | log-likelihood: 100.40479035699
## EM NMoE: Iteration: 48 | log-likelihood: 100.790566128436
## EM NMoE: Iteration: 49 | log-likelihood: 101.267262503262
## EM NMoE: Iteration: 50 | log-likelihood: 101.827927080155
## EM NMoE: Iteration: 51 | log-likelihood: 102.349893110243
## EM NMoE: Iteration: 52 | log-likelihood: 102.639647976158
## EM NMoE: Iteration: 53 | log-likelihood: 102.720766267014
## EM NMoE: Iteration: 54 | log-likelihood: 102.720822228815
nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
##
## NMoE model with K = 2 experts:
##
## log-likelihood df AIC BIC ICL
## 102.7208 8 94.72082 83.0702 83.18247
##
## Clustering table (Number of observations in each expert):
##
## 1 2
## 84 52
##
## Regression coefficients:
##
## Beta(k = 1) Beta(k = 2)
## 1 -12.667344888 -42.36250371
## X^1 0.006474835 0.02149288
##
## Variances:
##
## Sigma2(k = 1) Sigma2(k = 2)
## 0.01352324 0.01193088