A-quick-tour-of-NMoE

Introduction

NMoE (Normal Mixtures-of-Experts) provides a flexible modelling framework for heterogenous data with Gaussian distributions. NMoE consists of a mixture of K Normal expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:

It was written in R Markdown, using the knitr package for production.

See help(package="meteorits") for further details and references provided by citation("meteorits").

Application to a simulated dataset

Generate sample

n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
sigmak <- c(1, 1) # Standard deviations of the experts
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)

# Generate sample of size n
sample <- sampleUnivNMoE(alphak = alphak, betak = betak, sigmak = sigmak, x = x)
y <- sample$y

Set up tMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: -859.802294546189
## EM NMoE: Iteration: 2 | log-likelihood: -858.566551943307
## EM NMoE: Iteration: 3 | log-likelihood: -857.903799829465
## EM NMoE: Iteration: 4 | log-likelihood: -856.866888674095
## EM NMoE: Iteration: 5 | log-likelihood: -854.673341505402
## EM NMoE: Iteration: 6 | log-likelihood: -850.052507997069
## EM NMoE: Iteration: 7 | log-likelihood: -841.059782716473
## EM NMoE: Iteration: 8 | log-likelihood: -825.844666022858
## EM NMoE: Iteration: 9 | log-likelihood: -804.895811336778
## EM NMoE: Iteration: 10 | log-likelihood: -782.769232830412
## EM NMoE: Iteration: 11 | log-likelihood: -765.28112146296
## EM NMoE: Iteration: 12 | log-likelihood: -754.389013457001
## EM NMoE: Iteration: 13 | log-likelihood: -748.337452629062
## EM NMoE: Iteration: 14 | log-likelihood: -745.156904197516
## EM NMoE: Iteration: 15 | log-likelihood: -743.561746401266
## EM NMoE: Iteration: 16 | log-likelihood: -742.763294370501
## EM NMoE: Iteration: 17 | log-likelihood: -742.340802230305
## EM NMoE: Iteration: 18 | log-likelihood: -742.09362900551
## EM NMoE: Iteration: 19 | log-likelihood: -741.931204794879
## EM NMoE: Iteration: 20 | log-likelihood: -741.81296475435
## EM NMoE: Iteration: 21 | log-likelihood: -741.720158775935
## EM NMoE: Iteration: 22 | log-likelihood: -741.643557378
## EM NMoE: Iteration: 23 | log-likelihood: -741.578229900185
## EM NMoE: Iteration: 24 | log-likelihood: -741.521305676763
## EM NMoE: Iteration: 25 | log-likelihood: -741.470976134551
## EM NMoE: Iteration: 26 | log-likelihood: -741.426022507401
## EM NMoE: Iteration: 27 | log-likelihood: -741.385576619693
## EM NMoE: Iteration: 28 | log-likelihood: -741.34898972453
## EM NMoE: Iteration: 29 | log-likelihood: -741.315759152619
## EM NMoE: Iteration: 30 | log-likelihood: -741.285482682241
## EM NMoE: Iteration: 31 | log-likelihood: -741.257830199368
## EM NMoE: Iteration: 32 | log-likelihood: -741.23252520399
## EM NMoE: Iteration: 33 | log-likelihood: -741.209332309915
## EM NMoE: Iteration: 34 | log-likelihood: -741.188048504828
## EM NMoE: Iteration: 35 | log-likelihood: -741.168496840121
## EM NMoE: Iteration: 36 | log-likelihood: -741.150521738608
## EM NMoE: Iteration: 37 | log-likelihood: -741.133985413031
## EM NMoE: Iteration: 38 | log-likelihood: -741.118765070637
## EM NMoE: Iteration: 39 | log-likelihood: -741.104750690308
## EM NMoE: Iteration: 40 | log-likelihood: -741.091843227739
## EM NMoE: Iteration: 41 | log-likelihood: -741.079953147972
## EM NMoE: Iteration: 42 | log-likelihood: -741.068999213089
## EM NMoE: Iteration: 43 | log-likelihood: -741.058907471897
## EM NMoE: Iteration: 44 | log-likelihood: -741.049610411511
## EM NMoE: Iteration: 45 | log-likelihood: -741.041046239994
## EM NMoE: Iteration: 46 | log-likelihood: -741.033158275919
## EM NMoE: Iteration: 47 | log-likelihood: -741.025894425731

Summary

nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
## 
## NMoE model with K = 2 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -741.0259  8 -749.0259 -765.8843 -794.8574
## 
## Clustering table (Number of observations in each expert):
## 
##   1   2 
## 215 285 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)
## 1    -0.2860525   0.3402424
## X^1  -2.0261643   3.0165605
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##       1.048244      1.071049

Plots

Mean curve

nmoe$plot(what = "meancurve")

Confidence regions

nmoe$plot(what = "confregions")

Clusters

nmoe$plot(what = "clusters")

Log-likelihood

nmoe$plot(what = "loglikelihood")

Application to a real dataset

Load data

data("tempanomalies")
x <- tempanomalies$Year
y <- tempanomalies$AnnualAnomaly

Set up tMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: 49.297018175833
## EM NMoE: Iteration: 2 | log-likelihood: 51.1368888425327
## EM NMoE: Iteration: 3 | log-likelihood: 55.4912505009536
## EM NMoE: Iteration: 4 | log-likelihood: 63.0861878419702
## EM NMoE: Iteration: 5 | log-likelihood: 69.9617963035736
## EM NMoE: Iteration: 6 | log-likelihood: 73.3599689123057
## EM NMoE: Iteration: 7 | log-likelihood: 75.0875990552462
## EM NMoE: Iteration: 8 | log-likelihood: 76.5122053961316
## EM NMoE: Iteration: 9 | log-likelihood: 78.0229240013591
## EM NMoE: Iteration: 10 | log-likelihood: 79.7862795995033
## EM NMoE: Iteration: 11 | log-likelihood: 82.0353583290607
## EM NMoE: Iteration: 12 | log-likelihood: 85.1676860227326
## EM NMoE: Iteration: 13 | log-likelihood: 89.4913067503075
## EM NMoE: Iteration: 14 | log-likelihood: 93.6548162896807
## EM NMoE: Iteration: 15 | log-likelihood: 95.5131628082516
## EM NMoE: Iteration: 16 | log-likelihood: 96.0751118133032
## EM NMoE: Iteration: 17 | log-likelihood: 96.2790099302208
## EM NMoE: Iteration: 18 | log-likelihood: 96.3842738811018
## EM NMoE: Iteration: 19 | log-likelihood: 96.4626746927347
## EM NMoE: Iteration: 20 | log-likelihood: 96.5365440422305
## EM NMoE: Iteration: 21 | log-likelihood: 96.6139816153684
## EM NMoE: Iteration: 22 | log-likelihood: 96.6985299537183
## EM NMoE: Iteration: 23 | log-likelihood: 96.7919023489221
## EM NMoE: Iteration: 24 | log-likelihood: 96.8946589209055
## EM NMoE: Iteration: 25 | log-likelihood: 97.0062621533563
## EM NMoE: Iteration: 26 | log-likelihood: 97.12502679902
## EM NMoE: Iteration: 27 | log-likelihood: 97.2482174534687
## EM NMoE: Iteration: 28 | log-likelihood: 97.3724183099739
## EM NMoE: Iteration: 29 | log-likelihood: 97.4941856387689
## EM NMoE: Iteration: 30 | log-likelihood: 97.6108680064574
## EM NMoE: Iteration: 31 | log-likelihood: 97.7213488687057
## EM NMoE: Iteration: 32 | log-likelihood: 97.8264521152446
## EM NMoE: Iteration: 33 | log-likelihood: 97.9288302574943
## EM NMoE: Iteration: 34 | log-likelihood: 98.0323173063805
## EM NMoE: Iteration: 35 | log-likelihood: 98.1409534626323
## EM NMoE: Iteration: 36 | log-likelihood: 98.2580063176062
## EM NMoE: Iteration: 37 | log-likelihood: 98.3853375502151
## EM NMoE: Iteration: 38 | log-likelihood: 98.5233682051375
## EM NMoE: Iteration: 39 | log-likelihood: 98.6716465223696
## EM NMoE: Iteration: 40 | log-likelihood: 98.829684002568
## EM NMoE: Iteration: 41 | log-likelihood: 98.9977024084379
## EM NMoE: Iteration: 42 | log-likelihood: 99.1770456571134
## EM NMoE: Iteration: 43 | log-likelihood: 99.3703602328962
## EM NMoE: Iteration: 44 | log-likelihood: 99.5818170193955
## EM NMoE: Iteration: 45 | log-likelihood: 99.8176597707485
## EM NMoE: Iteration: 46 | log-likelihood: 100.087270945232
## EM NMoE: Iteration: 47 | log-likelihood: 100.40479035699
## EM NMoE: Iteration: 48 | log-likelihood: 100.790566128436
## EM NMoE: Iteration: 49 | log-likelihood: 101.267262503262
## EM NMoE: Iteration: 50 | log-likelihood: 101.827927080155
## EM NMoE: Iteration: 51 | log-likelihood: 102.349893110243
## EM NMoE: Iteration: 52 | log-likelihood: 102.639647976158
## EM NMoE: Iteration: 53 | log-likelihood: 102.720766267014
## EM NMoE: Iteration: 54 | log-likelihood: 102.720822228815

Summary

nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
## 
## NMoE model with K = 2 experts:
## 
##  log-likelihood df      AIC     BIC      ICL
##        102.7208  8 94.72082 83.0702 83.18247
## 
## Clustering table (Number of observations in each expert):
## 
##  1  2 
## 84 52 
## 
## Regression coefficients:
## 
##       Beta(k = 1)  Beta(k = 2)
## 1   -12.667344888 -42.36250371
## X^1   0.006474835   0.02149288
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##     0.01352324    0.01193088

Plots

Mean curve

nmoe$plot(what = "meancurve")

Confidence regions

nmoe$plot(what = "confregions")

Clusters

nmoe$plot(what = "clusters")

Log-likelihood

nmoe$plot(what = "loglikelihood")