Checking and Improving Results of package Synth

Generating the result of package Synth

The following code is thus essentially borrowed from the example section of the corresponding help page (all comments have been removed):

library(Synth)

## ##
## ## Synth Package: Implements Synthetic Control Methods.

## ## See http://www.mit.edu/~jhainm/software.htm for additional information.

data(synth.data)
dataprep.out <-
  dataprep(
    foo = synth.data,
    predictors = c("X1", "X2", "X3"),
    predictors.op = "mean",
    dependent = "Y",
    unit.variable = "unit.num",
    time.variable = "year",
    special.predictors = list(
      list("Y", 1991, "mean"),
      list("Y", 1985, "mean"),
      list("Y", 1980, "mean")
    ),
    treatment.identifier = 7,
    controls.identifier = c(29, 2, 13, 17, 32, 38),
    time.predictors.prior = c(1984:1989),
    time.optimize.ssr = c(1984:1990),
    unit.names.variable = "name",
    time.plot = 1984:1996
  )

synth.out <- synth(dataprep.out)

## 
## X1, X0, Z1, Z0 all come directly from dataprep object.
## 
## 
## **************** 
##  searching for synthetic control unit  
##  
## 
## **************** 
## **************** 
## **************** 
## 
## MSPE (LOSS V): 4.720551 
## 
## solution.v:
##  0.003972032 0.00222758 0.1288856 0.3277689 0.008144768 0.5290011 
## 
## solution.w:
##  0.001006191 0.005246634 0.1691651 0.2136205 0.6109586 2.9586e-06

Checking the result

We check the result by applying function improveSynth to synth.out and dataprep.out:

library(MSCMT)
synth2.out <- improveSynth(synth.out,dataprep.out)

## Results reported by package Synth
## =================================
## 
## Optimal V    : 0.00397203172199428 0.0022275802340665 0.128885551930044 
##                0.327768944797584 0.00814476840194525 0.529001122914366
## Optimal W*(V): 0.00100619117797987 0.00524663354000337 0.169165120873083 
##                0.213620460951039 0.610958634379188 2.95858187768921e-06
## with corresponding predictor loss ('loss W') of 0.004733129 
## and corresponding dependent loss ('loss V') of 4.720551.
## 
## 
## Components of W*(V) do not sum to 1, dependent loss ('loss V') of 
## rescaled W*(V) is 4.720551.
## 
## 
## Feasibility of W*(V)
## ====================
## 
## WARNING: W*(V) is NOT optimal and thus infeasible!
## 'True' W*(V): 0 0.00338943037392679 0.158483318047272 0.217351422652097 
##               0.620775828926704 0
## with corresponding predictor loss ('loss W') of 0.004715856 
## and corresponding dependent loss ('loss V') of 4.719351.
## 
## 
## Optimality of V
## ===============
## 
## WARNING: 'Optimal' V (as reported by package Synth) is not optimal (W*(V) was 
##          infeasible), (one of potentially many) 'true' optimal V* (with 
##          sum(V*)=1):
## Optimal V*    : 5.60003568150536e-09 0.560003568150536 1.77473253948985e-08 
##                 5.60003568150536e-09 5.60285964068872e-08 0.439996346873471
## Optimal W*(V*): 0 0 0.045688074469268 0.250556360555955 0.627179685612033 
##                 0.0765758793627435
## with corresponding predictor loss ('loss W') of 7.597302e-09 
## and corresponding dependent loss ('loss V') of 4.440946.

Package Synth generated a (slightly) infeasible solution, returning a (slightly) suboptimal weight vector w for the control units. However, the predictor weights v are (considerably) suboptimal anyway, because the original dependent loss of 4.720551 (as well as the dependent loss for the corrected w 4.719351) is considerably larger than the dependent loss 4.440946 for the optimal predictor weights obtained by improveSynth.

Generating the result of package Synth

genoud is switched on by the corresponding function argument. We capture the output with capture.output because it is very verbose. Furthermore, the calculation is quite lengthy, therefore the results have been cached.¹

if (file.exists("synth3.out.RData")) load ("synth3.out.RData") else {
  set.seed(42)
  out <- capture.output(synth3.out <- synth(dataprep.out,genoud=TRUE))
}  

Checking the result

We again check the result by applying function improveSynth to synth3.out and dataprep.out:

synth4.out <- improveSynth(synth3.out,dataprep.out)

## Results reported by package Synth
## =================================
## 
## Optimal V    : 2.0267533885719e-10 0.282598702518196 0.0021778890504857 
##                0.00278189732871801 0.000373699249646763 0.712067811650278
## Optimal W*(V): 0.0420138249827668 0.0112849868071455 0.0223432791829297 
##                0.218134146153805 0.595395073764248 0.110828688687666
## with corresponding predictor loss ('loss W') of 0.0001350879 
## and corresponding dependent loss ('loss V') of 4.328506.
## 
## 
## Components of W*(V) do not sum to 1, dependent loss ('loss V') of 
## rescaled W*(V) is 4.328506.
## 
## 
## Feasibility of W*(V)
## ====================
## 
## WARNING: W*(V) is NOT optimal and thus infeasible!
## 'True' W*(V): 0 0 0 0.229091487590933 0.715755416007496 0.0551530964015711
## with corresponding predictor loss ('loss W') of 4.4355e-05 
## and corresponding dependent loss ('loss V') of 6.09574.
## 
## 
## Optimality of V
## ===============
## 
## WARNING: 'Optimal' V (as reported by package Synth) is not optimal (W*(V) was 
##          infeasible), (one of potentially many) 'true' optimal V* (with 
##          sum(V*)=1):
## Optimal V*    : 5.60003568150536e-09 0.560003568150536 1.77473253948985e-08 
##                 5.60003568150536e-09 5.60285964068872e-08 0.439996346873471
## Optimal W*(V*): 0 0 0.045688074469268 0.250556360555955 0.627179685612033 
##                 0.0765758793627435
## with corresponding predictor loss ('loss W') of 7.597302e-09 
## and corresponding dependent loss ('loss V') of 4.440946.

Now, package Synth generated a solution with a dependent loss of 4.328506 which is even smaller than the dependent loss 4.440946 obtained by improveSynth. However, the solution generated by Synth is severely infeasible: the inner optimization failed, returning a suboptimal weight vector w for the control units, which itself lead to a wrong calculation of the dependent loss (which, of course, depends on w). Implanting the true optimal w (depending on v) leads to a large increase of the dependent loss, which uncovers the suboptimality of v.

improveSynth is able to detect this severe problem and calculates an improved and feasible solution (the improved solution matches the solution obtained from the first call to improveSynth above, with a dependent loss of 4.440946).