---
title: "Ensemble methods: the ionosphere"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Ensemble methods: the ionosphere}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---





Ensembles of trees against the single tree they are built on, and how two models level on
accuracy can distribute their errors quite differently.

The other vignettes are listed by `vignette (package = "fdm2id")`; they use the same
handful of functions on other data, and can be read in any order.


``` r
library (fdm2id)
```

# The data

Radar measurements of the ionosphere -- the upper layer of the atmosphere -- under
high-frequency waves. The goal is to detect free electrons there. 351 observations, 33 numeric
attributes, and a two-class target: `g` for a "good" return, one that shows structure in the
ionosphere, `b` for a return that does not.

The methods under study are logistic regression, CART, bagging, random forests, AdaBoost and
gradient boosting.


``` r
data (ionosphere)
dim (ionosphere)
#> [1] 351  34
table (ionosphere [, 34])
#> 
#>   b   g 
#> 126 225
```


``` r
plotdata (ionosphere [, -34], ionosphere [, 34], type = "pca")
```

<div class="figure" style="text-align: center">
<img src="figure/v06-ensemble-methods-ionosphere-unnamed-chunk-4-1.png" alt="plot of chunk unnamed-chunk-4"  />
<p class="caption">plot of chunk unnamed-chunk-4</p>
</div>

# Question 1. Which method predicts best?

`BAGGING` and `ADABOOST` need to be told what to ensemble; `performance` passes
`learningmethod` on to them.


``` r
# Variable on both counts: the bootstrap draws the ten resamples, and four of the six methods
# are randomised in themselves -- bagging and the forest draw their samples and their
# variables, the two boosting methods their subsamples. Without 'seed' the table moves by a
# few thousandths at every run, which is more than the gap between the two leaders.
performance (c (LR, CART, BAGGING, RANDOMFOREST, ADABOOST, GRADIENTBOOSTING),
             ionosphere [, -34], ionosphere [, 34], type = "evaluation",
             protocol = "bootstrap", eval = "accuracy", nruns = 10, seed = 0,
             learningmethod = CART)
#>                   accuracy
#> LR               0.8626247
#> CART             0.8864160
#> BAGGING          0.9071374
#> RANDOMFOREST     0.9447429
#> ADABOOST         0.9447429
#> GRADIENTBOOSTING 0.9178818
```

**Answer.** *Under a bootstrap evaluation, random forests and AdaBoost give the best
accuracies. Note the shape of the ranking rather than its exact order: every ensemble of trees
beats the single tree it is built on, and the single tree beats the linear model.*

# Question 2. How do the two best differ, in false positives and false negatives?

An accuracy hides which of the two errors a model makes. Three views of the same difference --
the ROC curves first, which judge the *ranking* of the observations rather than the decision:


``` r
performance (c (RANDOMFOREST, ADABOOST), ionosphere [, -34], ionosphere [, 34],
             type = "roc", protocol = "bootstrap", nruns = 10, fuzzy = TRUE, seed = 0,
             learningmethod = CART)
```

<div class="figure" style="text-align: center">
<img src="figure/v06-ensemble-methods-ionosphere-unnamed-chunk-6-1.png" alt="plot of chunk unnamed-chunk-6"  />
<p class="caption">plot of chunk unnamed-chunk-6</p>
</div>

then precision and recall, which are defined for one class against the rest -- `b` here, since
`performance` takes the first level of the target unless `positive` says otherwise:


``` r
performance (c (RANDOMFOREST, ADABOOST), ionosphere [, -34], ionosphere [, 34],
             type = "evaluation", protocol = "bootstrap",
             eval = c ("precision", "recall"), nruns = 10, seed = 0, learningmethod = CART)
#>              precision    recall
#> RANDOMFOREST 0.9503386 0.8863158
#> ADABOOST     0.9627907 0.8715789
```

then the two confusion matrices:


``` r
performance (RANDOMFOREST, ionosphere [, -34], ionosphere [, 34], type = "confusion",
             protocol = "bootstrap", nruns = 10, seed = 0)
```

<div class="figure" style="text-align: center">
<img src="figure/v06-ensemble-methods-ionosphere-unnamed-chunk-8-1.png" alt="plot of chunk unnamed-chunk-8"  />
<p class="caption">plot of chunk unnamed-chunk-8</p>
</div>

```
#>            Predicted labels
#> True labels          b          g
#>           b 0.89052632 0.10947368
#>           g 0.02415459 0.97584541
performance (ADABOOST, ionosphere [, -34], ionosphere [, 34], type = "confusion",
             protocol = "bootstrap", nruns = 10, seed = 0, learningmethod = CART)
```

<div class="figure" style="text-align: center">
<img src="figure/v06-ensemble-methods-ionosphere-unnamed-chunk-8-2.png" alt="plot of chunk unnamed-chunk-8"  />
<p class="caption">plot of chunk unnamed-chunk-8</p>
</div>

```
#>            Predicted labels
#> True labels          b          g
#>           b 0.87789474 0.12210526
#>           g 0.01449275 0.98550725
```

**Answer.** *The two are level on accuracy but do not distribute their errors the same way.
AdaBoost is the more reluctant of the two to answer `b`: it recovers `g` slightly better and
`b` slightly worse, so it is the more precise and the less sensitive on the minority class.
Which of the two that makes preferable is not a question the data answers -- it depends on
what a missed `b` costs against a false one.*