Automated Machine Learning with tidylearn

A First Run

result <- tl_auto_ml(
  iris, Species ~ .,
  time_budget = 30,
  cv_folds = 3
)

result$leaderboard
#> # A tibble: 8 × 3
#>   model            score evaluation
#>   <chr>            <dbl> <chr>     
#> 1 advanced_svm     0.96  cv        
#> 2 advanced_xgboost 0.96  cv        
#> 3 baseline_forest  0.947 cv        
#> 4 clustered_tree   0.94  cv        
#> 5 clustered_forest 0.94  cv        
#> 6 baseline_tree    0.933 cv        
#> 7 pca_tree         0.907 cv        
#> 8 pca_forest       0.9   cv

Three columns: which model, what it scored, and how the score was obtained.

result$best_model
#> tidylearn Model
#> ===============
#> Paradigm: supervised 
#> Method: svm 
#> Task: Classification 
#> Formula: Species ~ . 
#> 
#> Training observations: 150

result$task
#> [1] "classification"
result$metric
#> [1] "accuracy"
round(as.numeric(result$runtime, units = "secs"), 1)
#> [1] 1.9

Regression is the same call with a numeric response — task = "auto" reads it off the data:

result_reg <- tl_auto_ml(
  mtcars, mpg ~ .,
  time_budget = 30,
  cv_folds = 3
)

result_reg$task
#> [1] "regression"

result_reg$leaderboard
#> # A tibble: 11 × 3
#>    model            score evaluation
#>    <chr>            <dbl> <chr>     
#>  1 clustered_forest  2.50 cv        
#>  2 advanced_ridge    2.63 cv        
#>  3 pca_linear        2.68 cv        
#>  4 baseline_forest   2.96 cv        
#>  5 advanced_lasso    3.12 cv        
#>  6 baseline_linear   3.57 cv        
#>  7 pca_forest        3.77 cv        
#>  8 baseline_tree     4.00 cv        
#>  9 pca_tree          4.22 cv        
#> 10 clustered_tree    5.12 cv        
#> 11 clustered_linear  5.67 cv

How the Search Works

Four phases run in order, each gated on the remaining budget and on the use_reduction and use_clustering toggles.

Phase	What it does	Classification	Regression
1. Baselines	Standard models on the raw features	tree, logistic*, forest	tree, linear, forest
2. PCA variants	PCA preprocessing, then the baselines	pca_tree, pca_logistic*, pca_forest	pca_tree, pca_linear, pca_forest
3. Cluster variants	Cluster assignment added as a feature	clustered_tree, clustered_logistic*, clustered_forest	clustered_tree, clustered_linear, clustered_forest
4. Advanced	Heavier methods	svm, xgboost	ridge, lasso

* Logistic regression is binary only, so it appears only when the response has exactly two classes. On a multiclass problem such as iris the baselines are tree and forest.

You can see the phases in what actually got trained:

names(result$models)
#> [1] "baseline_tree"    "baseline_forest"  "pca_tree"         "pca_forest"      
#> [5] "clustered_tree"   "clustered_forest" "advanced_svm"     "advanced_xgboost"

# The same search on a two-class problem picks up the logistic variants
iris_binary <- iris %>%
  filter(Species != "setosa") %>%
  mutate(Species = droplevels(Species))

binary_result <- tl_auto_ml(iris_binary, Species ~ ., time_budget = 30,
                            cv_folds = 3)
names(binary_result$models)
#>  [1] "baseline_tree"      "baseline_logistic"  "baseline_forest"   
#>  [4] "pca_tree"           "pca_logistic"       "pca_forest"        
#>  [7] "clustered_tree"     "clustered_logistic" "clustered_forest"  
#> [10] "advanced_svm"       "advanced_xgboost"

That call emits a run of glm.fit: algorithm did not converge warnings, which this chunk hides only because there are a dozen of them. They are worth understanding rather than ignoring: versicolor and virginica overlap only slightly, and a three-fold split of 100 rows will sometimes produce a fold where the two are perfectly separable. Logistic regression has no finite maximum likelihood estimate on separable data, so the coefficients diverge and glm() says so.

The fit on the full data is fine:

tl_model(iris_binary, Species ~ ., method = "logistic")$spec$method
#> [1] "logistic"

It is the folds, not the dataset. Seeing this on your own data means the classes are close to separable at that sample size — a reason to prefer a regularised method, which is what ridge and lasso are doing in phase 4 of the regression search.

The evaluation column

Each model is fitted on the full training data, then cross-validated if the budget allows. When it does not, the leaderboard falls back to training-set metrics.

table(result$leaderboard$evaluation)
#> 
#> cv 
#>  8

Training metrics are optimistically biased, so a leaderboard mixing "cv" and "train" is not ranking like with like. If you see both, raise the budget or lower cv_folds.

The Time Budget

time_budget is in seconds and is checked between model fits, not during them. Once a model starts it runs to completion, because the wrapped packages execute C-level code that R cannot safely interrupt. Wall-clock time can therefore overshoot by the duration of whichever model started last.

The budget controls how much gets tried. The relationship on this data and this machine:

budgets <- c(2, 5, 10, 30)

sweep <- lapply(budgets, function(b) {
  t0 <- Sys.time()
  r <- tl_auto_ml(iris, Species ~ ., time_budget = b, cv_folds = 3)
  data.frame(
    budget = b,
    elapsed = round(as.numeric(difftime(Sys.time(), t0, units = "secs")), 1),
    models = nrow(r$leaderboard),
    cv_scored = sum(r$leaderboard$evaluation == "cv"),
    best = r$leaderboard$model[1]
  )
})

do.call(rbind, sweep)
#>   budget elapsed models cv_scored          best
#> 1      2     0.0      1         1 baseline_tree
#> 2      5     0.0      1         1 baseline_tree
#> 3     10     0.1      3         3 baseline_tree
#> 4     30     1.7      8         8  advanced_svm

iris is 150 rows, so everything here is quick and the budget is barely touched. On data where a single forest fit takes ten seconds the same numbers look very different, so run the sweep on your own data.

Two things are worth knowing regardless of size:

The forest baseline and every advanced model are only attempted when time_budget >= 30.
Cross-validation is the expensive step. tl_cv(folds = 5) fits five models where a plain fit costs one, so cv_folds is the most effective lever.

Controlling the Search

# Baselines only -- no PCA, no cluster features
baseline_only <- tl_auto_ml(
  iris, Species ~ .,
  use_reduction = FALSE,
  use_clustering = FALSE,
  time_budget = 30,
  cv_folds = 3
)

names(baseline_only$models)
#> [1] "baseline_tree"    "baseline_forest"  "advanced_svm"     "advanced_xgboost"

# Keep PCA, drop clustering
no_clustering <- tl_auto_ml(
  iris, Species ~ .,
  use_clustering = FALSE,
  time_budget = 30,
  cv_folds = 3
)

names(no_clustering$models)
#> [1] "baseline_tree"    "baseline_forest"  "pca_tree"         "pca_forest"      
#> [5] "advanced_svm"     "advanced_xgboost"

Naming a metric changes what the leaderboard optimises. Classification accepts accuracy, precision, recall, sensitivity, specificity, f1, auc and pr_auc; regression accepts rmse, mse, mae, mape and rsq.

by_f1 <- tl_auto_ml(iris_binary, Species ~ ., metric = "f1",
                    time_budget = 30, cv_folds = 3)

by_f1$metric
#> [1] "f1"
by_f1$leaderboard
#> # A tibble: 11 × 3
#>    model              score evaluation
#>    <chr>              <dbl> <chr>     
#>  1 clustered_logistic 0.982 cv        
#>  2 baseline_logistic  0.970 cv        
#>  3 baseline_forest    0.941 cv        
#>  4 advanced_xgboost   0.926 cv        
#>  5 advanced_svm       0.925 cv        
#>  6 clustered_forest   0.911 cv        
#>  7 baseline_tree      0.898 cv        
#>  8 pca_forest         0.894 cv        
#>  9 clustered_tree     0.891 cv        
#> 10 pca_logistic       0.837 cv        
#> 11 pca_tree           0.778 cv

Using the Result

split <- tl_split(iris, prop = 0.7, stratify = "Species", seed = 123)

automl <- tl_auto_ml(split$train, Species ~ ., time_budget = 30, cv_folds = 3)

test_preds <- predict(automl$best_model, new_data = split$test)
mean(test_preds$.pred == split$test$Species)
#> [1] 0.9333333

Any model on the leaderboard can be pulled out by name:

available <- names(automl$models)
available
#> [1] "baseline_tree"    "baseline_forest"  "pca_tree"         "pca_forest"      
#> [5] "clustered_tree"   "clustered_forest" "advanced_svm"     "advanced_xgboost"

scores <- vapply(available, function(nm) {
  preds <- predict(automl$models[[nm]], new_data = split$test)
  mean(preds$.pred == split$test$Species)
}, numeric(1))

data.frame(model = available, test_accuracy = round(scores, 3),
           row.names = NULL) %>%
  arrange(desc(test_accuracy))
#>              model test_accuracy
#> 1    baseline_tree         0.933
#> 2  baseline_forest         0.933
#> 3   clustered_tree         0.933
#> 4     advanced_svm         0.933
#> 5 clustered_forest         0.911
#> 6 advanced_xgboost         0.911
#> 7         pca_tree         0.889
#> 8       pca_forest         0.867

Comparing the leaderboard against held-out accuracy is worth doing: the model that won the search is not always the one that generalises, and with a dataset this small the difference between the top few is noise.

AutoML Against a Single Choice

manual <- tl_model(split$train, Species ~ ., method = "forest")
manual_acc <- mean(
  predict(manual, new_data = split$test)$.pred == split$test$Species
)

automl_acc <- mean(test_preds$.pred == split$test$Species)

data.frame(
  approach = c("forest, chosen by hand", "AutoML best"),
  test_accuracy = round(c(manual_acc, automl_acc), 3)
)
#>                 approach test_accuracy
#> 1 forest, chosen by hand         0.933
#> 2            AutoML best         0.933

On iris a random forest is already the right answer, so the search does not improve on it. That is the expected result on an easy, well-understood dataset — AutoML earns its cost when you do not already know which method suits the data.

Preprocessing First

AutoML does not impute. Handle missing values before the call, and apply the same transformation to any data you later predict on:

processed <- tl_prepare_data(
  split$train, Species ~ .,
  scale_method = "standardize",
  remove_correlated = TRUE
)

automl_processed <- tl_auto_ml(processed$data, Species ~ .,
                               time_budget = 30, cv_folds = 3)

automl_processed$leaderboard$model[1]
#> [1] "clustered_forest"

tl_pipeline() handles that bookkeeping properly — see vignette("tuning-and-pipelines").

When a Model Is Missing or Scores NA

A model that fails to fit, or errors during evaluation, is dropped with a message explaining why and never reaches the leaderboard. Run without message = FALSE to see which and why.

A model that evaluates but produces no value for the chosen metric appears with an NA score. That usually means the metric is not one tl_evaluate() recognises for the task. If every score is NA, tl_auto_ml() warns and returns the first model trained rather than pretending to have ranked them.

sum(is.na(result$leaderboard$score))
#> [1] 0

AutoML or Tuning?

They search different axes and compose in one direction:

tl_auto_ml() searches across methods at default hyperparameters.
tl_tune_grid() and tl_tune_random() search within one method.

Run AutoML first to find out which family suits the data, then tune the winner:

winner <- automl$best_model$spec$method
winner
#> [1] "svm"

tuned <- tl_tune_grid(
  split$train, Species ~ .,
  method = winner,
  param_grid = tl_default_param_grid(winner, size = "small"),
  folds = 3,
  verbose = FALSE
)

attr(tuned, "tuning_results")$best_params
#> $kernel
#> [1] "linear"
#> 
#> $cost
#> [1] 1

mean(predict(tuned, new_data = split$test)$.pred == split$test$Species)
#> [1] 0.9777778

Guidance

Start at time_budget = 10 to confirm the call is well formed, then raise it. Nothing about a 10-second run tells you which model is best.
Lower cv_folds before lowering time_budget. Going from 5 folds to 2 removes 60% of the evaluation cost and still gives out-of-sample estimates. A 30-second budget at cv_folds = 2 is more informative than the same budget at the default 5, which will skip CV entirely.
Preprocess first. AutoML does not impute or scale.
Score on held-out data. The leaderboard ranks; it does not report final performance.
Read past the first row. When the top few scores are within noise of each other, pick on interpretability or fit time instead.
Check the evaluation column before comparing scores.