Return a tibble of performance metrics for all models or submodels.

# S3 method for workflow_set
collect_metrics(x, summarize = TRUE, ...)

# S3 method for workflow_set
collect_predictions(
  x,
  summarize = TRUE,
  parameters = NULL,
  select_best = FALSE,
  metric = NULL,
  ...
)

Arguments

x

A workflow_set object where all workflows have been evaluated.

summarize

A logical for whether the performance estimates should be summarized via the mean (over resamples) or the raw performance values (per resample) should be returned along with the resampling identifiers. When collecting predictions, these are averaged if multiple assessment sets contain the same row.

...

Not currently used.

parameters

An optional tibble of tuning parameter values that can be used to filter the predicted values before processing. This tibble should only have columns for each tuning parameter identifier (e.g. "my_param" if tune("my_param") was used).

select_best

A single logical for whether the numerically best results are retained. If TRUE, the parameters argument is ignored.

metric

A character string for the metric that is used for select_best.

Value

A tibble.

Details

When applied to a workflow set, the metrics and predictions that are returned do not contain the actual tuning parameter columns and values (unlike when these collect functions are run on other objects). The reason is that workflow sets can contain different types of models or models with different tuning parameters.

If the columns are needed, there are two options. First, the .config column can be used to merge the tuning parameter columns into an appropriate object. Alternatively, the map() function can be used to get the metrics from the original objects (see the example below).

See also

Examples

#> #> Attaching package: ‘dplyr’
#> The following objects are masked from ‘package:stats’: #> #> filter, lag
#> The following objects are masked from ‘package:base’: #> #> intersect, setdiff, setequal, union
library(purrr) library(tidyr) # ------------------------------------------------------------------------------ # \donttest{ collect_metrics(two_class_res)
#> # A tibble: 52 × 9 #> wflow_id .config preproc model .metric .estimator mean n std_err #> <chr> <chr> <chr> <chr> <chr> <chr> <dbl> <int> <dbl> #> 1 none_cart Preprocesso… formula decisi… accura… binary 0.777 5 0.00517 #> 2 none_cart Preprocesso… formula decisi… roc_auc binary 0.839 5 0.0160 #> 3 none_cart Preprocesso… formula decisi… accura… binary 0.784 5 0.00812 #> 4 none_cart Preprocesso… formula decisi… roc_auc binary 0.841 5 0.0171 #> 5 none_cart Preprocesso… formula decisi… accura… binary 0.791 5 0.0111 #> 6 none_cart Preprocesso… formula decisi… roc_auc binary 0.853 5 0.00813 #> 7 none_cart Preprocesso… formula decisi… accura… binary 0.814 5 0.0104 #> 8 none_cart Preprocesso… formula decisi… roc_auc binary 0.829 5 0.0145 #> 9 none_cart Preprocesso… formula decisi… accura… binary 0.788 5 0.0133 #> 10 none_cart Preprocesso… formula decisi… roc_auc binary 0.852 5 0.0101 #> # … with 42 more rows
# Alternatively, if the tuning parameter values are needed: two_class_res %>% dplyr::filter(grepl("cart", wflow_id)) %>% mutate(metrics = map(result, collect_metrics)) %>% dplyr::select(wflow_id, metrics) %>% tidyr::unnest(cols = metrics)
#> # A tibble: 40 × 9 #> wflow_id cost_complexity min_n .metric .estimator mean n std_err .config #> <chr> <dbl> <int> <chr> <chr> <dbl> <int> <dbl> <chr> #> 1 none_cart 2.28e- 7 13 accuracy binary 0.777 5 0.00517 Prepro… #> 2 none_cart 2.28e- 7 13 roc_auc binary 0.839 5 0.0160 Prepro… #> 3 none_cart 1.32e- 5 16 accuracy binary 0.784 5 0.00812 Prepro… #> 4 none_cart 1.32e- 5 16 roc_auc binary 0.841 5 0.0171 Prepro… #> 5 none_cart 8.19e-10 26 accuracy binary 0.791 5 0.0111 Prepro… #> 6 none_cart 8.19e-10 26 roc_auc binary 0.853 5 0.00813 Prepro… #> 7 none_cart 9.52e- 3 9 accuracy binary 0.814 5 0.0104 Prepro… #> 8 none_cart 9.52e- 3 9 roc_auc binary 0.829 5 0.0145 Prepro… #> 9 none_cart 4.76e- 7 20 accuracy binary 0.788 5 0.0133 Prepro… #> 10 none_cart 4.76e- 7 20 roc_auc binary 0.852 5 0.0101 Prepro… #> # … with 30 more rows
# } collect_metrics(two_class_res, summarize = FALSE)
#> # A tibble: 260 × 8 #> wflow_id .config preproc model id .metric .estimator .estimate #> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <dbl> #> 1 none_cart Preprocessor1… formula decision… Fold1 accura… binary 0.786 #> 2 none_cart Preprocessor1… formula decision… Fold1 roc_auc binary 0.875 #> 3 none_cart Preprocessor1… formula decision… Fold2 accura… binary 0.766 #> 4 none_cart Preprocessor1… formula decision… Fold2 roc_auc binary 0.851 #> 5 none_cart Preprocessor1… formula decision… Fold3 accura… binary 0.778 #> 6 none_cart Preprocessor1… formula decision… Fold3 roc_auc binary 0.848 #> 7 none_cart Preprocessor1… formula decision… Fold4 accura… binary 0.766 #> 8 none_cart Preprocessor1… formula decision… Fold4 roc_auc binary 0.779 #> 9 none_cart Preprocessor1… formula decision… Fold5 accura… binary 0.791 #> 10 none_cart Preprocessor1… formula decision… Fold5 roc_auc binary 0.841 #> # … with 250 more rows