Skip to contents

Diagnose a POPMAPS tuning result

Source: R/tune_popmaps.R
diagnose_tuning.Rd

diagnose_tuning() summarizes whether parameter tuning found a clearly supported parameter combination or a broad set of similarly performing alternatives. It is intended to help users interpret tuning output biologically instead of choosing a row from tuning$results by eye.

Usage

diagnose_tuning(
  tuning,
  primary_metric = tuning$primary_metric,
  near_best_tolerance = 0.05,
  complete_only = TRUE
)

Arguments

tuning

A popmaps_tuning or popmaps_adaptive_tuning object returned by tune_popmaps() or adaptive_tune_popmaps().

primary_metric

Metric used to rank parameter combinations. Defaults to the metric stored in tuning.

near_best_tolerance

Non-negative relative tolerance used to define near-best parameter combinations. The default, 0.05, keeps combinations within 5% of the best score for the primary metric.

complete_only

Logical. If TRUE, diagnose only parameter combinations with no failed validation folds.

Value

A popmaps_tuning_diagnostics list with:

overview

One-row summary of tuning strength and near-best support.

near_best

Parameter combinations within near_best_tolerance of the best score.

parameter_ranges

Near-best and full-grid support for each tuning parameter.

parameter_effects

Average score by parameter value.

Examples

ex_raster <- raster::aggregate(hija_raster, fact = 240)
tuning <- tune_popmaps(
  input_raster = ex_raster,
  input_locs = hija_struc,
  empirical_pt_dist = c(0, 5),
  num_sites = c(5, 6),
  num_tested = c(2, 3),
  popmod = c(-0.01, -0.05),
  quiet = TRUE
)
diagnose_tuning(tuning)
#> POPMAPS tuning diagnostics
#> Primary metric: rmse (minimize)
#> Validation: loo
#> Near-best combinations: 8
#> 
#> Overview:
#>  validation primary_metric metric_goal n_combinations n_evaluated n_complete
#>         loo           rmse    minimize             16          16         16
#>  best_score median_score worst_score best_vs_median_delta
#>   0.1193331    0.1231894   0.1414239          0.003856351
#>  best_vs_median_percent best_vs_worst_delta best_vs_worst_percent
#>                3.130424          0.02209083               15.6203
#>  near_best_tolerance n_near_best
#>                 0.05           8
#> 
#> Near-best parameter support:
#>          parameter best_value near_best_min near_best_max near_best_unique
#>          num_sites    5.00000       5.00000       6.00000             5, 6
#>         num_tested    2.00000       2.00000       3.00000             2, 3
#>             popmod   -0.05000      -0.05000      -0.05000            -0.05
#>      half_distance   13.86294      13.86294      13.86294           13.863
#>   ten_pct_distance   46.05170      46.05170      46.05170           46.052
#>  empirical_pt_dist    0.00000       0.00000       5.00000             0, 5
#>  full_min  full_max
#>   5.00000   6.00000
#>   2.00000   3.00000
#>  -0.05000  -0.01000
#>  13.86294  69.31472
#>  46.05170 230.25851
#>   0.00000   5.00000