vector of observed class labels (0/1)

vector of uncalibrated predictions

0=selection, 1=averaging, Default: 0

number of folds in the cross-validation, Default: 10

random seed to alternate the split of data set partitions

specify if the input was scaled or transformed, scaled=1, transformed=2

list object that summarizes discrimination and calibration errors obtained during the CV

which BBQ prediction method was used during CV, 0=selection, 1=averaging

"BBQ"

vector of calibrated predictions that was used during the CV

respective vector of true values (0 or 1) that was used during the CV

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of folds for the cross-validation, Default: 10

PARAM_DESCRIPTION, Default: 2

random seed to alternate the split of data set partitions

specify if the input was scaled or transformed, scaled=1, transformed=2

list object that summarizes discrimination and calibration errors obtained during the CV

"GUESS"

which prediction method was used during CV

vector of calibrated predictions that was used during the CV

respective vector of true values (0 or 1) that was used during the CV

numeric vector of two integers. The first one is the number of cases in the bin; the second the number of instances in the bin

vector of observed class labels (0/1)

vector of uncalibrated predictions

vector of observed class labels (0/1)

vector of uncalibrated predictions

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins that should be used to build the binning model, Default: decide_on_break estimates optimal number of bins

vector of observed class labels (0/1)

vector of uncalibrated predictions

which calibration models should be implemented, 1=hist_scaled, 2=hist_transformed, 3=BBQ_scaled, 4=BBQ_transformed, 5=GUESS, Default: c(1, 2, 3, 4, 5)

computes internal errors for calibration models that were trained on all available actual/predicted tuples. Testing is performed with the same set. Be careful to interpret those error values, as they are not cross-validated. Default: TRUE

computes cross-validation error. folds times cross validation is repeated n_seeds times with changing seeds. The trained models and the their calibration and discrimination errors are returned. Evaluation of CV errors can take some time to compute, depending on the number of repetitions specified in n_seeds, Default: TRUE

number of folds in the cross-validation of the calibration model. If folds is set to 1, no CV is performed and summary_CV can be calculated. Default: 10

n_seeds determines how often random data set partition is repeated with varying seed. If folds is 1, n_seeds should be set to 1, too. Default: 30

nCores how many cores should be used during parallelisation. Default: 4

a list of all trained calibration models, which can be used in the predict_calibratR method.

a list containing information on the CV errors of the implemented models

a list containing information on the internal errors of the implemented models

calibrated predictions for the original predicted values

number of random data set partitions into training and test set for folds-times CV

vector of observed class labels (0/1)

vector of uncalibrated predictions

a single number from 1 to 5, indicating which calibration model should be implemented, 1=hist_scaled, 2=hist_transformed, 3=BBQ_scaled, 4=BBQ_transformed, 5=GUESS

vector of observed class labels (0/1)

vector of uncalibrated predictions

which calibration models should be implemented, 1=hist_scaled, 2=hist_transformed, 3=BBQ_scaled, 4=BBQ_transformed, 5=GUESS

number of folds in the cross-validation, Default: 10

n_seeds determines how often random data set partition is repeated with varying seed

nCores how many cores should be used during parallelisation. Default: 4

PARAM_DESCRIPTION

sequence for which the calibrated predictions should be plotted, Default: NULL

vector of observed class labels (0/1)

vector of uncalibrated predictions

cut-off to be used for the computation of npv, ppv, sensitivity and specificity, Default: value that maximizes sensitivity and specificity (Youden-Index)

sensitivity

specificity

accuracy

positive predictive value

negative predictive value

cut-off that was used to compute the error values

AUC value

instances from class 1

instances from class 0

single integer (0, 1 or 2). specify if the input should be formatted (=0), formatted and scaled (=1) or formatted and transformed (=2)

min value of the original data set, default=calculated on input

max value of the original data set, default=calculated on input

mean value of the original data set, default=calculated on input

formatted input. If input is set to 1 (2), the input is additionally scaled (transformed) using the method scale_me (transform_me)

minimum value among all instances

maximum value among all instances

mean value among all instances

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins of the underlying equal-frequency histogram, Default: 10

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins of the underlying equal-frequency histogram, Default: 10

vector of observed class labels (0/1)

vector of uncalibrated predictions

vector of observed class labels (0/1)

vector of uncalibrated predictions

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins for the equal-width binning model, default=10

CLE of class 1 instances

CLE of class 0 instances

list object that contains all error values for all trained calibration models. For the specific format, see the calling function visualize_calibratR.

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins for the equal-width binning model

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins for the binning model

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins used in the histogram binning scheme, Default: 15

number of folds in the cross-validation, Default: 10

random seed to alternate the split of data set partitions

specify if the input was scaled or transformed, scaled=1, transformed=2

list object that summarizes discrimination and calibration errors obtained during the CV

"hist"

vector of calibrated predictions that was used during the CV

respective vector of true values (0 or 1) that was used during the CV

output from build_GUESS()

if pred_idx=1 GUESS_1 is plotted; if pred_idx=2 GUESS_2 is plotted

output from the calibrate method.

sequence of ML scores over which the mapping function should be evaluated, Default: 100 scores from the minimum to the maximum of the original ML scores

output from the build_BBQ method

vector of uncalibrated probabilities

either 1 or 0; averaging=1, selecting=0

contains a vector of calibrated predictions

which option was used (averaging or selecting)

the percentage of new instances that was evaluated using significant prediction estimates

number of instances new in each bin of the selected model

output from the build_GUESS method

vector of uncalibrated probabilities

which density evaluation method should be used to infer calculate probabilities, Default: 2

if set to TRUE, class densities p(x|class) are returned, Default: FALSE

contains a vector of calibrated predictions

which density evaluation method was used

the percentage of new instances that was evaluated using significant prediction estimates

a vector containing the p(x|case) values

a vector containing the p(x|control) values

list of trained calibration models that were constructed using the calibrate method. The list components calibration_models from the calibrate output can be used directly.

vector of new uncalibrated instances. Default: 100 scores from the minimum to the maximum of the original ML scores

nCores how many cores should be used during parallelisation. Default: 4

a list containing the calibrated predictions for each calibration model

a list containing the percentage of new instances for which prediction estimates are statistically significant

a list containing the number of instances in each bin for the binning models

the output of build_hist_binning

vector of uncalibrated probabilities

contains a vector of calibrated predictions

the percentage of new instances that was evaluated using significant prediction estimates

a table containing the number of instances from new for each bin of the final binning scheme of histogram

vector of uncalibrated predictions

calibration model to be used for the calibration. Can be the output of build_BBQ,build_hist_binning or build_GUESS.

minimum value of the original data set

maximum value of the original data set

mean value of the original data set

specify if the model was build on original (=0), scaled(=1) or transformed (=2) data

list object that contains n-times the output from the reliability_diagramm. method.

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of bins in the reliability diagram, Default: 10

should the reliability diagram be plotted, Default: TRUE

vector of predictions

minimum of x, Default: NULL

maximum of x, Default: NULL

list that is returned from the calibrate function. The parameter n_seeds is available as a list component of the calibrate_object

Performs a paired two sided t.test over the error values (ECE, CLE1, CLE0, MCE, AUC, sensitivity and specificity) from the random partition splits comparing a possible significant difference in mean among the calibration models. All models and the original, scaled and transformed values are tested against each other. The p_value and the effect size of the t.test are returned to the user. Can only be performed, if the calibrate_object contains a summary_CV list object, else, an error is returned. Default: TRUE

returns important characteristics of the implemented calibration models, Default: TRUE

mean of calibration error values (ECE_equal_width, MCE_equal_width, ECE_equal_freq, MCE_equal_freq, RMSE, Class 1 CLE, Class 0 CLE, Brier Score, Class 1 Brier Score, Class 0 Brier Score) over n_seeds times repeated 10-fold CV. ECE and MCE are computed once using equal-width and once using equal-frequency binning for the construction of the underlying binning scheme. Only returned, if calibrate_object contains a summary_CV list object.

standard deviation of calibration error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

variation coefficient of calibration error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

mean of discrimination error (sensitivity, specificity, AUC, positive predictive value, negative predictive value, accuracy) values over n_seeds times repeated 10-fold CV. The "cut-off" is the cut-off value that maximizes sensitivity and specificity. Only returned, if calibrate_object contains a summary_CV list object.

standard deviation of discrimination error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

variation coefficient of discrimination error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

=list(p_value=t.test.calibration, effect_size=effect_size_calibration), only returned if t.test=TRUE

=list(p_value=t.test.discrimination, effect_size=effect_size_discrimination), only returned if t.test=TRUE

only returned if significance_models=TRUE

number of random data set partitions into training and test set for folds-times CV

list object that consists of the actual and predicted values of the original scores

vector of predictions

mean of x

vector of observed class labels (0/1)

vector of uncalibrated predictions

number of folds for the cross-validation, Default: 10

random seed to alternate the split of data set partitions

specify if the input was scaled or transformed, scaled=1, transformed=2

list object that summarizes discrimination and calibration errors obtained during the CV

"uncalibrated"

vector of input-preprocessed predictions that was used during the CV

respective vector of true values (0 or 1) that was used during the CV

the list component calibration_models from the calibrate method

returns the list components plot_calibration_models and plot_single_models

returns a density distribution plot of the calibrated predictions after CV (External) or without CV (internal)

returns a reliability diagram for each model

returns a list of ggplots. Each plot represents the calibrated predictions by the respective calibration model of the training set. If the list object predictions in the calibrate_object is empty, training_set_calibrated is returned as NULL.

returns a histogram of the original ML score distribution

returns a list of density distribution plots for each calibration method, the original and the two input-preprocessing methods scaling and transforming. The plot visualises the density distribution of the calibrated predictions of the training set. In this case, training and test set values are identical, so be careful to evaluate the plots.

returns a list of density distribution plots for each calibration method, the original and the two input-preprocessing methods scaling and transforming. The plot visualises the density distribution of the calibrated predictions, that were returned during Cross Validation. If more than one repetition of CV was performed, run number 1 is evaluated

maps the original ML scores to their calibrated prediction estimates for each model. This enables easy model comparison over the range of ML scores See also compare_models_visual.

returns a list of ggplots for each calibration model, also mapping the original ML scores to their calibrated prediction. Significance values are indicated. See also plot_model

returns a list of reliability diagrams for each of the implemented calibration models and the two input-preprocessing methods "scaled" and "transformed". The returned plot visualises the calibrated predictions that were returned for the test set during each of the n run of the n-times repeated CV. Each grey line represents one of the n runs. The blue line represents the median of all calibrated bin predictions. Insignificant bin estimates are indicated with "ns". If no CV was performed during calibration model building using the calibrate method, rd_plot is returned as NULL

returns a list of boxplots for the calibration error metrics ECE, MCE, CLE and RMSE. The n values for each model represent the obtained error values during the n times repeated CV. If no CV was performed during calibration model building using the calibrate method, calibration_error is returned as NULL

returns a list of boxplots for the discrimination error AUC, sensitivity and specificity. The n values for each model represent the obtained error values during the n times repeated CV. If no CV was performed during calibration model building using the calibrate method, discrimination_error is returned as NULL

If no CV was performed during calibration model building using the calibrate method, cle_class_specific_error is returned as NULL

returns a list of ggplots. Each plot represents the calibrated predictions by the respective calibration model of the training set. If the list object predictions in the calibrate_object is empty, training_set_calibrated is returned as NULL.

plots the the returned conditional probability p(x|Class) values of the GUESS_1 model

plots the the returned conditional probability p(x|Class) values of the GUESS_2 model

vector of observed class labels (0/1)

predict_calibratR$predictions object (list of calibrated predictions from calibration models)

vector of optimal cut-off thresholds for each calibration model

vector of observed class labels (0/1)

vector of uncalibrated predictions

ggplot histogram that visualizes the observed class distributions

list object that summarizes all relevant parameters (mean, sd, number) of the observed class distributions

list object that contains all error values for all trained calibration models. For the specific format, see the calling function visualize_calibratR.

boolean (TRUE or FALSE). If TRUE, discrimination errors are compared between models; if FALSE calibration error is compared, Default: TRUE

ggplot2 boxplot that compares all evaluated calibration models with regard to sensitivity.

ggplot2 boxplot that compares all evaluated calibration models with regard to specificity

ggplot2 boxplot that compares all evaluated calibration models with regard to accuracy

ggplot2 boxplot that compares all evaluated calibration models with regard to AUC

list object that contains all discrimination error values that were used to construct the boxplots

ggplot2 boxplot that compares all evaluated calibration models with regard to expected calibration error

ggplot2 boxplot that compares all evaluated calibration models with regard to maximum expected calibration error (MCE)

ggplot2 boxplot that compares all evaluated calibration models with regard to root mean square error (RMSE)

ggplot2 boxplot that compares all evaluated calibration models with regard to class 0 classification error (CLE)

ggplot2 boxplot that compares all evaluated calibration models with regard to class 1 classification error (CLE)

list object that contains all calibration error values that were used to construct the boxplots