Package {modelimpact}

modelimpact: assess the business impact of churn prediction models

Description

Calculate the financial impact of using a churn model in terms of cost, revenue, profit, ROI, gains, lift and threshold-based payoff, and visualise the results.

Author(s)

Maintainer: Peer Christensen hr.pchristensen@gmail.com

Authors:

• Peer Christensen hr.pchristensen@gmail.com

See Also

Useful links:

• https://github.com/PeerChristensen/modelimpact

• Report bugs at https://github.com/PeerChristensen/modelimpact/issues

Bootstrap confidence bands for the profit curve

Description

Resamples the observations with replacement a number of times, recomputes the cumulative profit curve for each resample, and returns pointwise quantile bands. This turns the single profit curve from [profit()] into a range, making it clear how much of the curve's shape is signal versus sampling noise.

Usage

bootstrap_profit(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes",
  n_boot = 200,
  probs = c(0.05, 0.5, 0.95)
)

Arguments

Value

A data frame with the following columns:

row = row numbers
prop_pop = proportion of the population targeted (row / n)
lower = lower quantile of profit across resamples
median = median profit across resamples
upper = upper quantile of profit across resamples

Examples

bootstrap_profit(predictions,
   fixed_cost = 1000,
   var_cost   = 100,
   tp_val     = 2000,
   prob_col   = Yes,
   truth_col  = Churn,
   n_boot     = 100)

Break-even and optimal targeting point

Description

Summarises the two key operating points of the cumulative profit curve produced by [profit()]: the point of maximum profit (the recommended share of customers to target) and the break-even point (the largest share that can be targeted while still turning a profit).

Usage

break_even(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_accept = 1,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A one-row data frame with the following columns:

optimal_row = row at which profit is maximised
optimal_pct = share of the population targeted at maximum profit
max_profit = the maximum profit
breakeven_row = last row at which cumulative profit is still non-negative
breakeven_pct = share of the population targeted at the break-even point

Examples

break_even(predictions,
   fixed_cost = 1000,
   var_cost   = 100,
   tp_val     = 2000,
   prob_col   = Yes,
   truth_col  = Churn)

Compare several models on a business-impact metric

Description

Computes a targeting curve for two or more models so they can be compared and overlaid on a single plot. Rather than picking a model on AUC alone, this lets you choose the one that produces the most profit (or the steepest gains, lift or ROI) at the share of customers you intend to target.

Usage

compare_models(
  x,
  prob_cols,
  truth_col = NA,
  metric = c("profit", "gains", "lift", "roi"),
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  positive = "Yes"
)

Arguments

Value

A data frame with the following columns:

model = the model (column) name
row = row numbers
prop_pop = proportion of the population targeted (row / n)
value = the chosen metric at that point

Examples

compare_models(predictions,
   prob_cols = c("Yes", "No"),
   truth_col = Churn,
   metric    = "gains")

Confusion matrix and payoff at a single threshold

Description

Classifies observations at a single probability threshold, returns the resulting confusion matrix (TP/FP/TN/FN) and the associated payoff. This is the single-threshold companion to [profit_thresholds()], which sweeps every threshold, and uses the same value model.

Usage

confusion_payoff(
  x,
  threshold = 0.5,
  var_cost = 0,
  prob_accept = 1,
  tp_val = 0,
  fp_val = 0,
  tn_val = 0,
  fn_val = 0,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A one-row data frame with the following columns:

threshold = the threshold used
tp = number of true positives
fp = number of false positives
tn = number of true negatives
fn = number of false negatives
payoff = total payoff at the threshold

Examples

confusion_payoff(predictions,
   threshold   = 0.3,
   var_cost    = 100,
   prob_accept = .8,
   tp_val      = 2000,
   fp_val      = 0,
   tn_val      = 0,
   fn_val      = -2000,
   prob_col    = Yes,
   truth_col   = Churn)

Calculate cost and revenue

Description

Calculates cost and revenue after sorting observations.

Usage

cost_revenue(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_accept = 1,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A data frame with the following columns:

row = row numbers
pct = percentiles
cost_sum = cumulated costs
cum_rev = cumulated revenue

Examples

cost_revenue(predictions,
   fixed_cost = 1000,
   var_cost   = 100,
   tp_val     = 2000,
   prob_col   = Yes,
   truth_col  = Churn)

Cumulative gains

Description

Calculates a cumulative gains curve after sorting observations by descending predicted probability. The gain at a given point is the proportion of all actual positives (events) that have been captured by targeting the top rows.

Usage

cumulative_gains(x, prob_col = NA, truth_col = NA, positive = "Yes")

Arguments

Value

A data frame with the following columns:

row = row numbers
pct = percentiles
prop_pop = proportion of the population targeted (row / n)
cum_events = cumulated number of actual positives captured
gain = proportion of all positives captured (cum_events / total positives)
baseline = expected gain from random targeting (equal to prop_pop)

Examples

cumulative_gains(predictions,
   prob_col  = Yes,
   truth_col = Churn)

One-line business-impact summary

Description

Rolls the ranking-based views (profit, ROI, gains and break-even) up into a single row of headline numbers for reporting. It answers: what share of customers should we target, how much profit does that make, what return does it represent, how many churners do we catch, how far can we go before losing money, and what happens if we simply target everyone.

Usage

impact_summary(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_accept = 1,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A one-row data frame with the following columns:

optimal_pct = share of the population targeted at maximum profit
max_profit = the maximum profit
roi_at_optimum = ROI at the maximum-profit point
capture_at_optimum = proportion of all positives captured at that point
breakeven_pct = largest share that can be targeted while staying profitable
profit_target_all = profit if every customer is targeted

Examples

impact_summary(predictions,
   fixed_cost = 1000,
   var_cost   = 100,
   tp_val     = 2000,
   prob_col   = Yes,
   truth_col  = Churn)

Lift

Description

Calculates a cumulative lift curve after sorting observations by descending predicted probability. Lift is the ratio between the proportion of positives captured by the model and the proportion that would be expected from random targeting. A lift of 1 is no better than random. Named 'lift_curve()' to avoid clashing with 'purrr::lift()'.

Usage

lift_curve(x, prob_col = NA, truth_col = NA, positive = "Yes")

Arguments

Value

A data frame with the following columns:

row = row numbers
pct = percentiles
prop_pop = proportion of the population targeted (row / n)
gain = proportion of all positives captured
lift = gain / prop_pop

Examples

lift_curve(predictions,
   prob_col  = Yes,
   truth_col = Churn)

Marginal profit per bin

Description

Splits observations into equally sized bins (deciles by default) after sorting by descending predicted probability, and calculates the profit contributed by each bin. This makes it easy to see where targeting additional customers stops paying off: the first bin whose 'marginal_profit' turns negative marks the point of diminishing returns. The cumulative sum of 'marginal_profit' reconciles with the cumulative profit reported by [profit()].

Usage

marginal_profit(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_accept = 1,
  bins = 10,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A data frame with the following columns:

bin = bin number (1 = highest probabilities)
n = number of observations in the bin
events = number of actual positives in the bin
cost = cost incurred in the bin
revenue = revenue generated in the bin
marginal_profit = revenue - cost for the bin
cum_profit = cumulative profit up to and including the bin

Examples

marginal_profit(predictions,
   fixed_cost = 1000,
   var_cost   = 100,
   tp_val     = 2000,
   bins       = 10,
   prob_col   = Yes,
   truth_col  = Churn)

Plot modelimpact results

Description

Every analysis function in the package returns a classed data frame that can be visualised directly with autoplot(), for example autoplot(profit(...)). The 'plot_*()' functions are thin, back-compatible wrappers around the corresponding 'autoplot()' method. ggplot2 is only required when a plot is actually drawn.

Usage

autoplot.mi_profit(object, ...)

autoplot.mi_cost_revenue(object, ...)

autoplot.mi_roi(object, ...)

autoplot.mi_gains(object, ...)

autoplot.mi_lift(object, ...)

autoplot.mi_marginal(object, ...)

autoplot.mi_thresholds(object, ...)

autoplot.mi_roc(object, slope = NULL, ...)

autoplot.mi_compare(object, ...)

autoplot.mi_bootstrap(object, ...)

plot_profit(data)

plot_cost_revenue(data)

plot_roi(data)

plot_gains(data)

plot_lift(data)

plot_marginal(data)

plot_thresholds(data)

plot_roc(data, slope = NULL)

Arguments

Value

A 'ggplot' object.

Payoff sensitivity grid

Description

Computes payoff across a grid of classification thresholds and offer-acceptance probabilities, so the robustness of the optimal operating point can be judged at a glance (for example as a heatmap). Uses the same value model as [profit_thresholds()] and [confusion_payoff()].

Usage

payoff_grid(
  x,
  thresholds = seq(0, 1, 0.02),
  prob_accept = seq(0, 1, 0.1),
  var_cost = 0,
  tp_val = 0,
  fp_val = 0,
  tn_val = 0,
  fn_val = 0,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A data frame with the following columns:

threshold = classification threshold
prob_accept = offer-acceptance probability
payoff = payoff for that combination

Examples

payoff_grid(predictions,
   var_cost  = 100,
   tp_val    = 2000,
   fn_val    = -2000,
   prob_col  = Yes,
   truth_col = Churn)

Predictions from a customer churn model.

Description

A dataset containing 2145 observations with four columns specifying predicted probabilities and predicted and actual class.

Usage

predictions

Format

A data frame with 2145 rows and 4 variables:

predict

Predicted class

No

Predicted probability of class 'No'

Yes

Predicted probability of class 'Yes'

Churn

Actual class

...

Calculate profit

Description

Calculates profit after sorting observations.

Usage

profit(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_accept = 1,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A data frame with the following columns:

row = row numbers
pct = percentiles
profit = profit for number of rows selected

Examples

profit(predictions,
     fixed_cost = 1000,
     var_cost   = 100,
     tp_val     = 2000,
     prob_col   = Yes,
     truth_col  = Churn)

Find optimal threshold for churn prediction (class)

Description

Finds the optimal threshold (from a business perspective) for classifying churners.

Usage

profit_thresholds(
  x,
  var_cost = 0,
  prob_accept = 1,
  tp_val = 0,
  fp_val = 0,
  tn_val = 0,
  fn_val = 0,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A data frame with the following columns:

threshold = prediction thresholds
payoff = calculated profit for each threshold

Examples

profit_thresholds(predictions,
   var_cost    = 100,
   prob_accept = .8,
   tp_val      = 2000,
   fp_val      = 0,
   tn_val      = 0,
   fn_val      = -2000,
   prob_col    = Yes,
   truth_col   = Churn)

ROC and precision-recall curve data

Description

Calculates the points of the ROC curve and the precision-recall curve by walking down the observations sorted by descending predicted probability. The returned data frame contains everything needed to draw both curves and to overlay an iso-profit / cost-sensitive operating point.

Usage

roc_pr(x, prob_col = NA, truth_col = NA, positive = "Yes")

Arguments

Value

A data frame with the following columns:

threshold = probability at each operating point
tp = true positives when classifying every row at or above the point as positive
fp = false positives
tpr = true positive rate / recall / sensitivity (tp / all positives)
fpr = false positive rate (fp / all negatives)
precision = tp / (tp + fp)
recall = same as tpr

Examples

roc_pr(predictions,
   prob_col  = Yes,
   truth_col = Churn)

Calculate Return on investment (ROI)

Description

Calculates ROI after sorting observations with ROI defined as (Current Value - Start Value) / Start Value

Usage

roi(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_accept = 1,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes"
)

Arguments

Value

A data frame with the following columns:

row = row numbers
pct = percentiles
cum_rev = cumulated revenue
cost_sum = cumulated costs
roi = return on investment

Examples

roi(predictions,
   fixed_cost = 1000,
   var_cost   = 100,
   tp_val     = 2000,
   prob_col   = Yes,
   truth_col  = Churn)

Launch the interactive modelimpact Shiny app

Description

Opens a small Shiny application for exploring the package interactively. You can use the bundled 'predictions' data or upload your own CSV (choosing comma or semicolon as the separator), pick the probability and outcome columns and the positive class, adjust the cost/value parameters, and watch every plot update live.

Usage

run_app(...)

Arguments

Details

Requires the shiny and ggplot2 packages, which are only needed for this function.

Value

Called for its side effect of starting the app; returns nothing.

Examples

## Not run: 
run_app()

## End(Not run)

Tornado sensitivity analysis of maximum profit

Description

Varies each cost/value assumption up and down by a fixed fraction, one at a time, and records how the maximum achievable profit (the peak of the [profit()] curve) responds. The result feeds a tornado plot, ordering the assumptions by how much they move the bottom line.

Usage

tornado(
  x,
  fixed_cost = 0,
  var_cost = 0,
  tp_val = 0,
  prob_col = NA,
  truth_col = NA,
  positive = "Yes",
  variation = 0.2
)

Arguments

Value

A data frame, one row per parameter, ordered by descending 'swing':

parameter = the assumption varied
low_value = parameter value on the low side
high_value = parameter value on the high side
low_profit = maximum profit at the low value
high_profit = maximum profit at the high value
base_profit = maximum profit at the baseline values
swing = absolute difference between low_profit and high_profit

Examples

tornado(predictions,
   fixed_cost = 1000,
   var_cost   = 100,
   tp_val     = 2000,
   prob_col   = Yes,
   truth_col  = Churn)