Feature Samplers

Feature samplers are a core component of perturbation-based feature importance methods (PFI, CFI, RFI) and any other methods based on some form of marginalization (SAGE). They determine how features are replaced or perturbed when evaluating feature importance. This vignette introduces the different types of feature samplers available in xplainfi and demonstrates their use.

Setup

We create two tasks: one with mixed features (the penguins data), and one with all-numeric features.

library(xplainfi)
library(mlr3)
library(mlr3learners)
library(data.table)

# Create a task for demonstration
task_mixed = tsk("penguins")
task_numeric = sim_dgp_correlated(n = 200)

Base Class: FeatureSampler

All feature samplers inherit from the FeatureSampler base class, which provides a common interface for sampling features.

Key Properties

Feature Type Support: Each sampler declares which feature types it supports:

# Check supported feature types for different samplers
task_mixed$feature_types
#> Key: <id>
#>                id    type
#>            <char>  <char>
#> 1:     bill_depth numeric
#> 2:    bill_length numeric
#> 3:      body_mass integer
#> 4: flipper_length integer
#> 5:         island  factor
#> 6:            sex  factor
#> 7:           year integer
permutation = MarginalPermutationSampler$new(task_mixed)
permutation$feature_types
#> [1] "numeric"   "factor"    "ordered"   "integer"   "logical"   "Date"     
#> [7] "POSIXct"   "character"

Two Sampling Methods:

$sample(feature, row_ids) - Sample from the stored task
$sample_newdata(feature, newdata) - Sample using external data

Let’s demonstrate both with the permutation sampler:

# Sample from stored task (using row_ids)
sampled_task = permutation$sample(
    feature = "bill_length",
    row_ids = 40:45
)
sampled_task
#>    species bill_depth bill_length body_mass flipper_length island    sex  year
#>     <fctr>      <num>       <num>     <int>          <int> <fctr> <fctr> <int>
#> 1:  Adelie       19.1        37.0      4650            184  Dream   male  2007
#> 2:  Adelie       18.0        36.5      3150            182  Dream female  2007
#> 3:  Adelie       18.4        44.1      3900            195  Dream   male  2007
#> 4:  Adelie       18.5        36.0      3100            186  Dream female  2007
#> 5:  Adelie       19.7        39.8      4400            196  Dream   male  2007
#> 6:  Adelie       16.9        40.8      3000            185  Dream female  2007

# Sample from "external" data
test_data = task_mixed$data(rows = 40:45)
sampled_external = permutation$sample_newdata(
    feature = "bill_length",
    newdata = test_data
)
sampled_external
#>    species bill_depth bill_length body_mass flipper_length island    sex  year
#>     <fctr>      <num>       <num>     <int>          <int> <fctr> <fctr> <int>
#> 1:  Adelie       19.1        37.0      4650            184  Dream   male  2007
#> 2:  Adelie       18.0        36.0      3150            182  Dream female  2007
#> 3:  Adelie       18.4        44.1      3900            195  Dream   male  2007
#> 4:  Adelie       18.5        39.8      3100            186  Dream female  2007
#> 5:  Adelie       19.7        36.5      4400            196  Dream   male  2007
#> 6:  Adelie       16.9        40.8      3000            185  Dream female  2007

Notice that:

The sampled feature values change (they are permuted)
Other features and the target remain unchanged
The data structure is preserved

Permutation Sampler

The MarginalPermutationSampler performs simple random permutation of features, breaking their relationship with the target and other features. This is the classic approach used in Permutation Feature Importance (PFI).

How it works:

Each feature is randomly shuffled (permuted) independently
The association between feature values and target values is broken
The association between feature values across rows is broken
The marginal distribution of each feature is preserved

# Create permutation sampler
permutation = MarginalPermutationSampler$new(task_mixed)

# Sample a continuous feature
original = task_mixed$data(rows = 1:10)
sampled = permutation$sample("bill_length", row_ids = 1:10)

# Compare original and sampled values
data.table(
    original_bill = original$bill_length,
    sampled_bill = sampled$bill_length,
    sex = original$sex # Unchanged
)
#>     original_bill sampled_bill    sex
#>             <num>        <num> <fctr>
#>  1:          39.1         36.7   male
#>  2:          39.5         39.5 female
#>  3:          40.3         34.1 female
#>  4:            NA           NA   <NA>
#>  5:          36.7         39.1 female
#>  6:          39.3         38.9   male
#>  7:          38.9         40.3 female
#>  8:          39.2         42.0   male
#>  9:          34.1         39.3   <NA>
#> 10:          42.0         39.2   <NA>

Note that the permutation is only performed within the requested row_ids.

Use in PFI: The permutation sampler is used by default in Permutation Feature Importance

Marginal Reference Sampler

The MarginalReferenceSampler is another type of marginal sampler that samples complete rows from reference data. Unlike MarginalPermutationSampler which shuffles feature values independently, this sampler preserves within-row dependencies by sampling intact observations.

Key differences from MarginalPermutationSampler:

MarginalPermutationSampler: Shuffles each feature independently – breaks dependencies between sampled features
MarginalReferenceSampler: Samples complete reference rows – preserves dependencies of the sampled features

This is the approach used in SAGE (Shapley Additive Global importancE) for marginal feature importance. In this SAGE implementation, the MarginalImputer, uses this approach. The “imputer” name comes from the “imputation” of model predictions using out-of-coalition features by sampling from their marginal distribution. In xplainfi, we separate the “feature sampling” and “model prediction” steps (for now), which is why we keep the sampling infrastructure independent.

How it works:

# Create marginal reference sampler with n_samples reference pool
marginal_ref = MarginalReferenceSampler$new(task_mixed, n_samples = 30L)

# Sample a feature - each row gets values from a randomly sampled reference row
original = task_mixed$data(rows = 1:5)
sampled = marginal_ref$sample("bill_length", row_ids = 1:5)

# Compare
data.table(
    original_bill = original$bill_length,
    sampled_bill = sampled$bill_length,
    sex = original$sex # Unchanged
)
#>    original_bill sampled_bill    sex
#>            <num>        <num> <fctr>
#> 1:          39.1         44.4   male
#> 2:          39.5         36.6 female
#> 3:          40.3         34.6 female
#> 4:            NA         45.1   <NA>
#> 5:          36.7         47.3 female

Parameters:

n_samples: Controls the size of the reference data pool
- If NULL: uses all task data
- If specified: subsamples that many rows from task

Preserving within-row correlations:

To demonstrate the difference, consider features that are correlated as in task_numeric. Here, x1 and x2 are correlated, and if we sample them jointly, only MarginalReferenceSampler retains their original correlation (approximately).

# Sample with MarginalPermutationSampler (breaks correlations)
perm = MarginalPermutationSampler$new(task_numeric)
sampled_perm = perm$sample(c("x1", "x2"), row_ids = 1:10)

# Sample with MarginalReferenceSampler (preserves within-row correlations)
ref = MarginalReferenceSampler$new(task_numeric, n_samples = 50L)
sampled_ref = ref$sample(c("x1", "x2"), row_ids = 1:10)

# Check correlations
cor_original = cor(task_numeric$data()$x1, task_numeric$data()$x2)
cor_perm = cor(sampled_perm$x1, sampled_perm$x2)
cor_ref = cor(sampled_ref$x1, sampled_ref$x2)

data.table(
    method = c("Original", "Permutation", "Reference"),
    correlation = c(cor_original, cor_perm, cor_ref)
)
#>         method correlation
#>         <char>       <num>
#> 1:    Original   0.8937600
#> 2: Permutation  -0.1390767
#> 3:   Reference   0.9352523

The reference sampler better preserves the correlation structure because it samples complete rows, while permutation completely breaks the dependency.

Conditional Samplers

Conditional samplers account for dependencies between features by sampling from $P(X_j | X_{-j})$ rather than the marginal $P(X_j)$. This is relevant when features are correlated.

All conditional samplers inherit from ConditionalSampler and support:

Specifying which features to condition on via conditioning_set
Both $sample() and $sample_newdata() methods
Mixed feature types (depending on the specific sampler)

Gaussian Conditional Sampler

The ConditionalGaussianSampler assumes features follow a multivariate Gaussian distribution and uses closed-form conditional distributions.

Advantages:

Very fast (no model fitting during sampling)
Deterministic given a seed
No hyperparameters

Limitations:

Assumes multivariate normality
Only supports continuous features
May produce out-of-range values

# Create Gaussian conditional sampler
gaussian = ConditionalGaussianSampler$new(task_numeric)

# Sample x1 conditioned on other features
sampled = gaussian$sample(
    feature = "x1",
    row_ids = 1:10,
    conditioning_set = c("x2", "x3", "x4")
)

# Compare original and conditionally sampled values
original = task_numeric$data(rows = 1:10)
data.table(
    original = original$x1,
    sampled = sampled$x1,
    x2 = original$x2 # Conditioning feature (unchanged)
)
#>        original     sampled          x2
#>           <num>       <num>       <num>
#>  1:  0.16263041  0.33337129  0.05484016
#>  2: -1.12534736 -0.01459085 -0.56460970
#>  3:  1.13740298  1.65973385  1.23103526
#>  4: -0.17251462 -0.74192788 -0.06467517
#>  5:  0.23242358  0.53626850  0.68399963
#>  6: -0.89523133  0.01776016 -0.65218453
#>  7:  1.37772605  0.67430459  1.36146493
#>  8:  0.01994107  0.08830762  0.18946096
#>  9: -0.38992835 -0.24925197 -0.90303808
#> 10:  0.23092028  0.58651185  0.23138698

Notice that the sampled values respect the conditional distribution - they’re different from the original but plausible given the conditioning features.

ARF Sampler

The ConditionalARFSampler uses Adversarial Random Forests to model complex conditional distributions. It’s the most flexible conditional sampler.

Advantages:

Handles mixed feature types (continuous, categorical, ordered)
No distributional assumptions
Captures non-linear relationships

Limitations:

Requires fitting ARF model (slower initialization and sampling for large data)
More hyperparameters to tune
Stochastic sampling

# Create ARF sampler (works with full task including categorical features)
arf = ConditionalARFSampler$new(task_mixed, num_trees = 20, verbose = FALSE)

# Sample island conditioned on body measurements
sampled = arf$sample(
    feature = "island",
    row_ids = 1:10,
    conditioning_set = c("bill_length", "body_mass")
)

# Compare original and sampled island
original = task_mixed$data(rows = 1:10)
data.table(
    original_island = original$island,
    sampled_island = sampled$island,
    bill_length = original$bill_length, # Conditioning feature
    body_mass = original$body_mass # Conditioning feature
)
#>     original_island sampled_island bill_length body_mass
#>              <fctr>         <fctr>       <num>     <int>
#>  1:       Torgersen      Torgersen        39.1      3750
#>  2:       Torgersen      Torgersen        39.5      3800
#>  3:       Torgersen          Dream        40.3      3250
#>  4:       Torgersen          Dream          NA        NA
#>  5:       Torgersen         Biscoe        36.7      3450
#>  6:       Torgersen      Torgersen        39.3      3650
#>  7:       Torgersen          Dream        38.9      3625
#>  8:       Torgersen      Torgersen        39.2      4675
#>  9:       Torgersen          Dream        34.1      3475
#> 10:       Torgersen         Biscoe        42.0      4250

Use in CFI: ConditionalARFSampler is the default for Conditional Feature Importance since it can be used with any task, unlike other samplers.

Ctree Conditional Sampler

The ConditionalCtreeSampler uses conditional inference trees to partition the feature space and sample from local neighborhoods.

Advantages:

Handles mixed feature types
Interpretable tree structure
Automatic feature selection (only splits on informative features)

Limitations:

Requires tree building (slower than kNN)
May produce duplicates if terminal nodes are small

# Create ctree sampler
ctree = ConditionalCtreeSampler$new(task_mixed)

# Sample with default parameters
sampled = ctree$sample(
    feature = "bill_length",
    row_ids = 1:10,
    conditioning_set = "island"
)

original = task_mixed$data(rows = 1:10)
data.table(
    island = original$island, # Conditioning feature
    original = original$bill_length,
    sampled = sampled$bill_length
)
#>        island original sampled
#>        <fctr>    <num>   <num>
#>  1: Torgersen     39.1    38.7
#>  2: Torgersen     39.5    36.6
#>  3: Torgersen     40.3    46.0
#>  4: Torgersen       NA    36.2
#>  5: Torgersen     36.7    35.9
#>  6: Torgersen     39.3    39.7
#>  7: Torgersen     38.9    40.3
#>  8: Torgersen     39.2    34.1
#>  9: Torgersen     34.1    39.3
#> 10: Torgersen     42.0    36.2

The ctree sampler partitions observations based on the conditioning features and samples from within the same partition (terminal node).

kNN Conditional Sampler

The ConditionalKNNSampler finds k nearest neighbors based on conditioning features and samples from them.

Advantages:

Very simple and intuitive
Fast (no model fitting)
Single hyperparameter (k)
Automatic distance metric selection
Supports mixed feature types (numeric, categorical, ordered, logical)

Limitations:

Sensitive to choice of k
May produce duplicates if k is small

Distance metric:

The sampler automatically selects the appropriate distance metric based on conditioning features:

Euclidean distance when all conditioning features are numeric/integer (standardized)
Gower distance when any conditioning features are categorical (factor, ordered, logical)

Example 1: All-numeric conditioning (Euclidean distance)

# Create kNN sampler with k=5 neighbors
knn_numeric = ConditionalKNNSampler$new(task_numeric, k = 5)

# Sample x1 based on nearest neighbors in (x2, x3) space
sampled_numeric = knn_numeric$sample(
    feature = "x1",
    row_ids = 1:5,
    conditioning_set = c("x2", "x3")
)

original_numeric = task_numeric$data(rows = 1:5)
data.table(
    x2 = original_numeric$x2,
    x3 = original_numeric$x3,
    original_x1 = original_numeric$x1,
    sampled_x1 = sampled_numeric$x1
)
#>             x2         x3 original_x1 sampled_x1
#>          <num>      <num>       <num>      <num>
#> 1:  0.05484016 -0.6513588   0.1626304 -0.5121550
#> 2: -0.56460970 -0.3418385  -1.1253474 -0.2887009
#> 3:  1.23103526 -1.2727419   1.1374030  0.5183023
#> 4: -0.06467517 -0.7588294  -0.1725146  0.1626304
#> 5:  0.68399963 -2.8095712   0.2324236  1.2354913

Example 2: Mixed-type conditioning (Gower distance)

# Use task with categorical features
knn_mixed = ConditionalKNNSampler$new(task_mixed, k = 5)

# Sample bill_length conditioning on island (categorical) and body_mass (numeric)
sampled_mixed = knn_mixed$sample(
    feature = "bill_length",
    row_ids = 1:5,
    conditioning_set = c("island", "body_mass")
)

original_mixed = task_mixed$data(rows = 1:5)
data.table(
    island = original_mixed$island,
    body_mass = original_mixed$body_mass,
    original_bill = original_mixed$bill_length,
    sampled_bill = sampled_mixed$bill_length
)
#>       island body_mass original_bill sampled_bill
#>       <fctr>     <int>         <num>        <num>
#> 1: Torgersen      3750          39.1           NA
#> 2: Torgersen      3800          39.5         39.5
#> 3: Torgersen      3250          40.3         38.8
#> 4: Torgersen        NA            NA         38.8
#> 5: Torgersen      3450          36.7         38.7

The kNN sampler finds the k most similar observations (based on conditioning features) and samples from their feature values. The distance metric is chosen automatically based on feature types.

Knockoff Samplers

Now that we’ve seen conditional samplers, we can understand an important limitation of knockoff samplers: unlike the conditional samplers above, knockoffs don’t support arbitrary conditioning sets.

Knockoff samplers create synthetic features (knockoffs) that satisfy specific statistical properties. They must fulfill the knockoff swap property: swapping a feature with its knockoff should not change the joint distribution.

Knockoffs are a separate category because:

They require special construction to satisfy theoretical guarantees, and implementations are limited
They don’t support conditional sampling with arbitrary conditioning sets
They’re used primarily for feature selection with FDR control, not general importance

Gaussian Knockoffs

For multivariate Gaussian data, we can construct exact knockoffs:

# Create Gaussian knockoff sampler (using task_numeric from earlier)
knockoff = KnockoffGaussianSampler$new(task_numeric)

# Generate knockoffs
original = task_numeric$data(rows = 1:5)
knockoffs = knockoff$sample(
    feature = task_numeric$feature_names,
    row_ids = 1:5
)

# Original vs knockoff values
data.table(
    x1_original = original$x1,
    x1_knockoff = knockoffs$x1,
    x2_original = original$x2,
    x2_knockoff = knockoffs$x2
)
#>    x1_original x1_knockoff x2_original x2_knockoff
#>          <num>       <num>       <num>       <num>
#> 1:   0.1626304   0.2797155  0.05484016  0.30939036
#> 2:  -1.1253474  -0.5596204 -0.56460970 -1.01756047
#> 3:   1.1374030   1.4330571  1.23103526  1.33693941
#> 4:  -0.1725146  -0.2972019 -0.06467517 -0.41006558
#> 5:   0.2324236   0.2766873  0.68399963 -0.05528232

Key properties of knockoffs:

Different values but similar statistical properties
Pairwise exchangeability with originals
Preserve correlation structure
Cannot specify which features to condition on (determined by the knockoff construction)

Conditional Independence Testing: Knockoffs are particularly relevant for conditional independence testing as implemented in the cpi package. You can combine knockoff samplers with CFI and perform inference:

# CFI with knockoff sampler for conditional independence testing
cfi_knockoff = CFI$new(
    task = task_numeric,
    learner = lrn("regr.ranger"),
    measure = msr("regr.mse"),
    sampler = knockoff
)

# Compute importance with CPI-based inference
cfi_knockoff$compute()
cfi_knockoff$importance(ci_method = "cpi")

See vignette("inference") for more details on statistical inference with feature importance.

Key takeaways:

Permutation sampling produces any value from the marginal distribution
Conditional samplers produce values consistent with conditioning features
Choice of sampler affects feature importance estimates, especially when features are correlated

Sampler	Feature Types	Assumptions	Speed	Use Case
`MarginalPermutationSampler`	All	None	Very fast	PFI, uncorrelated features
`KnockoffGaussianSampler`	Continuous	Multivariate normal	Fast	Model-X knockoffs
`ConditionalGaussianSampler`	Continuous	Multivariate normal	Very fast	CFI with continuous features
`ConditionalARFSampler`	All	None	Moderate	CFI, complex dependencies
`ConditionalCtreeSampler`	All	None	Moderate	CFI, interpretable sampling
`ConditionalKNNSampler`	All	None (auto-selects distance)	Fast	CFI, simple local structure

Feature Samplers

Setup

Base Class: FeatureSampler

Key Properties

Permutation Sampler

Marginal Reference Sampler

Conditional Samplers

Gaussian Conditional Sampler

ARF Sampler

Ctree Conditional Sampler

kNN Conditional Sampler

Example 1: All-numeric conditioning (Euclidean distance)

Example 2: Mixed-type conditioning (Gower distance)

Knockoff Samplers

Gaussian Knockoffs

Summary