Consider the following setting:
Gaussian graphical model (GGM) assumption:
The data Xn × d consists of independent and identically distributed samples X1, …, Xn ∼ Nd(μ, Σ).
Disjoint group structure:
The d variables can be partitioned into disjoint groups.
Goal:
Estimate the precision matrix Ω = Σ−1 = (ωij)d × d.
where:
$S = n^{-1} \sum_{i=1}^n (X_i-\bar{X})(X_i-\bar{X})^\top$ is the empirical covariance matrix.
λ ≥ 0 is the global regularization parameter controlling overall shrinkage.
α ∈ [0, 1] is the mixing parameter controlling the balance between element-wise and block-wise penalties.
γ is the additional parameter controlling the curvature and effective degree of nonconvexity of the penalty.
Pα, γ(Ω) is a generic bi-level penalty template that can incorporate convex or non-convex regularizers while preserving the intrinsic group structure among variables.
Pγidv(Ω) is the element-wise individual penalty component.
Pγgrp(Ω) is the block-wise group penalty component.
pγ(⋅) is a penalty kernel parameterized by γ.
Ωgg′ is the submatrix of Ω with the rows from group g and columns from group g′.
The Frobenius norm ‖Ω‖F is defined as ‖Ω‖F = (∑i, j|ωij|2)1/2 = [tr(Ω⊤Ω)]1/2.
Note:
The regularization parameter λ acts as the scale factor for the entire penalty term λPα, γ(Ω).
The penalty kernel pγ(⋅) is the shape function that governs the fundamental characteristics of the regularization.
λp(ωij) = λ|ωij|.
$$
\lambda p_\gamma(\omega_{ij}) = \lambda\frac{\vert\omega_{ij}\vert}{v_{ij}},
$$ where V = (vij)d × d = (|ω̃ij|γ)d × d is a matrix of adaptive weights, and ω̃ij is the initial estimate obtained using penalty = "lasso".
$$ \lambda p_\gamma(\omega_{ij}) = \lambda(\gamma+\frac{2}{\pi}) \arctan\left(\frac{\vert\omega_{ij}\vert}{\gamma}\right), \quad \gamma > 0. $$
$$ \lambda p_\gamma(\omega_{ij}) = \lambda\left[1-\exp\left(-\frac{\vert\omega_{ij}\vert}{\gamma}\right)\right], \quad \gamma > 0. $$
λpγ(ωij) = λ|ωij|γ, 0 < γ < 1.
$$ \lambda p_\gamma(\omega_{ij}) = \lambda\log\left(1+\frac{\vert\omega_{ij}\vert}{\gamma}\right), \quad \gamma > 0. $$
$$ \lambda p_\gamma(\omega_{ij}) = \begin{cases} \lambda\vert\omega_{ij}\vert - \dfrac{\omega_{ij}^2}{2\gamma}, & \text{if } \vert\omega_{ij}\vert \leq \gamma\lambda, \\ \dfrac{1}{2}\gamma\lambda^2, & \text{if } \vert\omega_{ij}\vert > \gamma\lambda. \end{cases} \quad \gamma > 1. $$
$$ \lambda p_\gamma(\omega_{ij}) = \begin{cases} \lambda\vert\omega_{ij}\vert & \text{if } \vert\omega_{ij}\vert \leq \lambda, \\ \dfrac{2\gamma\lambda\vert\omega_{ij}\vert-\omega_{ij}^2-\lambda^2}{2(\gamma-1)} & \text{if } \lambda < \vert\omega_{ij}\vert < \gamma\lambda, \\ \dfrac{\lambda^2(\gamma+1)}{2} & \text{if } \vert\omega_{ij}\vert \geq \gamma\lambda. \end{cases} \quad \gamma > 2. $$
Note:
For Lasso, which is convex, the additional parameter γ is not required, and the penalty kernel pγ(⋅) simplifies to p(⋅).
For MCP and SCAD, λ plays a dual role: it is the global regularization parameter, but it is also implicitly contained within the kernel pγ(⋅).
Figure 1 illustrates a comparison of various penalty functions λp(ω) evaluated over a range of ω values. The main panel (right) provides a wider view of the penalty functions’ behavior for larger |ω|, while the inset panel (left) magnifies the region near zero [−1, 1].
library(grasps) ## for penalty computation
library(ggplot2) ## for visualization
penalties <- c("atan", "exp", "lasso", "lq", "lsp", "mcp", "scad")
pen_df <- compute_penalty(seq(-4, 4, by = 0.01), penalties, lambda = 1)
plot(pen_df, xlim = c(-1, 1), ylim = c(0, 1), zoom.size = 1) +
guides(color = guide_legend(nrow = 2, byrow = TRUE))
Figure 2 displays the derivative function p′(ω) associated with a range of penalty types. The Lasso exhibits a constant derivative, corresponding to uniform shrinkage. For MCP and SCAD, the derivatives are piecewise: initially equal to the Lasso derivative, then decreasing over an intermediate region, and eventually dropping to zero, indicating that large |ω| receive no shrinkage. Other non-convex penalties show smoothly diminishing derivatives as |ω| increases, reflecting their tendency to shrink small |ω| strongly while exerting little to no shrinkage on large ones.
deriv_df <- compute_derivative(seq(0, 4, by = 0.01), penalties, lambda = 1)
plot(deriv_df) +
scale_y_continuous(limits = c(0, 1.5)) +
guides(color = guide_legend(nrow = 2, byrow = TRUE))