BugReports: points at
/-/issues, the form the CRAN incoming check
requires. The incoming check on R-devel notes any
gitlab.com BugReports: whose path is not
/-/issues; a browser is redirected to
/-/work_items. README and the generated
pagerankr-package help topic follow
(PAGE-mehcgwzv).
rurl now installs from CRAN; the
Remotes: field is gone. rurl reached
CRAN as 3.0.1 on 2026-09-09, so DESCRIPTION no longer needs
a Remotes: entry – a field CRAN does not honor and which
drew a NOTE on every --as-cran run. Every hard dependency
now resolves from CRAN (PAGE-agtgxupt).
The declared rurl floor is
>= 3.0.1, up from >= 3.0.0.
rurl went 1.2.0 straight to 3.0.1 on CRAN and never
published a 3.0.0, so the old floor named a version nobody could install
– the second time this package declared a minimum that existed nowhere.
3.0.1 is the lowest rurl that exists as a release and that
the suite has been run against (PAGE-tsbkxhoz).
canonical_profile() now pins
credential_handling = "strip". rurl
3.0.1 added the argument. Userinfo is not part of a node key, but this
knob decides whether a credentialed URL yields a key at all: the
alternative, "reject", returns NA, which
clean_url_columns() keeps as its raw self and so turns
every credentialed URL into an opaque node. "strip" is
rurl’s current default, so no node key changes – the pin
freezes a default that could otherwise flip underneath the
graph.
The package is now hosted on GitLab, and the install
command has changed to
devtools::install_gitlab("bart-turczynski/pagerankr"). The
previous install_github() form no longer resolves. Bug
reports move to the GitLab tracker, and security reports are now made by
email to the maintainer rather than through GitHub private vulnerability
reporting (see SECURITY.md).
Convergence controls and damping now fail as
documented instead of silently changing the request or raising an
internal R error. Three fixes, all at the public boundary of
both pagerank() and compute_pagerank():
niter is rejected. It was passed through
as.integer(), so niter = 2.7 became an
iteration cap of 2 with nothing said. Values above
.Machine$integer.max are rejected for the same reason: the
coercion produced NA and reached the solver as a missing
cap.damping = NA_real_ (and NA_integer_,
NaN, Inf, -Inf) reports
`damping` must be a single numeric value between 0 and 1.
compute_pagerank() tested damping < 0 on an
NA, so the if received NA and R
raised “missing value where TRUE/FALSE needed”. Bare NA is
logical and already produced the named error; only the typed missings
and the infinities slipped through. Both entry points now share one
validator.eps is rejected. Inf passed
the eps <= 0 test, and an infinite tolerance is not a
tolerance.Added deterministic fixtures for graph shapes whose convergence and mass diagnostics were unpinned: disconnected components (a closed component holds exactly its share of the teleport vector; a dangling node moves mass across the component boundary), zero-weight edges (equivalent to no edge, with the target left on teleport mass alone), and the all-zero-weight graph (a named weight error).
Node identity moves off a presentation dial:
canonical_profile() now pins
url_standard = "whatwg" and
path_encoding = "keep". This changes node keys.
path_encoding is documented by rurl as a
presentation knob whose "decode" value “may
re-encode or decode reserved octets (so %2F may fold to a
path-separating /)”; pagerankr had been using it as its
identity function. Two consequences, both defects, are fixed:
http://example.com/a%2Fb and
http://example.com/a/b are two nodes rather than one. Same
for http://example.com/..%2fa, which previously popped a
segment it should not have.rurl 3.0.0
stopped resolving them under the old pin by reordering decode after
dot-segment removal, so /%2e%2e/a keyed to
/../a instead of /a. Under
"whatwg" the standard recognizes the encoded form directly
and no presentation dial is involved.Three further classes improve for crawl data: paths containing a tab
or newline, and URLs with surrounding whitespace, now parse (they
previously returned NA and fell through to the raw-token
fallback), and a literal space is percent-encoded rather than failing
the parse. The cost is that "whatwg" preserves percent
spellings, so /a%7Eb and /a~b are now two
nodes, as they are to a crawler.
rurl_params is narrowed. Because the
profile now pins a standard selector, rurl rejects a
conflicting override of the axes that selector governs —
case_handling and path_normalization. Those
two are node identity, so a caller who could move them could silently
re-key the graph. Every other knob still overrides as before.
Note rurl’s profile = "seo" is deliberately
not used and is not a substitute: it bundles
protocol_handling = "https",
www_handling = "strip",
trailing_slash_handling = "strip" and
index_page_handling = "strip", each of which is a redirect
class pagerankr models as an edge. Folding those at canonicalization
time turns the redirect into a self-loop.
canonical_profile() now follows “parse, do
not fold”. pagerankr has redirects and canonical tags as
first-class inputs, and those are the site’s own statement about which
URLs are the same page. Canonicalization therefore normalizes only what
the standard makes the same resource and asserts nothing about
site configuration. Three knobs move to match:
port_handling = "strip_default" (was
the rurl default "exclude"). A non-default port is a
different origin, so http://host:8080/a and
http://host/a are now two nodes. :80 on http
and :443 on https are redundant per spec and still
fold.host_encoding = "idna" (was
"keep"). An IDN host and its punycode form are the same
request on the wire, so no redirect or canonical can ever fold them –
the graph has to. http://münchen.de/a and
http://xn--mnchen-3ya.de/a are now one node, keyed on the
punycode form. Pass
rurl_params = list(host_encoding = "keep") to restore the
split.query_handling = "filter" (was the
rurl default "drop"). A contentful parameter is part of the
resource: ?color=red and ?color=blue are two
pages, and no redirect or canonical need exist between them. Tracking
parameters are not, and are still dropped via the built-in rurl denylist
(utm_*, fbclid, gclid).
This is the largest behavior change here – on a faceted
site, node counts will rise. The six query sub-options
(params_keep, params_drop,
params_case_sensitive, sort_params,
empty_param_handling, decode_plus) are now
key-shaping and so are pinned explicitly too, all at the rurl defaults.
sort_params = FALSE matches the rurl key contract, under
which query order and duplicates are significant.The knobs that would fold a redirect class –
www_handling, trailing_slash_handling,
index_page_handling, protocol_handling – were
already pinned to their non-folding values and are unchanged.
The declared rurl minimum is
>= 3.0.0. Two corrections landed in this cycle.
Imports: previously required
rurl (>= 2.1.0), a version that was never released —
rurl has no 2.1.0 tag, and its 2.x line begins
at 2.2.0 — so the stated minimum named an artifact no user could
install; that was corrected to >= 2.2.0, verified
against rurl 2.2.0, 2.2.1 and 2.8.0 in isolated libraries
with byte-identical canonical keys across all three. The floor then
moved to >= 3.0.0, which is where the path-identity axis
the profile now depends on lives. Note the 2.2.0 floor had also become
impossible to test in practice: rurl inserted
scheme_policy and scheme_acceptance
before url_standard rather than appending them, so
a positional url_standard argument silently binds to the
wrong argument on eight functions across that range.
New vignette, vignette("case-study"), works
the two-crawl fixture end to end. A worked example that scores
the same site before and after an internal-linking change on two lenses
— the editorial (content-only) graph and the full graph — and shows that
a change which moves the top page from rank #1 to #38 on the first is
invisible on the second (Pearson 0.9999). Every figure is computed from
inst/extdata/ at build time rather than quoted, so the
vignette doubles as a regression check on the fixture. It also documents
the three traps the comparison sets: scores do not sum to 1, so shares
must be taken against sum(pagerank); the two crawls have
different node sets, so absolute PageRank is not comparable across them;
and pages must be matched on the full URL, since external hosts share
path strings with internal ones. _pkgdown.yml gains an
articles: index grouping all six vignettes.
A two-crawl case-study fixture ships in
inst/extdata/. Two Screaming Frog exports of the
same small site, taken either side of a deliberate internal-linking
intervention, are now available as package data:
system.file("extdata", "reviews-microsite-before", package = "pagerankr")
and its -after counterpart, each holding
all_inlinks.csv and internal_all.csv. Because
the same site is measured twice, a change in PageRank is attributable to
the intervention rather than to a difference between sites — which is
what makes the pair usable for validating the placement and boilerplate
models. Both exports are trimmed to the columns
sf_contract() declares (18 and 24, down from Screaming
Frog’s 103), and Link Path is retained deliberately as the
boilerplate detector’s signal. The whole fixture is roughly 70 KB
gzipped. The underlying site is pseudonymous: hosts, paths, anchors, alt
text and page free-text are replaced rather than redacted, and
inst/extdata/README.md documents exactly what is synthetic
and what is preserved verbatim.
pagerank() surfaces a per-URL
wasted_mass figure alongside
page_state. When indexability_df or
status_df is supplied, the result now carries a
wasted_mass column next to page_state: the
authority each page collected and black-holed, i.e. its share of the
shared waste sink’s stationary mass. page_state labels the
“collects but cannot pass” class per row; wasted_mass
quantifies the amount, answering the other half of “a 404 page amassed
X PageRank and it ends up in a black hole.” Because a
waste-class page routes its whole throughput through one edge to the
absorbing sink, the value is damping / (1 - damping) times
the page’s own reported score — larger than, and deliberately distinct
from, that score. Summed over the class it equals the evaporated mass in
the transition audit (mass$sink); a "live"
page routes nothing to the sink, so its wasted_mass is
0. Like page_state, the column appears only
when one of those inputs is present, mirroring how
prior_weight appears only with
prior_df.
Documented that page_state and
simulate_changes()’s node_status are distinct
axes, not two names for one thing. page_state
(live / noindex / robots_blocked / response_dead) is a page’s
health/index state; node_status (normal / new-target /
removed-dead) is a node’s role in a before/after comparison. They are
kept separate on purpose: new-target has no health
analogue, and removed-dead is the single value bridging
both axes (a node removed because its proposed health state is
a forced 404). The two @return sections now cross-reference
each other.
simulate_changes() gains a
remove_urls verb for URL-level what-ifs. Pass a
character vector of URLs to model them returning 404: each keeps its
inbound links (other pages still point at it) but their authority now
flows into a dead page and evaporates to the shared waste sink,
rather than redistributing across the site via teleport (dangle) or
self-amplifying via a self-loop. The page’s outbound links are dropped
and the node stays in the output flagged "removed-dead" in
node_status, so its residual absorbed mass is never misread
as earned authority. This is the first consumer of the
status_df / waste-sink dead-class mechanism:
remove_urls forces a 404 into the proposed
model only, leaving the baseline live. A URL appearing in both
remove_urls and redirect_urls_df is an error
(a page cannot be both a 301 and a 404); the manifest gains
urls_removed. Available on the Screaming Frog path via
simulate_changes_screaming_frog(remove_urls = ...), where
the forced 404 composes on top of the bundle’s real crawled
status.
Page-state modeling — one faithful policy for pages that
collect PageRank but cannot pass it (epic
PAGE-qzskzcfd). The three changes immediately
below — status_df, the unified waste sink, and
prior_exclude_waste — together replace three
accidental, inconsistent behaviors for the noindex /
robots-blocked / 4xx-5xx class with a single model: authority flows in,
is booked, and stops. Before, the same class behaved three ways: a
robots-blocked page self-looped and self-amplified up to
8.3× (scoring 0.8875 trapped vs 0.1065 evaporating — holding
89% of a graph), 4xx/5xx pages were scored as ordinary live vertices,
and uniform teleport let 1,000 fake dead URLs capture 95.2% of a
site’s PageRank for merely existing. The class now routes to a
shared waste sink, is excluded from the teleport vector, and is labeled
per-row in page_state so absorbed mass is never read as
earned authority. robots_blocked_action survives only as a
presentation toggle ("show" / "vanish"); the
self-looping "trap" is gone. Any graph containing a
class member changes score — accepted pre-CRAN (experimental,
single consumer). Reproducible experiment and full rationale:
notes/pagerank-behavior-field-notes.md §11.
pagerank() gains
prior_exclude_waste to keep the “collects but cannot pass”
class out of the teleport vector. With the default
TRUE, the noindex, robots-blocked, and 4xx/5xx pages
identified by indexability_df / status_df
receive zero teleport mass: they still collect the authority their
inlinks send, but are no longer paid the uniform teleport share for
merely existing, so a page can no longer manufacture authority by
pointing at many dead ends (Page & Brin 1998 criticize uniform
teleport for “valuing pages simply because they exist”). This is a
first-class replacement for the error-prone prior_df +
prior_alpha workaround previously needed to zero teleport
entries. Set FALSE to restore uniform teleport over every
page, matching igraph::page_rank() for canonical
comparisons. Scores change whenever a non-empty class is
present; graphs with no such class — and any run without
indexability_df/status_df — are numerically
unchanged. The synthetic evaporation and leak sinks are excluded from
teleport regardless of this flag, which is recorded in the transition
audit as config$prior_exclude_waste. No effect under
reverse = TRUE, where the class inputs are already
rejected.
pagerank() gains a status_df
input for HTTP response status. Supply a url +
status_code frame (column names configurable via
status_url_col and status_col) and pages that
returned a 4xx or 5xx code are identified as response-dead and
counted in the transition audit (config$has_status and
dropped$n_status_dead). 4xx and 5xx are treated identically
— a crawl is a snapshot, and a transient 503 is
indistinguishable from a permanent 404 at crawl time (the
same reason a 302 is folded exactly like a
301); 3xx stays with
redirects_df, and unparseable or sub-400 codes
are treated as live. pagerank_screaming_frog() now feeds
status from the bundle’s node table automatically instead of discarding
it, and the raw preset leaves it off. Not supported with
reverse = TRUE.
Unified waste-sink mechanism for the “collects PageRank but cannot pass it” class. noindex, robots-blocked, and response-dead (4xx/5xx) pages now share one mechanism: each loses its outgoing edges and gains exactly one edge to a shared internal waste sink, so it absorbs the authority its inlinks send but passes none back into the graph. This removes the old robots-blocked self-loop, which was an absorbing rank sink that compounded inbound authority every iteration (a measured 8.3× inflation); robots-blocked scores change accordingly. A no-outlink 404 now gets the one sink edge that stops it from dangling and recycling its authority to every page via teleport.
robots_blocked_action values are renamed
c("trap", "vanish") → c("show", "vanish"),
default "show" (the self-loop “trap” is gone). Both values
route the page’s throughput to the sink; they differ only in whether the
page is shown ("show") or removed with its own mass booked
as hidden ("vanish").nofollow_action: a noindex
page always routes to the sink, and nofollow_action now
governs only real rel=nofollow edges. This changes noindex
behavior only under nofollow_action "drop" /
"keep"; the default "evaporate" is numerically
unchanged.indexability_df or status_df is
supplied, the result gains a page_state column tagging each
visible page "live", "noindex",
"robots_blocked", or "response_dead", so
wasted mass is attributable per URL (robots-blocked > response-dead
> noindex when a page carries more than one signal). It appears only
with those inputs, mirroring how prior_weight appears only
with prior_df.simulate_changes() can now model URL-level
what-ifs, not just edge-level ones. The new
redirect_urls_df argument (a two-column
from/to frame) models retiring a page behind a
redirect: it strips the live source’s own outbound links before folding,
so the target inherits the source’s inbound authority only —
never its outlinks — and the source drops out of the proposed set. A row
for a source overrides any prior redirect for it (an
earlier row, or the baseline crawl’s real 3xx), so “repoint A→B into
A→C” is a single override. Redirects pass authority through at 100% (the
only per-hop loss is the global damping every edge already incurs). This
replaces the removed add_redirects_df, which folded a live
source without stripping its outedges and so modeled a content
move rather than a retire.
New
on_unknown_target = c("warn", "error", "allow")
controls what happens when a redirect or link target is not yet a node
in the graph (it may be a legitimate new page); the default
"warn" proceeds and models it as a new node carrying
inbound authority.
The comparison output gains a node_status
column ("normal" or "new-target")
plus proposed (the full proposed pagerank()
result, with its transition audit) and manifest (redirects
applied/overridden, link add/remove counts, unknown targets)
attributes.
New
simulate_changes_screaming_frog() applies the same
what-if verbs to a screaming_frog_bundle, reusing the same
bundle → pagerank() adapter as
pagerank_screaming_frog() so placement, boilerplate,
nofollow, and status are preserved automatically. Both entry points
share one changeset engine.
Positional decay: pagerank() can now weight
links by their reading order within the page. Point the new
position_col at a numeric per-source position index —
1 for the first link, 2 for the second — and
each edge’s weight is scaled by a reading-order decay. This is the
orthogonal axis of the edge-weighting model: where placement and
boilerplate describe templatedness and feed one graded axis
combined by minimum, position describes reading order and
composes by multiplication, so an above-the-fold
boilerplate CTA (0.5 × 1.0) outranks a trailing organic
link (1.0 × 0.2) with no special-casing. The decay reuses
the existing transform_weights() shapes:
position_transform is "zipf" (default,
1 / rank^position_alpha) or "rank_linear",
applied within each source page’s choice set, and
position_floor (default 0.01) clamps the
result strictly above zero so compounding two axes can never reach an
effective drop. Off unless position_col is supplied, and
crawler-neutral: any crawler that can report a link’s order can drive
it. Recorded separately in the transition audit’s
config$position, so an edge’s weight can be explained as
region times reading order.
screaming_frog_links() now materializes that
index at ingest. The edge table gains a
position_index column carrying each link’s reading-order
rank among its source page’s content links — computed from document
order for an All Outlinks export, and left
NA for All Inlinks, whose row order is
destination-alphabetical rather than document order.
pagerank_screaming_frog() therefore drives the axis with
position_col = "position_index"; it stays opt-in, so pass
that argument to switch it on.
pagerank_screaming_frog() gains a
preset argument, and preset = "raw" now
actually produces the as-crawled graph. Previously the wrapper
fed pagerank() the bundle’s declared canonical, redirect
and indexability tables unconditionally, so a preset — which sets
policy, never data — could not unset them, and
preset = "raw" still folded canonicals/redirects and
trapped robots-blocked pages. The "raw" view now switches
all three declared tables off at the wrapper level. An explicit
apply_canonicals or apply_redirects still
overrides the "raw" default for that table; the import
audit records apply_indexability alongside the existing two
flags.
Boilerplate detection: pagerank() can now
discount repetitive template links that sit in the main
content. Point the new container_col at a column
identifying each link’s source-side component and every edge is scored
by container-conditioned recurrence: of the pages that
component appears on, the share on which it points at this same target.
A recycled call-to-action that always links the same place scores near
1 and is discounted; a related-posts module that links
somewhere different on every page scores near 0 and keeps
full weight — even though both recur identically. This catches what
placement never can, because bylines, compliance links and recycled CTAs
are structurally content. Tuned by boilerplate_threshold
(default 0.5, the ratio at which an edge is
classified), min_container_pages (default
10, an evidence floor) and boilerplate_weight
(default 0.5, the discount applied once
classified). The last two constants are unrelated quantities that
happen to share a value. Off unless container_col is
supplied, and crawler-neutral: any crawler that can identify a link’s
component can drive it.
New vignette("boilerplate")
documents the detector: the container-conditioned metric and the
direction of its ratio, the four arguments and why
boilerplate_threshold and boilerplate_weight
are unrelated despite sharing a default, how region weighting and
recurrence compose as one graded axis rather than two multiplied ones,
and which kinds of page the detector characteristically
discounts.
New sf_container_from_path()
derives that component identity from a Screaming Frog
Link Path, and screaming_frog_links() now
carries a container column on both the observation and edge
tables (counted in the link diagnostics as container_rows).
The rule is: strip numeric [n] predicates,
keep [@class='…'], then drop the trailing
<a> step. This cuts the opposite way from
sf_region_from_path(), which strips class predicates so a
div[@class='site-footer'] is not read as a
<footer>; the two answer different questions (“which
region is this” versus “is this the same component”) and the
inconsistency is deliberate. Rows with no Link Path get
NA and stay unscored — unlike placement there is no
Link Position fallback, because a region label cannot
manufacture a component identity.
pagerank_screaming_frog() does not
enable the detector on its own. Pass
container_col = "container" to opt in, exactly as a
non-Screaming-Frog caller would; supplying the column is what switches
detection on, so passing it automatically would change the default
view.
Placement and recurrence are two detectors feeding one
graded axis, not two independent axes. A nav link is
boilerplate by construction, so the two factors are not
multiplied — that would discount the same link twice for the same fact.
The strongest applicable discount wins: chrome 0.1,
repetitive in-content 0.5, unique in-content
1. The transition audit still records both factor sets
separately in config$placement and
config$boilerplate, since the resulting weight alone cannot
say which detector produced it.
The internal synthetic weight column is renamed
.__pr_placement_weight__ to
.__pr_edge_weight__, since both weighting
axes now write it. It surfaces in
attr(result, "transition_audit")$config$weight_col.
Two new presets complete the set: "reversed"
and "content". pr_preset() now
registers raw · declared ·
reversed · content. "reversed" is
the feeder view (reverse = TRUE); it is a no-op rather than
an error under topic_feeder_pagerank(), which reverses the
graph itself. "content" carries the placement recipe
c(content = 1, nav = 0.1, header = 0.1, footer = 0.1, aside = 0.1),
so links found in site chrome are discounted to a tenth of an in-content
link — downweighted, never dropped. All five regions are named on
purpose: unnamed placements keep weight 1, so a partial recipe would
leave footer and aside outweighing nav tenfold. Because a preset sets
policy and never data, "content" still needs you to supply
placement_col; pagerank_screaming_frog()
supplies it from the bundle.
New vignette("presets") documenting
every preset’s full expansion, the precedence rule, and how provenance
is recorded in the transition audit.
Asking for placement_weights or
accepted_placements without placement_col now
names the preset responsible when a preset is what set them, instead of
reporting an argument the caller never typed.
Placement-aware scoring is now crawler-neutral.
pagerank() gains placement_col,
accepted_placements, and placement_weights:
point placement_col at a column holding the page region
each link sits in and the region can filter or weight edges directly,
e.g.
pagerank(edges, placement_col = "region", placement_weights = c(content = 1, nav = 0.1, header = 0.1, footer = 0.1, aside = 0.1)).
Placement is not a Screaming Frog concept — any crawler that reports
link regions can drive it, with a per-crawler adapter
(sf_normalize_position() for Screaming Frog) mapping vendor
labels onto the shared vocabulary. All three arguments default to
NULL, so nothing changes for callers that do not use them,
and because they are pagerank() formals they compose with
preset and are inherited by every wrapper that forwards
....
A Screaming Frog bundle’s placement is now
derived from the DOM path rather than from
Link Position, via a new exported
sf_region_from_path(). Link Position loses the
enclosing region whenever a <nav> is nested inside
one: on a site whose footer is marked up as
footer > nav > a, Screaming Frog reports every footer
link as Navigation and emits no Footer bucket
at all, so footer is not merely mislabeled but unreachable
— a user wanting footer at 0.05 and nav at 0.2 had no way to express it.
Other sites do emit Footer, so the vocabulary
silently varied with the site’s markup. The region is the outermost
layout container on the path (header, footer,
aside, else content), and nav
applies only to a <nav> that is not inside one of
those; Link Position remains the fallback for rows with no
path, counted in the link diagnostics as
placement_from_position_rows. Expect footer and header
navigation to reclassify out of nav — this changes which
edges accepted_placements selects, though not the usual
recipe, which weights all three alike.
The placement vocabulary term "sidebar" is renamed
"aside", matching both Screaming Frog’s
own label and the HTML element; "sidebar" is a layout word
rather than a semantic one. sf_normalize_position() now
returns "aside" where it previously returned
"sidebar", and "sidebar" is no longer accepted
in accepted_placements /
placement_weights.
pagerank_screaming_frog() keeps its signature but is
now a thin adapter: accepted_placements and
placement_weights are forwarded to pagerank(),
which owns the filtering and weighting. Two consequences for the
"screaming_frog_import" attribute:
scoring$scored_edge_rows is renamed
scoring$edge_rows_to_pagerank (it counts rows after the
wrapper-owned link origin filter, since placement filtering has moved
down a layer), and scoring$effective_weight_col is dropped
— the transition audit already carries it, alongside a new
config$placement entry recording the placement column,
filter, weights, and the number of rows the filter dropped.
New preset argument on pagerank() and a
new exported pr_preset() constructor: named argument
bundles for recurring views of a link graph, so a view is a
one-liner instead of a hand-assembled argument list. Two presets ship so
far: "raw" (the graph exactly as crawled – self loops and
isolates kept, rel=nofollow ignored, no relabeling onto
uncrawled fold targets) and "declared" (honor the signals
the site declares – nofollow evaporates, declared canonical and redirect
targets are followed, robots-blocked pages keep the authority they
collect, self loops and isolates dropped). "declared" is a
pure pin of the package defaults: it changes nothing
about how pagerank() behaves, it states the default view so
a run can record which view was intended and stay pinned to that bundle
if a default later moves. preset accepts a preset name, a
pr_preset() result, or any hand-rolled named list of
pagerank() arguments; bundles are plain named lists, so
they are inspectable and spliceable via do.call().
Precedence is explicit argument > preset > base
default – a preset value is applied only to arguments the
caller did not name – and this holds through the wrappers that forward
... to pagerank() (trustrank(),
topic_sensitive_pagerank(),
topic_feeder_pagerank(),
pagerank_screaming_frog()), with the boundary that
arguments a wrapper sets itself stay wrapper-owned. Presets are strictly
opt-in; no default behavior changes.
The transition_audit attached to a
pagerank() result now records which preset produced it, in
config$preset: the preset name for a registered preset
(passed by name or as a pr_preset() result),
"custom" for a hand-rolled bundle, and NULL
when no preset was used. Two runs that expand to the same configuration
are no longer indistinguishable in the audit – which named view
was asked for is itself a provenance fact, and it is the only durable
record of intent for a pin like "declared".
print() shows a Preset: line only when a
preset was used, so default output is unchanged.
Breaking: the six dot-prefixed Screaming Frog
helpers are renamed without their leading dot and are now documented
public API: .sf_contract(), .sf_read_input(),
.sf_parse_follow(), .sf_rel_nofollow(),
.sf_normalize_position(), and
.sf_graph_eligible() become sf_contract(),
sf_read_input(), sf_parse_follow(),
sf_rel_nofollow(), sf_normalize_position(),
and sf_graph_eligible(). A leading dot conventionally
signals “internal”, but these were already exported – and exported with
@noRd, so they had no help pages at all. They are intended
for downstream Screaming Frog workflows, so they now carry full
documentation (parameters, return values, examples) and are grouped as
@family Screaming Frog toolkit with their own pkgdown
reference section. Behavior and signatures are unchanged. The many
genuinely internal .sf_* helpers keep their leading dot and
remain unexported.
Breaking: resolve_urls() is renamed
to resolve_redirect_urls(). The function originally
resolved redirects and canonicals together; once that was split
apart (mixing the two is lossy) the old name no longer described what it
does. The new name states the operation and restores symmetry with the
rest of the resolve_* family, which pairs an edge-list form
with a URL-vector form: resolve_redirects() /
resolve_redirect_urls() alongside the existing
resolve_canonicals() /
resolve_canonical_urls(). Behavior, arguments, and return
value are unchanged.
The resolve_* naming convention is now documented
and enforced by grouping: functions taking a character vector of URLs
are tagged @family URL-vector resolvers
(resolve_redirect_urls(),
resolve_canonical_urls(),
resolve_folded_urls()) and functions taking an
edge_list_df are tagged
@family edge-list resolvers
(resolve_redirects(), resolve_canonicals(),
resolve_links()), so each help page cross-links its
input-type peers.
The weight/teleport transform vocabulary now uses one canonical
option order ("none", "log",
"percentile", "minmax", "zipf",
"rank_linear") at every call site –
transform_weights(), transform_edge_weights(),
pagerank(prior_transform=),
align_prior_to_vertices(), and
ga4_entrance_teleport(). No defaults change:
transform_edge_weights() still defaults to
"zipf" (now declared explicitly rather than by option
order) and every other site still defaults to
"none".
Documented that canonical_profile() deliberately
leaves rurl’s component-dropping knobs unpinned
(query_handling, port_handling, and the
url_standard selector added in rurl 2.2.0) –
they have no effect on the scheme+host+path node key at their defaults.
Added a behavioral guard in test-canonicalization.R
asserting a canonical key drops the port, query, and fragment, so a
future rurl default flip on an unpinned knob is caught here
rather than silently changing node identity. No node keys change;
verified against rurl 2.2.0.
canonical_profile() now pins
path_normalization = "dot_segments" and
path_encoding = "decode" (previously "none" /
"keep"). rurl 2.1.0 silently redefined those
two default values to keep the path verbatim, which changed node keys
for any URL with dot-segments (/a/../b) or percent-encoding
(/%41, %20) and desynced the pagerankr
<-> semantic node join. Pinning the explicit values restores the
original committed key (path percent-decoded, dot-segments removed) and
keeps node identities stable across the rurl upgrade. The
semantic sibling pins the identical profile (changed
together).
clean_url_columns() now preserves tokens that
rurl cannot parse as a URL (e.g. a dotless bare label such
as "A") as their raw value instead of turning them into NA.
Newer rurl (>= 2.1.0) normalizes such dotless tokens to
NA; combined with the rurl floor bump in the follow-up,
this had silently collapsed non-URL node identities to NA — they were
then dropped by get_unique_edges(), so [pagerank()]
returned an empty result for any graph built from bare labels.
Unparseable-but-present tokens are now kept as opaque nodes (only
genuinely missing NA inputs stay NA), mirroring
.apply_fold_map()’s leave-unmapped-values-untouched
contract.
filter_links_by_domain()’s encoding-independent
registrable-domain matching now reads rurl’s new
domain_ascii column (rurl >= 2.1.0) instead
of a separate IDNA-forced parse. .build_url_maps() parses
each unique URL once (host + domain_ascii) rather than
twice, and the .domain_profile() forced-idna helper is
gone. Behavior is unchanged — münchen.de and
xn--mnchen-3ya.de still fold to one key under every
host_encoding.
pagerank_screaming_frog() gains
apply_canonicals and apply_redirects toggles
(both TRUE by default, preserving current behavior).
Setting either to FALSE skips folding the bundle’s
canonical / redirect signals into [pagerank()] (passes
canonicals_df / redirects_df as
NULL), giving a supported as-crawled run that keeps the
crawled node identities — the escape hatch for crawls whose canonicals
point off the crawled domain (mirror / staging hosts) and would
otherwise relabel crawled pages onto uncrawled targets. The
reserved-argument guard still blocks the raw canonicals_df
/ redirects_df pagerank arguments.
screaming_frog_bundle() now reports an off-domain canonical
count (counts$canonicals_off_domain, surfaced in
summary()/print()), reusing the existing
absent-target classification, and the wrapper exposes it on the
screaming_frog_import audit so the mirror-staging scenario
is visible at import and scoring time.
pagerank() now detects fold-target
collisions: when a canonical/redirect relabels a crawled page’s
node onto an uncrawled URL that is ALSO independently
referenced as a genuine link endpoint, the two silently merge into one
vertex and the crawled page absorbs the inbound link equity of that
uncrawled URL. pagerank() emits a warning()
naming the merged URL(s) and records them in the fold
section of the transition_audit object under a new
collisions field (a data frame of target /
n_independent_refs / source, or
NULL when none). The crawl’s known-URL set
(indexability_df) is used to tell an uncrawled fold target
from a genuinely crawled leaf page, so the diagnostic is only computed
when an indexability_df is supplied.
pagerank() now warns when a
keep_domains / exclude_domains /
keep_hosts / exclude_hosts value matched the
crawled input but no node after folding — i.e. an out-of-scope
canonical/redirect rewrote the crawled domain/host away before filtering
(which runs after folding). The warning names the folded-away value(s)
and points at the fold as the cause. The fold-then-filter ordering is
now documented explicitly in the pagerank() and
filter_links_by_domain() docs; to scope the crawled input,
filter with filter_links_by_domain() before calling
pagerank().
out_of_scope_fold gains a third policy,
"leak": a crawled page whose canonical/redirect folds out
of scope is treated like an external redirect — its inbound equity is
routed onto a dedicated leak sink and evaporates out of the measured
graph (its outbound edges are dropped), so it does not rank and its
equity is not credited to any surviving page. The evaporated equity is
reported as a new leaked term in the
transition_audit mass accounting, which now
decomposes as reported + sink + leaked + hidden = total
(= 1); leaked is 0 for
"relabel"/"keep" runs so their totals are
unchanged. The fold audit section reports
policy == "leak".
pagerank() gains an out_of_scope_fold
argument ("relabel" default, or "keep")
governing composed fold-map entries whose target is not itself a crawled
node. "relabel" preserves current behavior (fold crawled
sources onto uncrawled canonical/redirect targets); "keep"
drops those out-of-scope entries before folding so crawled pages retain
their as-crawled identity rather than being relabeled to phantom
vertices (the same filtered map folds the TIPR prior). Regardless of
policy, the count and list of out-of-scope folds (source, target,
signal) are recorded in a new fold section of the
transition_audit object.
Code-quality pass: anyNA() replaces
any(is.na()), !all(x) replaces
any(!x), nested ifelse() replaced with
vectorized assignment,
expect_gt()/expect_lt()/expect_length()/expect_null()
adopted where applicable, and redundant c() wrappers around
single-string aliases removed.
pagerank() and compute_pagerank() gain
convergence controls and reporting. The new algo argument
selects the igraph::page_rank() back-end
("prpack", the fast exact default, or
"arpack", the iterative solver), and the friendly
eps / niter aliases re-introduce the L1
tolerance and maximum iteration count that modern igraph
dropped, mapping onto the ARPACK options$tol /
options$maxiter; supplying either transparently switches to
ARPACK. Every non-empty result now carries a "convergence"
attribute (a pagerank_convergence object) reporting the
solver, iteration count (when the solver exposes it), and a
solver-independent post-hoc L1 residual ||Gx - x||_1 of the
returned vector — a genuine quality check comparable across both
back-ends. Docs cover the damping/iteration-count rule of thumb
log10(eps) / log10(damping).
New topic_feeder_pagerank() answers the inverse of
topic_sensitive_pagerank(): not “which page is most
authoritative for this cluster” but “which pages feed /
power this cluster” — the internal hubs whose outlinks point into
the target pages. It seeds the teleport prior on the cluster and runs
pagerank() on the transposed graph
(reverse = TRUE), so mass walks backward along links and
accumulates on the feeders, attenuating with link distance. This is the
reverse-graph sibling of Topic-Sensitive PageRank and the cluster-biased
counterpart to the global inverse PageRank
(pagerank(reverse = TRUE)); the feeders are the
high-pagerank rows with prior_weight == 0
(cluster pages carry teleport mass directly). Pure orchestration over
the existing TIPR personalization path on the reversed graph — no new
solver. New topic_feeder_pagerank vignette walks through
the AI-Agent-cluster use case and contrasts it with the forward
authority view and with HITS hubs.
New salsa() and compute_salsa() add
Lempel & Moran’s (2001) SALSA hub and authority scores: a stochastic
variant of HITS that runs the mutual-reinforcement step as
PageRank-style random walks on the bipartite hub/authority graph, so the
scores are stationary distributions rather than dominant eigenvectors.
Computed over the same cleaned, redirect/canonical-folded,
domain-filtered, deduplicated link graph as pagerank(), so
hub, authority, and PageRank share node identities. Uses the
degree-based closed form (Proposition 6) — no eigenvector iteration —
with the required weakly-connected-component mass-weighting correction
so cross-component scores stay comparable on crawls with orphan
clusters. Each side sums to 1; coverage differs from PageRank by design
(hub is NA for pure sinks,
authority is NA for pure sources). v1 is
unweighted; a weighted extension is deferred. Documented as a site-graph
adaptation of the original focused-subgraph algorithm.
New trustrank() and the shared
seed_prior() builder add TrustRank-style seed-biased
PageRank (Gyöngyi, Garcia-Molina & Pedersen, 2004): personalized
PageRank whose teleport vector is concentrated on a set of trusted seed
pages, so trust flows outward from the seeds and attenuates with
distance (the damping factor is the attenuation).
seed_prior() builds a prior_df from a seed set
(character vector or weighted data.frame; equal weights
reproduce the original uniform seed distribution) and is
orientation-agnostic — the same builder feeds
topic_feeder_pagerank() on the reversed graph.
trustrank() is the one-call wrapper that builds the seed
prior and runs pagerank() with it. Pure orchestration over
the existing TIPR personalization path — no new solver; seed selection
is the caller’s (it is seed-biased PageRank, not a spam classifier). New
trustrank vignette walks through a worked example.
New topic_sensitive_pagerank() computes per-topic
PageRank by running the existing pagerank() engine once per
topic with a teleport prior biased toward each topic’s seed cluster,
then blends the per-topic scores into a single ranking. This is
Haveliwala’s (2002) Topic-Sensitive PageRank adapted to a single site:
each “topic” is a content cluster (e.g. the pricing or
AI-Agent area) given as a character vector of seed URLs or a
weighted data.frame. Returns one score column per topic
plus a weight-normalized blended column, with the per-topic
transition_audit objects attached. Pure orchestration over
the TIPR personalization path — no new solver, and topic membership is
supplied by the caller, not inferred.
New smooth_transitions() shrinks sparse empirical
page-transition shares (e.g. from ga4_page_transitions())
toward the crawl-graph link structure, so no valid crawled link is ever
assigned exactly zero probability. Uses a per-source
Dirichlet/pseudocount shrinkage weight
lambda_i = n_i / (n_i + k) that increases with the source
page’s sample size, with min_support fallback to the prior,
optional weighted priors, and an origin diagnostic
(both / empirical_only /
structural_only). Time decay and device/template/channel
segmentation are handled upstream by shaping the count input.
New hits() and compute_hits() add
Kleinberg’s HITS hub and authority scores, computed with
igraph::hits_scores() over the same cleaned,
redirect/canonical-folded, domain-filtered, deduplicated link graph as
pagerank(), so hub, authority, and PageRank share node
identities. Docs cover the matrix formulation (authority = dominant
eigenvector of A^T A, hub = dominant eigenvector of
A A^T) and the whole-graph caveat: unlike Kleinberg’s
query-focused base set, these are site-wide structural
centralities.
pagerank() now has an explicit
duplicate_edge_policy for repeated
from -> to rows after URL folding and filtering. The
default "collapse" preserves the standard
binary/destination-level PageRank convention and previous results;
opt-in "aggregate" sums duplicate numeric weights with
aggregate_edges() semantics; opt-in
"count_instances" models a link-slot surfer where repeated
links increase transition probability and records instance-count details
in the transition audit.
Recorded representative Screaming Frog crawl acceptance results and added package-level operational documentation for required exports, default graph policy, optional origin/placement policies, and contract pinning.
New pagerank_screaming_frog() scores a
screaming_frog_bundle() through the existing
pagerank() pipeline, feeding only graph-eligible hyperlink
edges while attaching Screaming Frog import diagnostics beside the
transition audit. Placement/origin filtering and placement-derived
weighting are explicit opt-ins.
New screaming_frog_bundle() composes Internal: All
and All Inlinks/Outlinks adapters into the stable crawl handoff object
with raw observations, graph edges, node signals, cross-table
reconciliation diagnostics, provenance, and concise print/summary
methods.
New canonical and composed URL-resolution helpers:
resolve_canonicals(),
resolve_canonical_urls(), and
resolve_folded_urls() expose the existing fold-map engine
for rel=canonical and redirect+canonical URL folding without duplicating
resolver logic.
Documented the indexed-corpus assumption used by
pagerank(): noindex pages may receive authority but their
outlinks are treated as nofollow for propagation within the indexed
graph. The docs now distinguish slot-consuming "evaporate",
slot-removing "drop", and normally followed
"keep" without attributing this package model to
Google.
New screaming_frog_links() imports All
Inlinks and All Outlinks with identical
Source-to-Destination orientation, preserving raw duplicate observations
while deriving explicit Hyperlink-only graph edges with nofollow,
placement, origin, endpoint, and exclusion diagnostics.
New screaming_frog_internal() imports UTF-8/BOM
Internal: All exports with alias-insensitive schema
detection and selective file reads. It returns deterministic node,
redirect, canonical, and indexability tables while preserving raw URLs
and reporting missing, duplicate, invalid, and ignored input
facts.
pagerank() now attaches a
transition_audit provenance object to its result as
attr(result, "transition_audit") (backward-compatible):
row/edge counts, behavioral-weight coverage, normalization total,
dropped data (NA / dedup / self-loop rows, unmatched prior URLs),
robots-blocked count, and the model configuration used. Has a
print method.
The transition_audit object’s mass
field now decomposes the page-mass deficit precisely into
reported (visible page mass), sink (evaporated
nofollow-sink mass), hidden (robots-blocked mass), and
total (= 1 by construction) — replacing undifferentiated
“leakage” language with precise evaporated/hidden accounting.
New aggregate_edges(): loss-aware post-fold edge
aggregation with explicit per-column semantics (sum counts, boolean
conflict policy any/all/
majority/error, preserve_cols
list-columns for placement features).
New transform_edge_weights(): per-source grouped
weight transforms reusing transform_weights() methods,
emitting a per-source transition_probability that sums to 1
within each by group.
New ga4_page_transitions(): consecutive-page-view
transition counts from a GA4 BigQuery export with a deterministic
session/event ordering contract (timestamp + batch tie-breaks). A
transition signal, not a link-click signal.
New ga4_entrance_teleport(): entrance/landing-page
counts as a teleport (reset) vector reusing the prior_df /
align_prior_to_vertices() machinery; documented as a proxy,
distinct from the backlink-authority prior.
Initial CI and lint baseline.
The OSS Index dependency audit in
tests/testthat/test-security.R scopes to hard dependencies
(Depends + Imports) instead of the
Suggests tree. oysteR::expect_secure() audits
Suggests too, which pulled in oysteR’s own recursive
dependencies – curl among them – and failed the pre-push
gate on a vulnerability in the auditor rather than in anything pagerankr
ships. Scoped to hard dependencies the audit covers 23 packages and is
clean; the old scope covered 126 (PAGE-aqnbdkov).
CI adds CRAN behind the image’s Posit Package Manager repo. p3m
lags CRAN by up to a day for a new release – on the day rurl 3.0.1 was
published, the 2026-09-09 p3m snapshot and p3m
latest both still served 1.2.0 – so
Imports: rurl (>= 3.0.1) was unresolvable at any pin and
every dependency-resolving job failed. p3m stays first, so packages
still arrive as prebuilt binaries; CRAN only supplies what p3m has not
synced yet.
The rurl-floor CI job resolves the floor from CRAN
(current or archived) instead of reading the source forge out
of Remotes: and installing a git tag. It now runs on every
merge request rather than manually, because it can pass. Its
GitHub-hosted twin, which had no live runner and hard-failed on the
removed Remotes: field, was deleted; the GitLab job
supersedes it.
The pre-push hook’s non-gating rurl skew notice compares the
installed rurl against CRAN rather than against the
Remotes: target. The skew it warns about is not
hypothetical: a local 3.0.0 build sat in front of CRAN 3.0.1 here and
hid a new get_clean_url argument from the suite.