The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
?sbert now shows the package’s own title and
description instead of the pre-0.5.5 ones. R/package.R
hard-coded “Python-Free Sentence-BERT Inference” above
"_PACKAGE", which overrode the DESCRIPTION
fields that 0.5.5 retitled, so the package help page and the
DESCRIPTION disagreed.Language: en-US (modelled,
labelled, colour, neighbour,
normalised, centre). Stemming test fixtures
keep their British forms.@examples and descriptions of their own rather than a
repeat of the title.plot() on a topic hierarchy is titled “Topic
hierarchy”; it read “Topic topic_hierarchy”, a leftover from the 0.5.4
rename, as did the reduce_topics() description and the
under-two-topics error.Description now single-quotes
only software names. The six \dontrun{} examples are now
\donttest{} guarded by interactive() (they
download a model or install the runtime), and each of those Rd files
gains a small unwrapped example that runs offline.withr or
rmarkdown is missing, and clean up the files they create in
tempdir(). Language: en-US added._pkgdown.yml reference index, which
aborted every site build since 2026-09-28.topics()
golden-reference comparison and the
rowsum-vs-sweep/apply pooling equivalence —
now skip_on_cran(). They assert bit-for-bit equality
against a reference generated on one platform, which does not hold
across CRAN’s different OSes, BLAS builds, and C math libraries (the
last digit of a distance or centre can differ). They remain strict
same-platform regression guards for local runs and CI; the portable
guarantee (topics() is deterministic across reruns on a
given machine) keeps running everywhere.plot(type = "fit") no longer errors on a topic whose
terms carry a non-finite metric value. A degenerate topic can reach a
term panel with an NA/NaN score or beta, or an
Inf; base graphics then failed with an opaque
need finite 'ylim' values,
missing value where TRUE/FALSE needed, or
need finite 'xlim' values from deep inside
barplot(). The panel now drops the non-finite bars
(plotting the finite ones) and falls back to the same annotated empty
panel as the no-terms case when none remain. All-finite input is
untouched, so the normal picture is byte-identical.encode() assembles its result with lower peak memory:
instead of collecting every batch in a list and rbind-ing
them (which held the whole corpus of embeddings twice), it fills one
pre-allocated matrix batch by batch. Output is byte-identical, and the
saving flows through topics() and
topic_gamma() whenever they encode. Measured about 225 MB
lower peak assembling 120,000 embeddings at 768 dimensions.normalize_embedding_rows() no longer allocates a full
logical copy of the matrix to validate it (an O(n) range scan replaces
all(is.finite())) and normalizes in row blocks; the
farthest-point initializer and the per-document centroid cosine are
likewise computed in blocks rather than materializing corpus-sized
temporaries. All byte-identical. The dominant cost of
topics() on a large corpus remains
stats::kmeans itself, which holds the embedding matrix;
there a 384-dimension model roughly halves peak memory against a
768-dimension one, keep_embeddings = FALSE drops the stored
copy from the result, and clustering distinct sentences via
dedupe() shrinks the matrix directly.select_topics() is renamed
compare_topics(). The verb compares candidate topic counts
and hands the choice to fitted(); it never selected one, so
the old name overpromised. select_topics() remains as a
deprecated alias that forwards every argument and warns once per
session.topics(), compare_topics(), and
topic_corpus() gain a segment argument
("document", the default, or "sentence",
"clause", "phrase") that splits every document
with segment() before encoding and fits the topics on the
segments, so long documents are modelled in full instead of being
truncated to the encoder’s context window. The segment options travel
with it under their segment() names:
max_tokens (counted with the fitting model’s own tokenizer
when a model is used), merge_below, and
min_content. A precomputed embeddings matrix
must then have one row per segment; a mismatch reports the segment count
it expected. The fitted $documents has one row per segment
with its document_id, segment position, and
any data-frame metadata expanded to match, and $settings
records the segmentation so every later verb can reuse it.topic_sizes() gains by = "document" to count
source documents rather than segments; topic_gamma() needs
no text for a segmented model (the stored segments already
carry each document’s mixture) and, given new text, segments it exactly
as the fit was; predict() does the same and returns one row
per segment; representatives() on the fitted units now
carries document_id (and segment) so every
example can be traced; reduce_topics() keeps the segment
columns and carried metadata, refreshes the label columns,
and applies the fitted numbers,
roman_numerals, and section_numbers filters.
print() and summary() report documents and
segments separately.compare_topics() accepts a data frame with
column, like topics().compare_topics() compares segmentations as well as
counts: pass several levels,
segment = c("document", "sentence", "clause", "phrase"),
and every level is fitted across every candidate. The table gains a
leading segment column, plot() draws one line
per level (colour, point shape, and line type, with a legend), the print
hint names both, and fitted() takes segment =
alongside n_topics. Precomputed embeddings are then a list
of matrices named by level. The segment-only options apply to the
segmented levels and are dropped for "document".plot() of a comparison labels its middle panel “Topic
diversity” (was “Topic topic_diversity”).plot(comparison, type = "fit", n_topics = k) draws the
per-topic fit report for the retained k-topic model of
every segment level, one figure per level with the level in its title,
so the same count can be read across segmentations.
plot.sbert_topic_model() gains main for
type = "fit".plot(type = "terms"),
plot(type = "fit"), and the per-topic figures). Such a
topic now draws an empty, labelled panel. A device too small to stack
one row per topic in plot(type = "fit") raises a classed
sbert_plot_too_small error naming the remedies
(per_topic = TRUE, a topics = subset, or a
taller device) instead of the opaque base-graphics “invalid graphics
state” message.Added clean_corpus(), one place for the text-level
cleaning that should happen before encoding, so the same clean text
feeds the encoder, the topic terms, and the representatives. It repairs
junk and strips non-content markup — HTML entities and tags, URLs, DOIs,
email addresses, bracketed numeric citations ([12]), page
references (p. 1, pp. 15-30), curly quotes,
dashes, bullets, control, zero-width, BOM and replacement characters,
and non-breaking spaces — then strips list markers (1.,
(i), (a), (109), and bare
Roman-numeral section markers such as IV. and a leading
V.) and section/reference numbering (1.2.3,
and alphanumeric indices such as II.2 and
AA.2), applies any custom remove patterns (the
extension point for domain-specific noise the generic cleaners leave,
for example remove = "OJ No L?\\s*\\d+"), optionally
removes numeric and Roman-numeral tokens, and drops documents below a
min_content alphabetic-density floor — citation lists,
reference footnotes, number tables — carrying a data frame’s other
columns along so metadata stays aligned. Supporting helpers
strip_list_markers() and content_ratio() are
exported too. content_ratio() is the fraction of a string’s
non-space characters that are letters, a domain-agnostic measure of
prose versus reference noise (real sentences score near 1; “OJ No L 297,
24.11.1979, p. 1.” scores about 0.25), and min_content
judges it on the repaired text before the number strips so a stripped
reference cannot slip through. Because Sentence-BERT tolerates a little
noise, this targets broken text and non-content rows rather than
scrubbing every token.
The numbers, roman_numerals, and
section_numbers term filters stay on topics(),
select_topics(), topic_corpus(), and
terms(): they tidy the topic labels without
touching the document text or its embedding, which is the point of
keeping them separate from clean_corpus(). Text-level
cleaning that would change the embedding lives in
clean_corpus().
representatives() and a fitted model’s built-in
representatives now return distinct texts. A representative list of the
same string repeated is never useful, and duplicates are common once
documents are segmented.
segment() splits a run-together sentence boundary —
a lower-case letter, a period, then an upper-case letter, as in
“results.The next” — rather than treating it as an acronym; genuine
acronyms (“U.S.”) and decimals (“3.14”) are left intact.
topic_gamma() now accepts either raw documents
(segmented internally, as before) or the data frame returned by
segment(), used as-is. Passing your own segmentation keeps
every segment option — level, max_tokens,
merge_below — in segment() and means a
supplied embeddings matrix always lines up with the
segments, instead of having to match an internal re-segmentation. The
two paths give identical gamma when the segmentation
matches.
segment() gains max_tokens, a cap so no
segment overruns an encoder’s context window and is silently truncated.
An over-long segment is re-split at the finest logical boundaries —
clause hinges, ;, :, -, and
commas — and the pieces are packed back up to the budget, so splits land
on punctuation wherever possible. A run with no such boundary is chopped
further, but even then the break is placed just before a function word
(a coordinator, preposition, article, or relative pronoun) near the
budget edge rather than mid-phrase — so “the schools” is kept together
instead of stranding “the”. By default the budget counts whitespace
words (offline, deterministic). Passing a loaded model
counts that model’s exact sub-word tokens instead, so
max_tokens can be the model’s real limit; token-counted
segmentation runs serially since the tokenizer is not forked.
NULL (default) leaves segments uncapped, byte-identical to
before.
encode() gains a cache argument: a path
to a content-addressed embedding store. Each document is keyed by a
SHA-256 digest of its own text together with the model identity,
revision, dimension, maximum length, pooling method and
normalize setting, so only documents with no matching entry
are encoded and the file is updated with whatever was newly computed.
Corpora are usually edited rather than replaced, and encoding is about
97% of the cost of a topic workflow, so reuse dominates: re-running an
unchanged 3,847-abstract corpus cost 0.07 s against 36.59 s
(515x, output identical()), and re-running
it after editing 38 documents cost 0.41 s (89x) with
unchanged rows identical to the previous run and changed rows identical
to a fresh encode. Reordering or subsetting a cached corpus encodes
nothing at all. Duplicate documents within one call are encoded once and
expanded by position — 295 of the 3,847 bundled covid
abstracts are duplicates.
The key deliberately excludes batch_size and
sort_by_length, which move embeddings by around 1e-7
through batch composition alone; cached values are therefore stable to
whatever computed them first rather than bit-identical to a fresh encode
under different batching. A cache that cannot be read, or was written
for a different embedding dimension, is discarded with a warning and
rebuilt rather than trusted, and writes go to a temporary file and are
renamed so an interrupted save cannot leave a half-written cache
behind.
load_model() accepts threads = "auto",
which asks the platform for its performance-core count instead of its
logical one. ONNX inference is about 88% of a batch and the only
threaded part of the package, but it scales with performance cores only.
Measured over three repetitions at 200, 600 and 1,500 documents on an
Apple M4 (4 performance + 6 efficiency), median speedups against a
single thread were 1.22x at two threads, 1.29x at four,
1.22x at six and 1.18x at eight; four threads was fastest at both larger
sizes (1.34x at 1,500 documents), after which the efficiency cores
contribute contention rather than throughput. "auto"
selects that knee and is capped at 8 so a library never seizes a whole
machine uninvited. Raising the thread count has no numerical
consequence: output was identical() in all 45
runs. The default stays 1 so results remain reproducible
across machines and within the two-core limit that check environments
impose. Note that threaded timings are noisy — run-to-run spread reached
26% at the smallest size against under 3% single-threaded — so a single
timing run is not sufficient evidence for tuning.
Mask-aware mean pooling is about 6.3x faster, and
encode() about 1.18x faster end to end, with identical
output on platforms whose sum() and rowsum()
accumulate in the same precision. pool() built a second 24
MB copy of every batch with sweep() and then summed it with
apply(., c(1, 3), sum), an R-level loop over a 3-D array.
Because an R array is column-major and its first two margins are
adjacent, the batch re-dimensions to
(batch * sequence) x hidden without moving any data, so
masking becomes one recycled multiply and the per-document sum becomes a
single C-level rowsum(). Verified as
identical() against the previous formulation across 60,000+
shape, mask, method and normalization combinations and on real ONNX
output.
One caveat, and it is a real one: R’s sum() accumulates
into a long double while rowsum() accumulates into a
double. On builds with extended-precision long doubles — common x86-64
Linux and Windows builds, where .Machine$sizeof.longdouble
is 16 — the two can therefore differ in the last bits on inputs with
heavy cancellation. Equality was confirmed on arm64 macOS, where
sizeof.longdouble is 8 and the question does not arise; on
extended-precision builds expect agreement to around 1e-16 rather than
to the bit. Embedding values of that magnitude do not change any topic
assignment.
The element-scanning part of pool()’s validation is now
skipped on the internal encoding path, where the mask has already been
validated by the tokenizer step. The structural checks are kept
— dimension agreement is O(1) and its removal would have let a wrongly
shaped ONNX output recycle silently into the wrong number of rows.
Finiteness is now checked on the pooled result rather than the raw
batch: the same guarantee at roughly 1/256 the cost, since any
non-finite token value must surface there (an unmasked Inf
sums to Inf, a masked one becomes
Inf * 0 = NaN). pool() itself is unchanged for
direct callers and still validates in full.
encode() and topic_gamma() gain
sort_by_length (default FALSE). Every sequence
in a batch is padded to the longest member, so a batch costs the model
its maximum length rather than its mean. Grouping inputs of similar
length into the same batch shrinks that padding: at clause level 60.1%
of all token work is padding in input order against 1.6% when sorted,
measured at 1.89x faster for topic_gamma(level = "clause")
and 1.08x for whole documents, which vary less in length. The
permutation is inverted before returning, so row order always matches
the input. Off by default because regrouping changes batch composition,
which moves embeddings by about 1e-7 — the same trade-off already
documented for dedupe_segments. On the bundled
covid corpus that drift changed no gamma value
and no document’s dominant topic, but it is opt-in rather than
assumed.
topics(), select_topics(),
topic_corpus(), and terms() gain
roman_numerals and section_numbers, two
independent token filters alongside numbers.
roman_numerals = "remove" drops chapter/section/list
markers written as Roman numerals (ii, iv,
xii), bounded at 100 so that common abbreviations that are
also valid numerals — ml, mm, cc,
ci, cv — are always kept.
section_numbers = "remove" strips both multi-level section
and reference indices (1.2.3, 4.5.6.7) and
enumeration or list markers (1., 2.,
figure 12. — a standalone one- or two-digit number followed
by a period) before tokenizing; decimals, four-digit years
(2020.), hyphenated numbers (covid-19.), and
larger counts survive, so genuine values are kept even under
numbers = "keep". Both default to "keep"
(byte-identical to before) and are recorded on the model so
coherence() stays consistent.
topics(), select_topics(), and
topic_corpus() gain a numbers argument.
"keep" (default, unchanged) keeps numeric tokens;
"remove" drops tokens made only of digits — years, counts —
from topic terms while retaining alphanumerics such as
covid19. The setting is recorded on the model so
coherence() stays consistent. Custom stop words already
compose through the stop_words argument of these verbs and
keywords(); stop_words(add =) keeps the
defaults plus your terms, a bare vector replaces the list, and
character() disables filtering. Documented with
examples.
The topic tokenizer is about 1.5x faster: a single-pass base-R
strsplit split replaces the previous gregexpr
+ regmatches pair. Output is byte-identical (same token
grammar, verified across Unicode, apostrophe, and empty-document edge
cases) and no dependency is added. This speeds up topics(),
select_topics(), topic_corpus(), and
coherence(), and stacks with the cores
argument.
segment() and topic_gamma() gain a
cores argument. Splitting documents into sentences or
clauses is the dominant non-encoding cost of sentence-level
topic_gamma(), and it is per-document independent, so
forking it across cores gives a several-fold speed-up (about 4-5x on
eight cores) with byte-identical output. Encoding is parallelized
separately via load_model(threads =).
topic_gamma() gains dedupe_segments
(default FALSE). Real corpora often repeat many segments
(boilerplate sentences, short clauses); enabling this encodes each
distinct segment once and expands by position, roughly halving the
encoding cost on such corpora. Off by default because encoding a smaller
batched set can shift a rare borderline assignment at the ~1e-7 level;
with the static potion-base-8M model the result is
identical.
topics(), select_topics(),
topic_corpus(), and coherence() gain a
cores argument for parallel tokenization;
select_topics() also fits its independent candidates in
parallel. Because document tokenization and per-candidate fitting are
deterministic and order-preserving, results are byte-identical for any
core count — cores changes only speed, never output.
Parallelism uses parallel::mclapply on Unix-alikes and
falls back to serial on Windows and for small corpora; the default is
cores = 1. On an 8-core machine a five-count sweep on
15,000 documents drops from about 12 s to 7 s.
Added topic_corpus(): prepare a corpus once —
embedding every document and tokenizing it for term scoring — and reuse
it across many models. topics() and
select_topics() accept the prepared corpus in place of raw
text and skip both the encoding and the tokenization, which previously
repeated on every call. A sweep or hand-written loop over topic counts
now costs one tokenization (and one encoding) plus the per-model
clustering, rather than repeating the whole corpus pass per model; the
speedup grows with the number of models fitted. Results are
byte-identical to preparing the corpus inside each call, and
select_topics() now shares one prepared corpus across all
candidates internally.
select_topics() no longer re-tokenizes the corpus
inside coherence() on every candidate — the dominant cost
of a sweep. coherence() gains an optional
token_lists argument so the shared corpus tokenization is
reused; scores are byte-identical. Combined with the corpus reuse above,
a five-count sweep on 15,000 documents dropped from about 164 s to 12 s
(~13x) with no change to any result.
topics() (and select_topics(), which
calls it per candidate) no longer exhausts memory or stalls on large
corpora. The farthest-point centroid initializer was rewritten from an
O(n_topics^2 * n * d) recompute-everything pass — which repeatedly
allocated full document-by-dimension matrices and could crash the R
session — to an incremental O(n_topics * n * d) pass that keeps a
running nearest-centroid distance (about 24x faster at 20k documents,
with a fraction of the peak memory). The expensive up-front distinct-row
dedup was dropped in favour of an exact in-loop check, and per-document
centroid cosines are now vectorized. Topic assignments, terms, and
representatives are bit-identical to previous versions.
Added the covid dataset: 4,170 COVID-19 research
abstracts (2020-2024) on education, children, schools, and society, each
with a publication year. Editorial notices (retractions, corrections,
errata) are removed during preparation; 323 records carry a
"[No abstract available]" placeholder and 3,671 abstracts
are distinct. A companion pkgdown article, “Topic Modeling COVID-19
Research Abstracts”, walks the full sweep-fit-read workflow.
Added plot(topic_model, type = "representatives"), a
per-topic ranked text list of the centroid-nearest documents (with their
cosine similarity to the centroid), an n_representatives
argument, and type = "fit", a per-topic report placing all
three term views (count, TF-IDF, beta) alongside the representative
documents, one row per topic. Term bar panels now annotate each bar with
its value.
plot(type = "terms") gains a by
argument selecting one or more of "score" (class-based
TF-IDF, the default), "beta" (generative word probability),
and "frequency" (within-topic count); several metrics draw
one row per topic with a column each. A new topics argument
restricts "terms", "representatives", and
"fit" to chosen topic numbers, and
per_topic = TRUE draws a separate figure for each topic
instead of one gridded figure.
README updated for the 0.5 API: data-frame input to
topics(), the retained-model sweep with
fitted(), and terms() retuning without a
refit. Adds a pointer to the tutorial vignette.
Repaired README prose damaged by the 0.5.0 rename, where the English words “membership” and “diversity” had been rewritten as function names.
Corrected the model count in the README from thirteen to fourteen.
Named the n_topics argument in the README’s
reduce_topics() example, and stopped shadowing
topic_hierarchy() with a variable of the same
name.
R CMD check --as-cran is clean: 0 errors, 0 warnings, 0
notes.column argument of
topics().\eqn macro in the
terms() documentation.graphics::par() in the topic-hierarchy plot
method.withr in Suggests; the session tests use
it..DS_Store, TODO.md and knitr
caches from the package build.sbert_ prefix is goneEvery exported function lost its prefix. The package has never been released, so the old names were removed outright rather than deprecated:
sbert_topics() is now topics(),
sbert_encode() encode(),
sbert_segment() segment(),
sbert_blend() blend(),
sbert_keywords() keywords(),
sbert_dedupe() dedupe(),
sbert_coherence() coherence(),
sbert_select_topics() select_topics(),
sbert_representatives() representatives(),
sbert_load_model() load_model(),
sbert_models() models(), and so on for the
cache and runtime verbs.
Six names are qualified because the bare form collides with a package
this audience is likely to have attached — igraph,
quanteda, purrr:
sbert_membership() ->
topic_membership()sbert_diversity() ->
topic_diversity()sbert_hierarchy() ->
topic_hierarchy()sbert_similarity() ->
topic_similarity()sbert_reduce() -> reduce_topics()sbert_stopwords() -> stop_words()Two are qualified because the bare form would mask base R:
sbert_gamma() -> topic_gamma(),
sbert_palette() -> topic_palette().
Class names keep the prefix (sbert_topic_model,
sbert_model), so S3 methods read
terms.sbert_topic_model().
terms()
retunes without refittingTopic terms depend only on the fitted assignments and the document text, so every term setting can now be changed without repeating the clustering or the encoding:
terms(fit, n = 12, weighting = "bm25", stem = TRUE)
terms(fit, n = 8, sort_by = "beta") # words a topic uses most
terms(fit, n = 8) # words that distinguish itsort_by chooses between the class-based weight
(distinctive terms, the default) and raw p(term | topic).
n is applied after ordering. The returned frame carries
beta, which absorbs the removed
sbert_beta().
topics() accepts a data frame plus
column = naming the text column. Remaining columns ride
along into $documents. Rows whose text is missing, blank,
or a [NO ABSTRACT AVAILABLE] placeholder are dropped once,
and any supplied embeddings are subset to match.keep_embeddings now defaults to TRUE.
topic_membership(), reduce_topics() and
plot(type = "map") previously failed on a freshly fitted
model and told the user to refit — which meant re-encoding.label is carried into $documents,
$terms and $representatives, so joining it
back by hand is never necessary.representatives(fit) defaults to the fitted corpus and
its stored embeddings.select_topics()
keeps every model it fitsThe sweep no longer discards the models it built.
fitted(sweep, n_topics =) returns one without refitting,
and the result gains print, plot and
as.data.frame methods while remaining a plain data frame
with the same columns.
feedback_translations data with
precomputed embeddings, so it builds without any download.sbert_beta() (now the beta column
of terms()).model object.potion-base-8M (Model2Vec static
embeddings, MIT): encoding is a token-lookup and mean in pure base R —
no ONNX session — which makes it roughly two orders of magnitude faster
than transformer models (10,000 sentences in under a second) at a modest
quality cost. Loaded through the same verbs
(sbert_model_download(),
sbert_load_model("potion-base-8M"),
sbert_encode()); the safetensors weight matrix is read with
a minimal base-R reader and verified by SHA-256 like every other
artifact. sbert_models(detail = TRUE) gains an
engine column (“onnx” or “static”). Numerical parity with
Python model2vec: max abs diff 3.3e-08.sbert_topics() gains seeds,
seed_embeddings, and fixed_seeds: supply seed
words or topic descriptions and they become the first cluster centroids.
Seeded topics keep their position (topic i is seed i, never reordered by
size) and take the seed’s name as their label when seeds is
named. Remaining topics up to n_topics are initialized
deterministically away from the seeds. With
fixed_seeds = TRUE the centroids never move — documents are
zero-shot assigned to the nearest seed, and seeded topics may
legitimately be empty.sbert_keywords(): embedding-based keyword and
phrase extraction (the KeyBERT design) — each document’s own unigrams
and n-grams are embedded with the same model, ranked by cosine
similarity to the document, and selected by maximal marginal relevance
(diversity argument) so keywords are relevant without being
redundant.sbert_stopwords() gains add and
remove: exclude a corpus-wide vocabulary from topic terms
and keyword candidates, or un-list built-in entries.sbert_select_topics(): fits the model across
candidate topic counts on one shared embedding matrix and returns
coherence, diversity, and between-topic variance share per count —
granularity chosen by numbers, not habit.sbert_hierarchy(): deterministic agglomerative
merge tree of the topic centroids (cosine distance) with a tidy merge
table, labeled dendrogram plot, and readable print.sbert_reduce(): cuts the hierarchy at a target
count and rebuilds a full sbert_topic_model (merged
assignments, recomputed centroids, terms, labels, sizes,
representatives) — every downstream verb works on the reduced model
unchanged.sbert_blend(): context-aware embeddings for
segments — each segment keeps alpha of its
context-orthogonal residual and inherits the rest of its direction from
the parent document embedding, renormalized. Blending the residual
(rather than the raw segment vector) avoids the near-collinearity of a
segment with its own document. Accepts a sbert_segment()
data frame plus the original documents (encoding both), or precomputed
embedding matrices. Output drops into sbert_topics(),
sbert_representatives(), and predict()
unchanged. This is the package’s supported way to “mix” granularities:
per unit, never by pooling units of different levels into one training
set.sbert_representatives(): evidence selection by
distinctiveness margin (distance to the second-best centroid minus
distance to the own centroid) instead of raw closeness, which
systematically favors long units. rank = "distance"
restores the old behavior."ETC." removed from the default abbreviation gazetteer:
sentence boundaries only fire at period-space-word, and mid-sentence
“etc.” always carries trailing punctuation, so protecting it could only
glue true sentence ends together.Added six modern embedding models to the registry:
bge-small-en-v1.5 and bge-base-en-v1.5 (BAAI,
CLS pooling), multilingual-e5-small (100+ languages,
automatic "query: " prefix),
nomic-embed-text-v1.5 and
jina-embeddings-v2-small-en (8,192-token context), and
mxbai-embed-large-v1 (1,024 dimensions). The encoder now
supports CLS pooling (sbert_pool(method = "cls")) and
per-model input prefixes (sbert_encode(prefix = ...)
overrides the pinned default). Parity vs Python: at or below 2.9e-7 for
the BGE/E5/nomic models, 8e-8 cross-runtime for jina; mxbai agrees to
cosine 0.9996 with its float16-distributed reference (the pinned ONNX is
the full-precision float32 artifact).
Added a registry of pinned, SHA-256-verified models. Besides the
default all-MiniLM-L6-v2, the package now supports six
further models: all-MiniLM-L12-v2 (deeper English MiniLM),
paraphrase-MiniLM-L3-v2 (the smallest and fastest, 69 MB),
multi-qa-MiniLM-L6-cos-v1 (semantic search, 512-token
inputs), paraphrase-multilingual-MiniLM-L12-v2 (384
dimensions, 50+ languages), all-mpnet-base-v2 (768
dimensions, higher-quality English), and
paraphrase-multilingual-mpnet-base-v2 (768 dimensions, 50+
languages). sbert_models() lists them tidily. Every model
remains locked to an immutable revision with byte-size and SHA-256
verification, and every MiniLM addition was verified numerically
identical to Python SentenceTransformers output (max
difference below 3e-7).
sbert_model_download() and
sbert_load_model() now take the model name as their first
argument (sbert_load_model("all-mpnet-base-v2"));
sbert_model_status() and sbert_model_remove()
gained a trailing model argument. Calls that passed
cache_dir positionally to
sbert_model_download()/sbert_load_model() must
now name it.
The encoder adapts to each model’s input signature: MPNet- and
XLM-RoBERTa-based graphs receive two ONNX inputs (no
token_type_ids) and are padded with their own pad token;
every model uses its published maximum sequence length.
sbert_load_custom(): loads any public Hugging
Face repository that ships an ONNX encoder export and a
tokenizer.json, with trust-on-first-use pinning (revision,
byte sizes, and SHA-256 recorded in a local manifest on first download
and verified on every later load). Pooling, maximum length, and the pad
token are auto-detected from the repository’s Sentence-Transformers
configuration and tokenizer; the input signature and embedding dimension
are read from the ONNX graph itself. Returns a regular
sbert_model, so every downstream verb works unchanged.
Requires the jsonlite package (Suggests). Unlike registry
models, no parity certificate is implied.sbert_encode(text) uses the default
all-MiniLM-L6-v2,
sbert_encode(text, model = "bge-small-en-v1.5") switches by
name, and sbert_topics(), predict(), and
sbert_gamma() resolve models the same way. Loaded models
are kept in a session cache and reused.sbert_model_download() or
options(sbert.download = TRUE).sbert_models()
menu pointer.sbert_dedupe(): collapses a corpus to its
distinct non-blank texts (first-appearance order) with occurrence
counts, so repeated templates cannot dominate the clustering geometry
while frequencies remain available as weights.sbert_topic_sizes(): topic sizes on the distinct
and weighted scales in one call; the gap between proportion
and weighted_share measures how template-driven each topic
is.predict() for sbert_topic_model:
assigns unseen documents to the nearest fitted centroid under cosine
distance, returning a tidy data frame with topics, labels, and
distances.sbert_membership(): fuzzy-c-means soft topic
probabilities on the stored centroids with a sharpness
fuzzifier (default 1.15; the textbook value 2 degenerates toward uniform
membership in high-dimensional embedding spaces). The within-document
topic ranking is sharpness-invariant.sbert_beta(): the generative multinomial
p(term | topic) from within-topic token counts with
optional Laplace smoothing — the frequent- words view that complements
(and must not be confused with) the discriminative class-based TF-IDF
terms.sbert_gamma(): per-document topic distributions
obtained by segmenting each document with sbert_segment()
and assigning each segment to its nearest centroid — parameter-free
mixed membership.feedback_translations dataset: 8,757
multilingual AI-generated mathematics feedback messages from the Levebee
educational application paired with their English translations (8,005
distinct translations, template repetition preserved), giving every
example and vignette a realistic offline corpus for the embedding,
segmentation, and topic-modeling workflow.sbert_segment(), deterministic rule-based text
segmentation at sentence, clause (default), or phrase granularity.
Sentence boundaries are guarded by a word-boundary-anchored,
case-insensitive abbreviation gazetteer
(sbert_abbreviations()), a decimal-number guard, and a
parenthetical guard; clause level adds splits at semicolons, colons,
spaced dashes, and subordinating hinges while keeping comma enumerations
whole. Returns a tidy one-row-per-segment data frame and preserves
letter case. Promoted from the benchmarked analysis module (F1 0.999 on
realistic abbreviation-rich abstracts versus 0.80-0.82 for ICU and NLTK
Punkt references).it's
and it’s are one token.sbert_topics()
(stem = TRUE, using SnowballC) that collapses
inflected forms while displaying the most frequent surface form of each
stem, giving cleaner, deduplicated topic terms.sbert_coherence() (UMass and NPMI) and
sbert_diversity(), with a summary() method
that reports cluster separation, coherence, and diversity alongside a
tidy per-topic table.plot() method for topic models
(type = "sizes", "terms", "map")
and the sbert_palette() colour helper. The document map
uses classical MDS on cosine distance.weighting = "bm25" and
reduce_frequent_words options to
sbert_topics(), implementing the BM25 and square-root
class-based TF-IDF variants of Mendonca and Figueira (2025).all-MiniLM-L6-v2 inference through
tok and onnxr.These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.