The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Using textrecipes as a sparse matrix engine

library(textrecipes)
library(recipes)

This vignette walks through getting a document-term matrix, such as one to hand to glmnet, xgboost, a clustering routine, or your own code, straight out of a recipe. Build a recipe, prep() it on your training text, then bake() with composition = "dgCMatrix".

Creating a sparse matrix

We will use a handful of short documents.

train <- tibble::tibble(
  text = c(
    "the cat sat on the mat",
    "the dog chased the cat",
    "dogs and cats are pets",
    "a mat is a small rug"
  )
)

new_docs <- tibble::tibble(
  text = c("the cat chased a small dog", "a rug for the mat")
)

Set sparse = "yes" in the step to get sparse output.

rec <- recipe(~text, data = train) |>
  step_tokenize(text) |>
  step_tfidf(text, sparse = "yes") |>
  prep()

Baking the training data with composition = "dgCMatrix" returns a sparse matrix from the Matrix package, one row per document and one column per token.

x_train <- bake(rec, new_data = NULL, composition = "dgCMatrix")
x_train
#> 4 x 16 sparse Matrix of class "dgCMatrix"
#>   [[ suppressing 16 column names 'tfidf_text_a', 'tfidf_text_and', 'tfidf_text_are' ... ]]
#>                                                                           
#> [1,] .         .         .         0.1831020 .         .         .        
#> [2,] .         .         .         0.2197225 .         0.3218876 0.3218876
#> [3,] .         0.3218876 0.3218876 .         0.3218876 .         .        
#> [4,] 0.5364793 .         .         .         .         .         .        
#>                                                                          
#> [1,] .         .         0.183102 0.2682397 .         .         0.2682397
#> [2,] .         .         .        .         .         .         .        
#> [3,] 0.3218876 .         .        .         0.3218876 .         .        
#> [4,] .         0.2682397 0.183102 .         .         0.2682397 .        
#>                         
#> [1,] .         0.3662041
#> [2,] .         0.4394449
#> [3,] .         .        
#> [4,] 0.2682397 .

The prepped recipe remembers the vocabulary and the inverse document frequencies learned from the training data, so new documents are mapped onto the same columns.

x_new <- bake(rec, new_data = new_docs, composition = "dgCMatrix")
x_new
#> 2 x 16 sparse Matrix of class "dgCMatrix"
#>   [[ suppressing 16 column names 'tfidf_text_a', 'tfidf_text_and', 'tfidf_text_are' ... ]]
#>                                                                                
#> [1,] 0.2682397 . . 0.183102 . 0.2682397 0.2682397 . . .         . . .         .
#> [2,] 0.4023595 . . .        . .         .         . . 0.2746531 . . 0.4023595 .
#>                         
#> [1,] 0.2682397 0.1831020
#> [2,] .         0.2746531

identical(colnames(x_train), colnames(x_new))
#> [1] TRUE

Tokens that weren’t seen during prep() are dropped, so the columns always line up between the training and new data.

The result is a regular dgCMatrix, which is the format that many R packages take as input.

# Fit a lasso model with glmnet
fit <- glmnet::glmnet(x_train, y_train, family = "binomial")
predict(fit, newx = x_new)

Steps that produce sparse data

These steps have a sparse argument, which needs to be set to "yes" to get sparse output:

Controlling the number of columns

A vocabulary can grow very large, so you will often want to put a cap on the number of columns. There are two options.

Use step_tokenfilter() to keep only the most frequent tokens.

recipe(~text, data = large_corpus) |>
  step_tokenize(text) |>
  step_tokenfilter(text, max_tokens = 5000) |>
  step_tf(text, sparse = "yes") |>
  prep() |>
  bake(new_data = NULL, composition = "dgCMatrix")

Or use feature hashing with step_texthash(). The number of columns is fixed up front with num_terms, no matter how large the vocabulary is, and no vocabulary has to be stored. This makes it a good match for sparse output when the vocabulary is too big to hold in memory.

recipe(~text, data = large_corpus) |>
  step_tokenize(text) |>
  step_texthash(text, num_terms = 4096, sparse = "yes") |>
  prep() |>
  bake(new_data = NULL, composition = "dgCMatrix")

step_dummy_hash() does the same for nominal predictors with many levels.

Using workflows instead

If you are fitting a model with tidymodels, you don’t need to set sparse yourself. A workflow turns it on for models that support sparse data. See ?sparse_data in the recipes package for details.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.