Introduction to Vector Comprehension

List Comprehension

Let us start with a simple example: we have a vector of fruit names and want to create a sequence for each based on the number of characters. One common method is to write a for statement where we loop across each fruit name, create a sequence from the numbers, and place it in a list.

x <- c("apples", "bananas", "pears")

fruit_chars <- vector("list", length(x))
for (i in seq_along(x)){
  fruit_chars[[i]] <- 1:nchar(x[i])
}
fruit_chars
#> [[1]]
#> [1] 1 2 3 4 5 6
#> 
#> [[2]]
#> [1] 1 2 3 4 5 6 7
#> 
#> [[3]]
#> [1] 1 2 3 4 5

Alternatively, we could use a faster, vectorized function such as lapply. This is similar to the for loop, except that a specified function is applied to each fruit name and the results are wrapped in a list.

fruit_chars <- lapply(x, function(i) 1:nchar(i))

The eList package allows for a hybrid variant of the two using the List() function. The for loop evaluates the expression and merges the results into a list. Beneath the scenes, List is parsing the statement into lapply and evaluating it. As we will see, though, there are additional features in List.

library(eList)
#> 
#> Attaching package: 'eList'
#> The following object is masked from 'package:utils':
#> 
#>     zip
fruit_chars <- List(for (i in x) 1:nchar(i))

If you are coming from Python or another language that uses list comprehension, you will notice that this syntax is a bit different; it is written more similar to standard for loops in R. The for statement comes first, then the variable name and sequence are placed in parentheses. Finally the expression is placed afterwards. The expression can be wrapped in braces {} if desired.

Assigning Names

The lists of numbers in the previous examples may be confusing. Luckily, List and other comprehension functions allow us to assign names to each result using the notation: name = expr.

List(for (i in x) i = 1:nchar(i))
#> $apples
#> [1] 1 2 3 4 5 6
#> 
#> $bananas
#> [1] 1 2 3 4 5 6 7
#> 
#> $pears
#> [1] 1 2 3 4 5

The name can be any type of expression. Though complex expressions should be wrapped in parentheses so that the parser does not confuse who has the = sign.

List(for (i in x) (if (i == "bananas") "good" else "bad") = 1:nchar(i))
#> $bad
#> [1] 1 2 3 4 5 6
#> 
#> $good
#> [1] 1 2 3 4 5 6 7
#> 
#> $bad
#> [1] 1 2 3 4 5

Conditions - if-else

The results of a comprehension can be filtered using standard if statements after naming the variables and sequence, making sure that the condition is placed in parentheses. The statement below only returns the sequence for "bananas".

List(for (i in x) if (i == "bananas") 1:nchar(i))
#> [[1]]
#> [1] 1 2 3 4 5 6 7

Any statement that evaluates to NULL is automatically filtered from the results. else statements can also be included so that the results are not filtered out (unless the else statement evaluates to NULL).

List(for (i in x) if (i == "bananas") "delicious" else "ewww")
#> [[1]]
#> [1] "ewww"
#> 
#> [[2]]
#> [1] "delicious"
#> 
#> [[3]]
#> [1] "ewww"

Each entry can still be assigned a name. Furthermore, the expression can be as complex as necessary for the task with any number of else if checks.

List(for (i in x) i = {
  n <- nchar(i)
  if (n > 6) "delicious"
  else if (n > 5) "ok"
  else "ewww"
})
#> $apples
#> [1] "ok"
#> 
#> $bananas
#> [1] "delicious"
#> 
#> $pears
#> [1] "ewww"

Multiple Variables

Comprehensions can have multiple variables if the variables are separated using "." within a single name. To see how this works, let us use the enum function in the eList package. When enum is used on a variable, the first value becomes its index in the loop and the second is the value of the vector at that index. Now, when we use (i.j in enum(x)), i = the index number of each item in x, while j = the value of the item in x (the name of the fruit).

List(for (i.j in enum(x)) j = i)
#> $apples
#> [1] 1
#> 
#> $bananas
#> [1] 2
#> 
#> $pears
#> [1] 3

Let us inspect enum(x) to see what is going on beneath the surface. enum took the vector x and created a new list at each index. The first list contains two elements: the first being 1 (the index number), the second being "apples" (the first value in x).

enum(x)
#> [[1]]
#> [[1]][[1]]
#> [1] 1
#> 
#> [[1]][[2]]
#> [1] "apples"
#> 
#> 
#> [[2]]
#> [[2]][[1]]
#> [1] 2
#> 
#> [[2]][[2]]
#> [1] "bananas"
#> 
#> 
#> [[3]]
#> [[3]][[1]]
#> [1] 3
#> 
#> [[3]][[2]]
#> [1] "pears"

When multiple variables like i.j are passed to the for loop, then i is assigned to the first element and j to the second element for item in the sequence. There can be any number of variables as long as they are separated by ".". There does not have to be a variable for each element; but there does need to be an element for each item or else an “out-of-bounds” error will be produced.

y <- list(a = 1:3, b = 4:6, c = 7:9)
List(for (i.j.k in y) (i+j)/k)
#> $a
#> [1] 1
#> 
#> $b
#> [1] 1.5
#> 
#> $c
#> [1] 1.666667

Similarly, variables can be skipped. The function below extracts the first and third elements of each item in the list.

List(for (i..j in y) c(i, j))
#> $a
#> [1] 1 3
#> 
#> $b
#> [1] 4 6
#> 
#> $c
#> [1] 7 9

If only the first or second item is needed, though, you should use two dots: i.. for the first, ..i for the second.

Another convenient function that can be used on the sequence is items(). It is similar to enum, except that the name of each item is used instead of its index number. See the documentation for eList for other helper functions.

List(for (i.j in items(y)) paste0(i, j))
#> [[1]]
#> [1] "a1" "a2" "a3"
#> 
#> [[2]]
#> [1] "b4" "b5" "b6"
#> 
#> [[3]]
#> [1] "c7" "c8" "c9"

Variables can also be separated using a comma as long as the variables are surrounded in backticks.

z1 <- 1:3
z2 <- 4:6
List(for (`i, j` in zip(z1, z2)) i + j)
#> [[1]]
#> [1] 5
#> 
#> [[2]]
#> [1] 7
#> 
#> [[3]]
#> [1] 9

Other Return Types

All of the examples so far have returned a list. However, eList supports a variety of different types of comprehension. For example, Num returns a numeric vector, while Chr returns a character vector. Env can be used to produce an environment, similar to dictionary comprehensions in Python. Note that each entry in an environment must have a unique name. These work by coercing the result into particular type, producing an error if unable.

Num(for (i.j.k in y) (i+j)/k)
#>        a        b        c 
#> 1.000000 1.500000 1.666667
Chr(for (i.j.k in y) (i+j)/k)
#>                  a                  b                  c 
#>                "1"              "1.5" "1.66666666666667"

One convenience function is ... It can be used as either ..[for ...] or ..(for ...). By default, it attempts to simplify the results in an array, but can mimic any other type of comprehension.

..[for (i.j.k in y) (i+j)/k]
#> [1] 1.000000 1.500000 1.666667

Nested Loops

Multiple for loops can be used within a comprehension. As long the subsequent for statements immediately follow the variables & sequence of the previous one, or immediately follow a if-else statement, then it will be parsed into a vectorized lapply style function. Nested loops should be avoided unless necessary since they can be difficult to understand. The following are a couple examples using matrix and numeric comprehension.

Mat(for (i in 1:3) for (j in 1:6) i*j)
#>      [,1] [,2] [,3]
#> [1,]    1    2    3
#> [2,]    2    4    6
#> [3,]    3    6    9
#> [4,]    4    8   12
#> [5,]    5   10   15
#> [6,]    6   12   18

Num(for (i in 1:3) for (j in 1:6) if (i==j) i*j)
#> [1] 1 4 9

Vector Comprehensions can also be nested within each other. The code below nests a numeric comprehension within a character comprehension.

Chr(for (i in Num(for (j in 1:3) j^2)) paste0(letters[sqrt(i)], i))
#> [1] "a1" "b4" "c9"

Parallelization

Vector comprehensions also for parallel computations using the parallel package. All comprehensions allow the user to supply a cluster, which activates the parallel version of sapply or lapply. eList comes with a function that allows for the quick creation of a cluster based on the user’s operating system and by auto-detecting the number of cores available. Users are recommended to explicitly create a cluster though the functions in the parallel package unless a quick parallel calculation is needed.

cluster <- auto_cluster(2)
Num(for (i in 1:100) sample(1:100, 1), clust=cluster)
close_cluster(cluster)

Note that the additional overhead involved with parallelization means that the gain in performance will be relatively small (and often negative) unless the sequence is sufficiently large. Large comprehensions, though, can experience significant gains by specifying a cluster.

Summary Comprehensions

The eList package contains a number of summary functions that accept vector comprehensions and/or normal vectors. These functions allow users to combine multiple comprehensions and other vectors in a single function, then apply a summary function to all entries. Each summary comprehension accepts a cluster for parallel computations and the na.rm argument. Some examples:

# Are all values greater than 0?
All(for (i in 1:10) i > 0)
#> [1] TRUE

# Are no values TRUE? (combines comprehension with other values)
None(for (i in 1:10) i < 0, TRUE, FALSE)
#> [1] FALSE

# factorial(5)
Prod(for (i in 1:5) i)
#> [1] 120

# Summary statistics from a random draw of 1000 observations from normal distribution
Stats(for (i in rnorm(1000)) i)
#> $min
#> [1] -2.975354
#> 
#> $q1
#> [1] -0.6803642
#> 
#> $med
#> [1] 0.09313939
#> 
#> $q3
#> [1] 0.809412
#> 
#> $max
#> [1] 3.731964
#> 
#> $mean
#> [1] 0.06918359
#> 
#> $sd
#> [1] 1.04056

# Every other letter in the alphabet as a single character
Paste(for (i in seq(1,26,2)) letters[i], collapse=", ")
#> [1] "a, c, e, g, i, k, m, o, q, s, u, w, y"