Generate a data dictionnary and search for variables with look_for()

Joseph Larmarange

Showing a summary of a data frame

Default printing of tibbles

It is a common need to easily get a description of all variables in a data frame.

When a data frame is converted into a tibble (e.g. with dplyr::as_tibble()), it as a nice printing showing the first rows of the data frame as well as the type of column.

library(dplyr)
iris %>% as_tibble()
## # A tibble: 150 x 5
##    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
##           <dbl>       <dbl>        <dbl>       <dbl> <fct>  
##  1          5.1         3.5          1.4         0.2 setosa 
##  2          4.9         3            1.4         0.2 setosa 
##  3          4.7         3.2          1.3         0.2 setosa 
##  4          4.6         3.1          1.5         0.2 setosa 
##  5          5           3.6          1.4         0.2 setosa 
##  6          5.4         3.9          1.7         0.4 setosa 
##  7          4.6         3.4          1.4         0.3 setosa 
##  8          5           3.4          1.5         0.2 setosa 
##  9          4.4         2.9          1.4         0.2 setosa 
## 10          4.9         3.1          1.5         0.1 setosa 
## # … with 140 more rows

However, when you have too many variables, all of them cannot be printed and their are just listed.

data(fertility, package = "questionr")
women
## # A tibble: 2,000 x 17
##    id_woman id_household weight interview_date date_of_birth   age residency
##       <dbl>        <dbl>  <dbl> <date>         <date>        <dbl> <dbl+lbl>
##  1      391          381  1.80  2012-05-05     1997-03-07       15 2 [rural]
##  2     1643         1515  1.80  2012-01-23     1982-01-06       30 2 [rural]
##  3       85           85  1.80  2012-01-21     1979-01-01       33 2 [rural]
##  4      881          844  1.80  2012-01-06     1968-03-29       43 2 [rural]
##  5     1981         1797  1.80  2012-05-11     1986-05-25       25 2 [rural]
##  6     1072         1015  0.998 2012-02-20     1993-07-03       18 2 [rural]
##  7     1978         1794  0.998 2012-02-23     1967-01-28       45 2 [rural]
##  8     1607         1486  0.998 2012-02-20     1989-01-21       23 2 [rural]
##  9      738          711  0.192 2012-03-09     1962-07-24       49 2 [rural]
## 10     1656         1525  0.192 2012-03-15     1980-12-25       31 2 [rural]
## # … with 1,990 more rows, and 10 more variables: region <dbl+lbl>,
## #   instruction <dbl+lbl>, employed <dbl+lbl>, matri <dbl+lbl>,
## #   religion <dbl+lbl>, newspaper <dbl+lbl>, radio <dbl+lbl>, tv <dbl+lbl>,
## #   ideal_nb_children <dbl+lbl>, test <dbl+lbl>

Note: in R console, value labels (if defined) are usually printed but they do not appear in a R markdown document like this vignette.

dplyr::glimpse()

The function dplyr::glimpse() allows you to have a quick look at all the variables in a data frame.

glimpse(iris)
## Rows: 150
## Columns: 5
## $ Sepal.Length <dbl> 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.4, 4.9, 5.4, 4…
## $ Sepal.Width  <dbl> 3.5, 3.0, 3.2, 3.1, 3.6, 3.9, 3.4, 3.4, 2.9, 3.1, 3.7, 3…
## $ Petal.Length <dbl> 1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1.5, 1…
## $ Petal.Width  <dbl> 0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.2, 0…
## $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, setosa, …
glimpse(women)
## Rows: 2,000
## Columns: 17
## $ id_woman          <dbl> 391, 1643, 85, 881, 1981, 1072, 1978, 1607, 738, 16…
## $ id_household      <dbl> 381, 1515, 85, 844, 1797, 1015, 1794, 1486, 711, 15…
## $ weight            <dbl> 1.803150, 1.803150, 1.803150, 1.803150, 1.803150, 0…
## $ interview_date    <date> 2012-05-05, 2012-01-23, 2012-01-21, 2012-01-06, 20…
## $ date_of_birth     <date> 1997-03-07, 1982-01-06, 1979-01-01, 1968-03-29, 19…
## $ age               <dbl> 15, 30, 33, 43, 25, 18, 45, 23, 49, 31, 26, 45, 25,…
## $ residency         <dbl+lbl> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,…
## $ region            <dbl+lbl> 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2,…
## $ instruction       <dbl+lbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 2, 1, 0,…
## $ employed          <dbl+lbl> 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1,…
## $ matri             <dbl+lbl> 0, 2, 2, 2, 1, 0, 1, 1, 2, 5, 2, 3, 0, 2, 1, 2,…
## $ religion          <dbl+lbl> 1, 3, 2, 3, 2, 2, 3, 1, 3, 3, 2, 3, 2, 2, 2, 2,…
## $ newspaper         <dbl+lbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,…
## $ radio             <dbl+lbl> 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0,…
## $ tv                <dbl+lbl> 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,…
## $ ideal_nb_children <dbl+lbl>  4,  4,  4,  4,  4,  5, 10,  5,  4,  5,  6, 10,…
## $ test              <dbl+lbl> 0, 9, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0,…

It will show you the first values of each variable as well as the type of each variable. However, some important informations are not displayed:

labelled::look_for()

look_for() provided by the labelled package will print in the console a data dictionnary of all variables, showing variable labels when available, the type of variable and a list of values corresponding to:

library(labelled)
look_for(iris)
##       variable           label
## 1 Sepal.Length            <NA>
## 2  Sepal.Width            <NA>
## 3 Petal.Length Length of petal
## 4  Petal.Width  Width of Petal
## 5      Species            <NA>
look_for(women)
##             variable                              label
## 1           id_woman                           Woman Id
## 2       id_household                       Household Id
## 3             weight                      Sample weight
## 4     interview_date                     Interview date
## 5      date_of_birth                      Date of birth
## 6                age Age at last anniversary (in years)
## 7          residency            Urban / rural residency
## 8             region                             Region
## 9        instruction               Level of instruction
## 10          employed                          Employed?
## 11             matri                 Matrimonial status
## 12          religion                           Religion
## 13         newspaper                    Read newspaper?
## 14             radio                   Listen to radio?
## 15                tv                          Watch TV?
## 16 ideal_nb_children           Ideal number of children
## 17              test               Ever tested for HIV?

Note that lookfor() and generate_dictionary() are synonyms of look_for() and works exactly in the same way.

Searching variables by key

When a data frame has dozens or even hundreds of variables, it could become difficult to find a specific variable. In such case, you can provide an optional list of keywords, which can be simple character strings or regular expression, to search for specific variables.

# Look for a single keyword.
look_for(iris, "petal")
##       variable           label
## 3 Petal.Length Length of petal
## 4  Petal.Width  Width of Petal
look_for(iris, "s")
##       variable label
## 1 Sepal.Length  <NA>
## 2  Sepal.Width  <NA>
## 5      Species  <NA>
# Look for with a regular expression
look_for(iris, "petal|species")
##       variable           label
## 3 Petal.Length Length of petal
## 4  Petal.Width  Width of Petal
## 5      Species            <NA>
look_for(iris, "s$")
##   variable label
## 5  Species  <NA>
# Look for with several keywords
look_for(iris, "pet", "sp")
##       variable           label
## 3 Petal.Length Length of petal
## 4  Petal.Width  Width of Petal
## 5      Species            <NA>
# Look_for will take variable labels into account
look_for(women, "read", "level")
##       variable                label
## 9  instruction Level of instruction
## 13   newspaper      Read newspaper?

By default, look_for() will look through both variable names and variables labels. Use labels = FALSE to look only through variable names.

look_for(women, "read")
##     variable           label
## 13 newspaper Read newspaper?
look_for(women, "read", labels = FALSE)
## Nothing found. Sorry.

Similarly, the search is by default case insensitive. To make the search case sensitive, use ignore.case = FALSE.

look_for(iris, "sepal")
##       variable label
## 1 Sepal.Length  <NA>
## 2  Sepal.Width  <NA>
look_for(iris, "sepal", ignore.case = FALSE)
## Nothing found. Sorry.

Advanced usages of look_for()

look_for() returns a detailed tibble which is summarized before printing. To deactivate default printing and see full results, simply use dplyr::as_tibble(), dplyr::glimpse() or even utils::View().

look_for(iris) %>% as_tibble()
## # A tibble: 5 x 2
##   variable     label          
##   <chr>        <chr>          
## 1 Sepal.Length <NA>           
## 2 Sepal.Width  <NA>           
## 3 Petal.Length Length of petal
## 4 Petal.Width  Width of Petal 
## 5 Species      <NA>
glimpse(look_for(iris))
## Rows: 5
## Columns: 2
## $ variable <chr> "Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"…
## $ label    <chr> NA, NA, "Length of petal", "Width of Petal", NA

The tibble returned by look_for() could be easily manipulated for advanced programming.

When a column has several values for one variable (e.g. levels or value_labels), results as stored with nested named list. You can convert named lists into simpler character vectors, you can use convert_list_columns_to_character().

look_for(iris) %>% convert_list_columns_to_character()
## # A tibble: 5 x 2
##   variable     label          
##   <chr>        <chr>          
## 1 Sepal.Length <NA>           
## 2 Sepal.Width  <NA>           
## 3 Petal.Length Length of petal
## 4 Petal.Width  Width of Petal 
## 5 Species      <NA>

Alternatively, you can use lookfor_to_long_format() to transform results into a long format with one row per factor level and per value label.

look_for(iris) %>% lookfor_to_long_format()
##       variable           label
## 1 Sepal.Length            <NA>
## 2  Sepal.Width            <NA>
## 3 Petal.Length Length of petal
## 4  Petal.Width  Width of Petal
## 5      Species            <NA>

Both can be combined:

look_for(women) %>%
  lookfor_to_long_format() %>%
  convert_list_columns_to_character()
## # A tibble: 17 x 2
##    variable          label                             
##    <chr>             <chr>                             
##  1 id_woman          Woman Id                          
##  2 id_household      Household Id                      
##  3 weight            Sample weight                     
##  4 interview_date    Interview date                    
##  5 date_of_birth     Date of birth                     
##  6 age               Age at last anniversary (in years)
##  7 residency         Urban / rural residency           
##  8 region            Region                            
##  9 instruction       Level of instruction              
## 10 employed          Employed?                         
## 11 matri             Matrimonial status                
## 12 religion          Religion                          
## 13 newspaper         Read newspaper?                   
## 14 radio             Listen to radio?                  
## 15 tv                Watch TV?                         
## 16 ideal_nb_children Ideal number of children          
## 17 test              Ever tested for HIV?

If you just want to use the search feature of look_for() without computing the details of each variable, simply indicate details = FALSE.

look_for(women, "id", details = FALSE)
##             variable                    label
## 1           id_woman                 Woman Id
## 2       id_household             Household Id
## 7          residency  Urban / rural residency
## 16 ideal_nb_children Ideal number of children