---
title: "Introduction to the cellKey-Package"
author: "Bernhard Meindl"
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
  toc: true
toc_depth: 5
number_sections: false
vignette: >
  %\VignetteIndexEntry{Introduction to the cellKey-Package}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, echo=FALSE}
library(rmarkdown)
```

## About the cellKey package
This package implements methods to provide perturbation for statistical tables. The implementation if greatly inspired by the the paper *Methodology for the Automatic Confidentialisation of Statistical Outputs from Remote Servers at the Australian Bureau of Statistics* (Thompson, Broadfoot, Elazar). This approach however was generalized and a new specification on how record keys are specified and the lookup-tables are defined is used. This package makes usage of perturbation tables that can be defined in the [**ptable**](https://github.com/sdcTools/ptable) package. This document describes the usage of version ``r packageVersion("cellKey")`` of the package.

## Main Features
In the **cellKey** package it is possible to pertub multidimensional count and magnitude tables. Functionality to generate suitable record keys is provided in function `ck_generate_rkeys()`. Using sha1-checksums on the input datasets, we can make sure that the same record keys are generated whenever the same input data set is used for which record keys should be generated. However, it is also possible of course to use already pre-generated record keys available as variable in the microdata set.

The package allows to make use of sampling weights. Thus, both weighted and unweighted count and magnitude tables can be perturbed. The hierarchical structure of the variables spanning the table can be arbitrarily complex. To generate these hierarchies, the **cellKey** packages depends on functionality available in package [**sdcHierarchies**](https://cran.r-project.org/package=sdcHierarchies). Finally, auxiliary methods are provided that allow to extract valuable information from objects created by the main function of this package `perturbTable()`. For example `mod_counts()` returns a data object showing for each cell the perturbation value and where it was found in the lookup table.

What is left is of course identifying bugs and issues and also to optimize performance of the package as well as to include some real world examples, eg. census tables.

## An Example
We now show the capabilities of the **cellKey** package by running an example that 

### Load the Package
```{r, include = FALSE, echo = TRUE}
library(cellKey)
packageVersion("cellKey")
```

```{r, echo = TRUE}
library(cellKey)
packageVersion("cellKey")
```

The first step in this approach is to generate a statistical table, which can be achieved using `ck_setup`. This function generates an object that contains all the information required to perturb count- (and optionally continuously scaled) variables. There are however a few inputs that `ck_setup()` requires and that need to be generated beforehand. These inputs are:

- `x`: a `data.frame` or `data.table` containing microdata
- `rkey`: defines where to find or how to compute record keys. It is possible to either specify a scalar character name or a number. If a number is specified, record keys are internally created by sampling from a uniform distribution between 0 and 1 with the random numbers rounded to the specified value of digits. If a name is specified, this name is interpreted as a variable name within `x` already containing record keys.
- `dims`: a named list where each list element contains a hierarchy created with functionality from package [**sdcHierarchies**](https://cran.r-project.org/package=sdcHierarchies) and each name refers to a variable in `x`
- `w`: either `NULL` (no weights) or the name of a variable in `x` that contains weights
- `countvars`: an optional character vector of variables in `x` holding counts. In any case a special count variable `total` is internally generated that is `1` for each row of `x`
- `numvars`: an optional character vector of variables in `x` holding numerical variables that can later be perturbed.

We continue to show how to generate the required inputs. The first step is to prepare the inputdata.

### Specifying inputdata
The testdata set we are using in this example contains information on person-level along with sampling weights as well as some categorical and continuously scaled variables. We also compute some binary variables for which we also want to create perturbed tables. 
```{r}
dat <- ck_create_testdata()
dat <- dat[, c("sex", "age", "savings", "income", "sampling_weight")]
dat[, cnt_highincome := ifelse(income >= 9000, 1, 0)]
```

We are now adding record keys (which will be referenced later) with 7 digits to the data set using function `ck_generate_rkeys` as shown below:

```{r}
dat$rkeys <- ck_generate_rkeys(dat = dat, nr_digits = 7)
print(head(dat))
```

To ensure that the same record keys are computed each and every time for the same data set, a seed based on the sha1-hash of the input dataset is computed by default in `ck_generate_rkeys`. This seed is used before sampling the record keys and may be overwritten using the `seed` argument.

The goal of this introduction is to create a perturbed table of counts of variables `sex` by `age` for all observations as well as for the subgroups given by `cnt_highincome` that are non-zero. We also want to create perturbed tables of continously scaled variables `savings` and `income` also giving the hierarchical structure defined by `sex` by `age`.

### Specifying dimensions
It is required to define hierarchies for each of the classifying variables of the desired table. There are two ways how the hierarchical structure of these variables (including sub-totals) can be specified. One way is to use `"@; value` format as (also) used in [**sdcTable**](https://cran.r-project.org/package=sdcTable). We would suggest however, to use the second alternative which is using functionality from package [**sdcHierarchies**](https://cran.r-project.org/package=sdcHierarchies). In this vignette, we only show the preferred way to generate hierarchies for categorical variables `age` and `sex` below:

```{r}
dim_sex <- hier_create(root = "Total", nodes = c("male", "female"))
hier_display(dim_sex)
```

For variable `age` the process is very much the same:
```{r}
dim_age <- hier_create(root = "Total", nodes = paste0("age_group", 1:6))
hier_display(dim_age)
```

The idea of [**sdcHierarchies**](https://github.com/bernhard-da/sdcHierarchies) is to create a tree object using `hier_create()` and then add (using `hier_add()`), delete (with `hier_delete()`) or rename (using `hier_rename()`) elements from this hierarchical structure. For more examples have a look at the package vignette of the package which can be accessed with `sdcHierarchies::hier_vignette()`. [**sdcHierarchies**](https://github.com/bernhard-da/sdcHierarchies) also contains a shiny-based app that can be called with `hier_app()` which allows to interactively change and modify a hierarchy and allows to convert tree- and data.frame based inputs into various formats. For more information, have a look at `?hier_app` and the other help-files of the package.

After all dimensions have been specified, these inputs must then be combined into a named list. In this object, the list-names refer to variable names of the input data set and the elements the data objects that hold the hierarchy specification itself. This is why the list elements of `dims` in this example have to be named `sex` and `age` as the specification refers to variables `age` and `sex` in input set `dat`.

```{r}
dims <- list(sex = dim_sex, age = dim_age)
```

### Setup a table instance
We now have prepared the inputs and can define a generic statistical table using `ck_setup()` as shown below:

```{r}
tab <- ck_setup(
  x = dat,
  rkey = "rkeys",
  dims = dims,
  w = "sampling_weight",
  countvars = "cnt_highincome",
  numvars = c("income", "savings"))
```

`ck_setup()` returns a `R6` class object that contains not only the relevant data but also all available methods. Thus, it is not required to assign the results of such methods to new objects, instead, the object itself is automatically updated.

These objects also have a custom print method, showing some general information about the object:

```{r}
print(tab)
```


### Defining perturbation parameters
#### Perturbation parameters for count variables
The next task is to define parameters that are used to perturb count variables which can be achieved with `ck_params_cnts`. This function requires as input the result of either `pt_create_pParams`, `pt_create_pTable` or `create_cnt_ptable` from the [**ptable**](https://github.com/sdcTools/ptable) package. Please refer also to the documentation of this package for information on the required parameters. In this example we are going to use - amongst others - exemplary ptables that can are provided by the `ptable`-pkg for demonstration purposes:

```{r, message=FALSE, message=FALSE}
# two different perturbation parameter sets from the ptable-pkg
# an example ptable provided directly
ptab1 <- ptable::pt_ex_cnts()

# creating a ptable by specifying parameters
para2 <- ptable::create_cnt_ptable(
  D = 8, V = 3, js = 2, pstay = 0.5, 
  optim = 1, mono = TRUE)
```

We then need to create the required inputs for the cellKey package.

```{r}
p_cnts1 <- ck_params_cnts(ptab = ptab1)
p_cnts2 <- ck_params_cnts(ptab = para2)
```

`ck_params_cnts()` returns objects that can be used as inputs in method `params_cnts_set()`. In argument `v` one may specify count variables for which the supplied perturbation parameters should be used. If `v` is not specified, the perturbation parameters are used for all count variables. 

```{r}
# use `p_cnts1` for variable "total" (which always exists)
tab$params_cnts_set(val = p_cnts1, v = "total")

# use `p_cnts2` for "cnt_highincome"
tab$params_cnts_set(val = p_cnts2, v = "cnt_highincome")
```

It is therefore entirely possible to use different parameter sets for different variables. Modifying perturbation parameters for some variables is easy, too. It is only required to apply the `params_cnts_set()`-method again which will replace any previously defined parameters.

#### Perturbation parameters for continuous variables
Setting and defining perturbation parameters for continuous variables works similarily. The required functions are `ck_params_num()` to create input objects that can be set with the `params_nums_set` method.  Please note that it is possibly by specifying the `path` argument in both `ck_params_nums()` and `ck_params_cnts()` to save the parameters additionally as yaml-file. Using `ck_read_yaml()`, these files can later be imported again. This feature is useful for re-using parameter settings.

The underlying framework on how to perturb continuous tables differs from the proposed method from `ABS`. One possible approach is based on a *"flex function"*. This approach (which is described in deliverable `D4.2` in the project [perturbative confidentiality methods](https://ec.europa.eu/eurostat/cros/content/perturbative-confidentiality-methods_en) allows to apply different magnitude of noise to larger and smaller cells. Users can define the required parameters for the flex-approach with function `ck_flexparams()`. The required inputs are:

- `fp`: the flexpoint defining at which point should the underlying noise coefficient function reach its desired maximum (which is defined by the first element of `p`)
- `p`: numeric vector of length `2` with `p[1] > p[2]` where both elements specify a percentage. The first value refers to the desired maximum perturbation percentage for small cells (depending on fp) while the second element refers to the desired maximum perturbation percentage for large cells.
- `epsilon`: a numeric vector in descending order with all values in `[0; 1]` and with the first element forced to equal `1`. The length of this parameter must correspond with the number of `top_k` specified in `ck_params_nums()` (which will be discussed later). 

```{r}
# parameters for the flex-function
p_flex <- ck_flexparams(
  fp = 1000,
  p = c(0.3, 0.03),
  epsilon = c(1, 0.5, 0.2))
```

In the [**cellKey**](https://github.com/sdcTools/cellKey) package it is possible to select the underlying data that form the base for the perturbation differently. In `ck_params_nums()` the specific approach can be selected in argument `type`. The valid choices for this argument are:

- `"top_contr"`: the `k` largest contributions to each cell are used in the perturbation procedure with the number `k` required to be specified in argument `top_k`
- `"mean"`: weighted cellmeans are used as starting points
- `"range"`: the difference between largest and smallest unweighted contributions for each cell are used as base for the perturbation procedure
- `"sum"`: weighted cellvalues are used as starting points for the perturbation


Another, more basic approach, is to use a constant perturbation magnitude for all cells, independent on their (weighted) values. The required parameters can be defined with `ck_simpleparams()` as shown below:

```{r}
# parameters for the simple approach
p_simple <- ck_simpleparams(
  p = 0.05,
  epsilon = 1)
```

In this appraoch it is only required to specify a single percentage value `p` and - as in the case for the flex function - a vector of epsilons that are used in the case when `top_k > 1`. 

Further important parameters for `ck_params_nums()` are:

- `mu_c`: an extra amount of perturbation applied to sensitive cells (restricted to the first of `top_k` noise components). In the following example we demonstrate how to identify sensitive cells for numeric variables.
- `same_key`: a logical value specifying if the original cell key (`TRUE`) should be used for the lookup of the largest contributor of a cell or if a perturbation of the cellkey itself (`FALSE`) should take place.
- `use_zero_rkeys`: a logical value defining if record keys of units not contributing to a specific numeric variables should be used (`TRUE`) or ignored (`FALSE`) when cell keys are computed.

A very important parameter is `ptab` which actually holds the perturbation tables in which perturbation values are looked up. This input can be specified differently in the case when numeric variables should be perturbed. It can be either an object derived from `ptable::pt_create_pTable(..., table = "nums")` in the most simple case. More advanced is to supply a named list, where the allowed names are shown below and each element must be the output of `ptable::pt_create_pTable(..., table = "nums")`.

- `"all"`: this ptable will be used for all cells; if specified, list-elements named `"even"`or `"odd"` are ignored
- `"even"`: this perturbation table will be used to look up perturbation values for cells with an even number of contributors
* `"odd"`: will be used to look up perturbation values for cells with an odd number of contributors
* `"small_cells"`: if specified, this ptable will be used to extract perturbation values for very small cells

Please note, that if the goal is to have different perturbation tables for cells with an even/odd number of contributors, both `"even"` or `"odd"` must be available in the input list. In the chunk below we create four different perturbation tables. For details on the parameters, please look at the documentation of the [`ptable`](https://github.com/sdcTools/ptable) package, especially in `ptable::create_num_ptable()`.

```{r}
# same ptable for all cells except for very small ones
ex_ptab1 <- ptable::pt_ex_nums(parity = TRUE, separation = TRUE)
```

We can now use these tables to finally create objects containing all the required information to create perturbed magnitude tables using `ck_params_nums`. In the first case we want the same perturbation table (`ptab_all`) for cells with an even/odd number of contributors but want to use `ptab_sc` for very small cells. 
```{r}
p_nums1 <- ck_params_nums(
  type = "top_contr",
  top_k = 3,
  ptab = ex_ptab1,
  mult_params = p_flex,
  mu_c = 2,
  same_key = FALSE,
  use_zero_rkeys = TRUE)
```

The second input we generate should use different ptables for cells with an even/odd number of contributing units (`ptab_even` and `ptab_odd`) but should not use a specific perturbation table for very small cells.

```{r}
ex_ptab2 <- ptable::pt_ex_nums(parity = FALSE, separation = FALSE)
```

As above, we need to use `ck_params_nums()` to compute suitable inputs.

```{r}
p_nums2 <- ck_params_nums(
  type = "mean",
  ptab = ex_ptab2,
  mult_params = p_simple,
  mu_c = 1.5,
  same_key = FALSE,
  use_zero_rkeys = TRUE)
```

The package internally computes the separation point that is used for very small cells in case this is required. Details on this can also be found in deliverable [`D4.2`](https://ec.europa.eu/eurostat/cros/content/perturbative-confidentiality-methods_en).

Now we can attach the results from `ck_params_nums()` to numeric variables using the `params_nums_set()`-method as shown below:

```{r}
tab$params_nums_set(v = "income", val = p_nums1)
tab$params_nums_set(v = "savings", val = p_nums1)
```

In order to make use of parameter `mu_c` that allows ab add extra amount of protection to sensitive cells, one may identify sensitive cells according to some rules. The following methods to identify sensitive cells are implemented:

- `supp_p()`: identify sensitive cells based on p%-rule
- `supp_pp()`: identify sensitive cells based on pq%-rule
- `supp_nk()`: identify sensitive cells based on nk-dominance rule
- `supp_freq()`: identify sensitive cells based on minimal frequencies for (weighted) number of contributors
- `supp_val()`: identify sensitive cells based on (weighted) cell values
- `supp_cells()`: identify sensitive cells based on their "names"

We now want to set all cells for variable `income` as sensitive to which less than `15` units contribute.

```{r, eval = TRUE}
tab$supp_freq(v = "income", n = 15, weighted = FALSE)
```


To set specific cells independent on values but their names, one may use the `$supp_cells()`-method. This cell requires a `data.frame` as input that contains a column for each dimensional variable specified. Each row of this input is considered as a cell where `NAs` are used as placeholders that match any characteristic of the relevant variable. Using the `data.frame` `inp` show below, the programm would suppress the following cells:

- `female` x `age_group1`
- `male` x `age_group3`
- `male` x any age group available in the data


```{r, eval = FALSE}
inp <- data.frame(
  "sex" = c("female", "male", "male"),
  "age" = c("age_group1", "age_group3", NA)
)
```


### Compute perturbed tables
It is now possible to finally perturbed variables using the `perturb()`-method. As `tab` - the object created with `ck_setup()` - already contains all possible data, the only required input is the name of a variable that should be perturbed.

- `v`: a character vector specifying one or more variables that should be perturbed.

```{r}
tab$perturb(v = "total")
```

After this call, object `tab` is updated and contains now also perturbed values for variable `total`. We note that no explicit assignment is required. The following code shows that we also can perturb cnt- and numerical variables in one single call.

```{r}
tab$perturb(v = c("cnt_highincome", "savings", "income"))
```

A `data.table` containing original and perturbed values can now be extracted using the `freqtab()`- and `numtab()` methods as discussed next.

### Extracting results
#### Obtain perturbed tables for count tables
Applying the `freqtab()`-method to one ore more already perturbed variables returns a `data.table` that contains for each table cell the unpertubed and perturbed (weighted and/or unweighted) counts. This function has the following arguments:

- `v`: one or more variable names of already perturbed count variables
- `path`: if not `NULL`, a (relative or absolute) path to which the resulting output table should be written. A `csv` file will be generated and `.csv` will be appended to the value provided.

The method returns a `data.table` with all combinations of the dimensional variables in the first $n$ columns and after those the following columns:

- `vname`: name of the perturbed variable
- `uwc`: unweighted counts
- `wc`: weighted counts
- `puwc`: perturbed unweighted counts
- `pwc`: perturbed weighted counts

```{r}
tab$freqtab(v = c("total", "cnt_highincome"))
```

#### Obtain perturbed tables for magnitude tables
Using the `numtab()`-method allows to extract results for continuous variables. The required inputs are the same as for the `freqtab()`-method and the output returns a `data.table` with all combinations of the dimensional variables in the first $n$ columns and the following additional columns:

- `vname`: name of the perturbed variable
- `uws`: unweighted sum
- `ws`: weighted cellsum
- `pws`: perturbed weighted sum of the given cell

We now have a look at the results of the variables `savings` and `income` that we already have perturbed.

```{r}
tab$numtab(v = c("savings", "income"))
```

### Utility-measures
#### Utility measures for count variables
Method `measures_cnts()` allows to compute information loss measures for perturbed count variables. Its application is as simple as:

```{r}
tab$measures_cnts(v = "total", exclude_zeros = TRUE)
```

This function returns a list with several utility measures that are now discussed. For further information have a look at `?ck_cnt_measures` as the same set of measures can also be computed for two vectors containing original and perturbed values.

- `overview`: a `data.table` with the following three columns:
    * noise: amount of noise computed as orig - pert
    * cnt: number of cells perturbed with the value given in column noise
    * pct: percentage of cells perturbed with the value given in column noise

- `measures`: a `data.table` containing measures of the distribution of three different distances between original and perturbed values of the unweighted counts. Column `what` specifies the computed measure. The three distances considered are:
    * `d1`: absolute distance between original and masked values
    * `d2`: relative absolute distance between original and masked values
    * `d3`: absolute distance between square-roots of original and perturbed values

- `cumdistr_d1`, `cumdistr_d2` and `cumdistr_d3`: for each distance d1, d2 and d3, a data.table with the following three columns:
    * `cat`: a specific value (for d1) or interval (for distances d2 and d3)
    * `cnt`: number of records smaller or equal the value in column cat for the given distance
    * `pct`: proportion of records smaller or equal the value in column cat for the selected distance

- `false_zero`: number of cells that were perturbed to zero
- `false_nonzero`: number of cells that were initially zero but have been perturbed to a number different from zero

If argument `exclude_zeros` is `TRUE` (the default setting), empty cells are excluded when computing distance-based measures `d1`, `d2` and `d3`.

#### Utility measures for continuous variables
For now, no utility measures for continuous variables are available. This will change in a future version.

### Additional Features
Finally we note that there are `print` and `summary` methods implemented for objects created with from `ck_setup()` which can be used as shown below:

The `print()`-method shows the dimension of the table as well as the variables that are (or possibly can be) perturbed. It is also displayed if the table was constructed using weights. 

```{r}
tab$print() # same as (print(tab))
```

The `summary`-method shows some utility statistics for already perturbed variables as shown below:
```{r}
tab$summary()
```

## Summary
The package is *"work in progress"* and therefore, suggestions and/or bugreports are welcome. Please feel free to file an issue at [**our issue tracker**](https://github.com/sdcTools/userSupport/issues) or contribute to the package by filing a [**pull request**](https://github.com/sdcTools/cellKey/pulls) against the master branch.
