---
title: "Daff"
subtitke: "Diff, patch and merge for data.frames"
author: "Edwin de Jonge (@edwindjonge), Gregory Warnes"
date: "UseR!2017, July 6, 2017"
output:
  beamer_presentation:
    keep_tex: yes
    includes:
      in_header: header.tex
    theme: "Frankfurt"
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = FALSE)
library(magrittr)
```

## Who am I (Edwin)

\begin{picture}(0,0)
  \put(280,0){\includegraphics[width=1cm]{fig/cbs}}
\end{picture}

- Statistical consultant / methodologist
- Statistics Netherlands (CBS): all official statistics of NL
- twitter: @edwindjonge
- github: https://github.com/edwindj

**Stepping in for [Greg Warnes, coauthor](https://github.com/warnes)**

## What is daff? 

### Short version

> Daff is a diff for `data.frame`s

- Detect changes: **`diff_data`**, **`differs_from`**
- Store and restore diff: **`write_diff`**, **`read_diff`**
- Patch updated data: **`patch_data`**, **`merge_data`**
- Render a diff: **`render_diff`**

And now for the long version...

## Diff?

### `diff`

- Command-line utility for comparing text files.
- Used in all source code version control systems.
- `diff` checks *lines*:
      - which lines have changed, removed or added.

### `daff`

\begin{picture}(0,0)
  \put(250,-70){\includegraphics[width=2cm]{fig/utility.png}}
\end{picture}

- Utility for comparing tables
- `daff` compares *records* and *columns*: 
      - which values changed
      - rows added/removed
      - columns added/removed.

## Why o why?

\begin{picture}(0,0)
  \put(120,-60){\includegraphics[width=5cm]{fig/why.png}}
\end{picture}

- To support version control of `data.frame`.
- To log changes of data.
- e.g. subsequent steps in data process: what did these steps do?
- To support monitoring external data changes beyond your control.
- To make even manual editing reproducible (_Note: manual editing is really really bad_).

## Use case: data update

\begin{picture}(0,0)
  \put(120,-10){\includegraphics[width=2cm]{fig/update}}
\end{picture}

### Raw data update

- You have build a nice R script:
    - takes raw data as input
    - removes errors
    - fits a model
    - calculates output

You get an updated raw data file: what are the changes? 

- Did output change?
- Also input: should the script be adapted, e.g. data cleaning?

## Use case: manual editing

- **bad practice**, but it happens: e.g. implausible values, manual data correction.

### Manual editing
\begin{picture}(0,80)
  \put(120,0){\includegraphics[height=0.3\textheight]{fig/blackbox}}
\end{picture}
- Compare the input and output
- Make the manual step _reproducible_: all process steps can be re-executed:

    - data + changes = new data
    - in `diff` parlor: version1 + patch = version2

## Daff protocol


### Highlighter diff format

- highlighter diff format: https://paulfitz.github.io/daff-doc/spec.html
- diff protocol for tabular data.
- shows rows/columns that changed.
- supports patching data.
- format itself is in tabular format (nifty!)
- can be stored in txt (csv) or db.

## Detecting changes

`daff` detects the following changes:

- changed a value.
- row added.
- row removed.
- column added.
- column removed
- type change of a column (partially)

    - `daff` supports it, but highlighter format not

## `diff_data`: value was changed

```{r, echo=TRUE}
library(daff)
x         <- data.frame(A=1, B=  1)
x_changed <- data.frame(A=1, B=100)
patch <- diff_data(x, x_changed)
print(patch)
```
## `patch_data`: apply the change

```{r, echo=TRUE}
x
patch_data(x, patch)
```

replay` the change on original data:

- when org. data is updated, same procedure!

## `diff_data`: row was added

```{r, echo=TRUE}
x         <- data.frame(A=1  , B=1)
x_changed <- data.frame(A=1:2, B=1:2)
diff_data(x,x_changed)
```

## `diff_data`: row was deleted

```{r, echo=TRUE}
x         <- data.frame(A=1:2, B=1:2)
x_changed <- data.frame(A=1  , B=1)
diff_data(x,x_changed)
```

## `diff_data`: column was added

```{r, echo=TRUE}
x         <- data.frame(A=1, B=1)
x_changed <- data.frame(A=1, B=1, C=1)
diff_data(x,x_changed)
```
## `diff_data`: column was removed

```{r, echo=TRUE}
x         <- data.frame(A=1, B=1, C=1)
x_changed <- data.frame(A=1, B=1)
diff_data(x,x_changed)
```
## diff_data options

```{r, echo=TRUE, eval=FALSE}
diff_data( data_ref, data
         , always_show_header       = TRUE
         , always_show_order        = FALSE
         , columns_to_ignore        = c()
         , count_like_a_spreadsheet = TRUE
         , ids                      = c()
         , ignore_whitespace        = FALSE
         , never_show_order         = FALSE
         , ordered                  = TRUE
         , ... 
         )
```
## `differs_from`

- Pipe-friendly version of `diff_data`

```{r, echo=TRUE, eval=FALSE}
x_changed %>% 
  differs_from(x)

# same as

diff_data(x, x_changed)
```


## Merging


- Combine two derived `data.frame`s from a common parent.

```{r, echo=TRUE}
x   <- data.frame(A =   1, B=  1)
# two changes were made in parallel
x_a <- data.frame(A = 100, B=  1)
x_b <- data.frame(A =   1, B=100)
merge_data(x, x_a, x_b)
```

## Reading and writing table diffs

```{r, echo=TRUE}
x         <- data.frame(A = 1, B =   1)
x_changed <- data.frame(A = 1, B = 100)
# write diff to disk
diff_data(x, x_changed) %>% 
  write_diff("diff.csv")

# and read it again from disk
read_diff("diff.csv") %>% 
  patch_data(x, .)
```

## Render diff

```{r, echo=T, eval=FALSE}
x         <- data.frame(A = 1:2, B = 1:2)
x_changed <- data.frame(         B = 2  , C = 1)

x_changed %>% 
  differs_from(x) %>% 
  render_diff(use.DataTable=FALSE)
```

\begin{center}
  \includegraphics[width=7cm]{fig/daff.png}
\end{center}

## Implementation

- Wraps the library **`daff.js`**, by Paul Fitzpatrick (@fitzyfitzyfitzy).
- library actually written in Haxe, which compiles to js, python, C++
- Uses R package **`V8`** to run `daff.js`, by Jeroen Ooms(@opencpu):
  
    - With `V8` any js library can be run from R!

## Other R libs

### `diffobj`

Other R library:

- **`diffobj`**: very good general purpose diff for all R objects.
- Provides nice visualizations for differences between R objects.
- Also includes a `diffCsv` function, but more limited then `daff`.
- `daff` specialized in `data.frame`:
    - data aligning: `id` columns.
    - ignoring columns.
    - no patching of data.
    - ...

## 

\Large{Thank you for your attention!}

Interested?

```{r, echo=TRUE, eval=FALSE}
install.packages("daff")`
```

or visit: 

- http://github.com/edwindj/daff
