---
title: "Integration of dimension instances"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Integration of dimension instances}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

# Introduction

It is generally recommended to perform data cleansing as close as possible to the original data sources. Sometimes, the need to perform data cleaning operations is detected when the multidimensional system is already implemented, either because it hasn't been detected in the data sources or because it comes from multiple data sources. For example, we can integrate several stars with common dimensions in a constellation and, when conforming the dimensions, we observe that there are instances that are actually the same and should be merged. For these cases, the functions described in this document have been developed.

The document is structured as follows: First, we show examples of functions that allow us to explore the instances of the dimensions; the following shows how to modify instances of dimensions in a star database for several attributes or for a single attribute; the next section explains how the integration of instance modification operations is done when we define a constellation; finally, we end with conclusions.

# Exploration of dimensions

To show by example the available operations, we are going to use the multidimensional designs developed in the vignette titled *Star databases and RDBMS through the `dm` and `rolap` packages*, `vignette("v20-rdbms-dm")`, which are available as a `star_database` objects (`db_summary` and `db_finest`). 

## Consult tables and dimensions

We will start working on one of the star databases, `db_summary`. As shown below, we can get the names of the available tables and, exporting them as a list of `tibble` objects, we can see their number of instances.

```{r}
library(rolap)

class(db_summary)

names <- db_summary |>
  get_table_names()

db_tl <- db_summary |>
  as_tibble_list()

for (n in names) {
  cat(sprintf("name: %s, %d rows\n", n, nrow(db_tl[[n]])))
}
```

If instead of all the tables, we want to focus on the dimensions, we can get their names using the following function.

```{r}
db_summary |>
  get_dimension_names()
```

Although the design includes all of the above dimensions, we have several dimensions that are role-playing and role dimensions, as shown below.

```{r}
db_summary |>
  get_role_playing_dimension_names()
```

This means that these dimensions are considered to be represented by a single table.

## Detect similar attribute values in dimensions

The goal is to find dimension instances whose attribute values differ in possible different spellings (number of white space, case, use of punctuation). It does not detect abbreviations.

An initial exploration can be done considering individual attributes of the dimensions that are indicated or of all the dimensions, if none are indicated.

```{r}
db_summary |>
  get_similar_attribute_values_individually()
```

Since no dimension has been indicated, all the dimensions have been considered but the role-playing and role dimensions have been taken into account to show the results of only one of them.

In a complementary way, we could consider combinations of attributes within each dimension. If we do not indicate any dimension, the combination of all the attributes of each of the dimensions is considered, as shown below.

```{r}
db_summary |>
  get_similar_attribute_values()
```

To more clearly show how these functions work, we are going to modify a value in a test star database and perform the query on a single dimension.

```{r}
db_test <- db_summary
(value_seg1 <- db_test$dimensions$where_chain$table$chain[3])
(value_seg2 <- db_test$dimensions$where_chain$table$chain[4])
db_test$dimensions$where_chain$table$chain[3] <- ' ,1-.% $1)='
db_test$dimensions$where_chain$table$chain[4] <- ' ,1-.% $1)='

db_test |>
  get_similar_attribute_values_individually('where_chain')

db_test |>
  get_similar_attribute_values('where_chain')
```

## Get unique values of attributes in dimensions

If we focus on a dimension, we can query its attributes and get the unique value formed by combinations of them. We can indicate a set of attributes and, if we do not indicate any, all dimension attributes are considered.

```{r}
db_test |> 
  get_attribute_names('where_chain')

db_test |>
  get_unique_attribute_values('where_chain', attributes = c('country', 'segment'))

db_test |>
  get_unique_attribute_values('where_chain', attributes = 'chain')

db_test |>
  get_unique_attribute_values('where_chain')
```


# Modifying attribute instances in dimensions

In this section we are going to show several cases of modification of attribute values. Surely, the most suitable thing would be to carry out the necessary modification operations in the data sources (or as close to them as possible). In any case, we are going to do them here to show the possibilities of the `rolap` package.

## Modification of a single value indicating several attributes

First, let's start by fixing the bug introduced in the test star database. For all of the above functions, we can define the parameter `col_as_vector` that indicates to display the values as a vector. We will see later that this can be useful to make changes to the values, since it is the way to indicate them. 

```{r}
db_test |>
  get_unique_attribute_values(
    'where_chain',
    attributes = c('chain', 'country'),
    col_as_vector = "As a vector"
  )
```

Using the following function, we can indicate one or more attributes and the old and new values for them. If we indicate several attributes, in each call we can only change a combination of values. In this way we restore the original values that we had saved, as you can check below.

```{r}
db_summary2 <- db_test |>
  replace_attribute_values(
    name = 'where_chain',
    attributes = c('chain', 'country'),
    old = c(' ,1-.% $1)=', 'CZE'),
    new = c(value_seg1, 'CZE')
  ) |>
  replace_attribute_values(
    name = 'where_chain',
    attributes = c('chain', 'country'),
    old = c(' ,1-.% $1)=', 'SVK'),
    new = c(value_seg2, 'SVK')
  )

db_summary2 |>
  get_unique_attribute_values(
    'where_chain',
    attributes = c('chain', 'country'),
    col_as_vector = "As a vector"
  )
```

## Modifying multiple values of an attribute

If we want to modify the values of a single attribute, we can indicate several values that correspond to a new value.

In this case, we are going to make modifications in two of the dimensions. First, in dimension *what*.

```{r}
t <- db_summary2 |>
  get_unique_attribute_values('what', col_as_vector = "As a vector")
```

Suppose that, in view of the available values, we consider that we do not need details of all those products, but that we need a simplification: The one shown below. 


```{r}
db_summary3 <- db_summary2 |>
  replace_attribute_values(
    name = 'what',
    attributes = 'product',
    old = c('Autokosmet.'),
    new = c('Autokozmetik')
  ) |>
  replace_attribute_values(
    name = 'what',
    attributes = 'product',
    old = c('Diesel aditiv'),
    new = c('Diesel +')
  ) |>
  replace_attribute_values(
    name = 'what',
    attributes = 'product',
    old = c('Nafta Plus'),
    new = c('Nafta +')
  ) |>
  replace_attribute_values(
    name = 'what',
    attributes = 'product',
    old = c('Nat.Super', 'Natural Plus', 'Natural Spec'),
    new = c('Natural +')
  ) |>
  replace_attribute_values(
    name = 'what',
    attributes = 'product',
    old = c('Autoprísluš.', 'Dalnic.popl.', 'Knihy,nov.', 'LPG', 'Mytí vozidel',
            'Nemrz.kapal.', 'Obcerstvení', 'Oleje,tuky', 'Potraviny', 'Prev.náplne',
            'Provoz.nápl.', 'Umývanie voz', 'Zboží nesp.', 'Zpr.nakupu'),
    new = c('Other')
  ) |>
  group_dimension_instances(name = 'what')
```

We change the name of some products because there seems to be an error or they were spelled two different ways; in others cases we group several products into a new one, the largest grouping is for products that are renamed *Other*. Finally, if, as a result of the modifications, we have reduced the number of distinct instances of the dimension, when we finish making modifications, we must group the dimension instances to simplify it using the `group_dimension_instances()` function. As a side result, the facts may also be grouped.

The result for the dimension can be seen below.

```{r}
db_summary3 |>
  get_unique_attribute_values('what')
```

If we get the number of instances of the tables (as we did at the beginning of this document) we verify that the number of records in the fact table has been reduced. We also check that the number of original records (indicated by measure `transactions`) from which these have been obtained is maintained. That is, several fact table records have been added as a side effect of the dimension changes.


```{r}
db_tl <- db_summary3 |>
  as_tibble_list()

for (n in names) {
  cat(sprintf("name: %s, %d rows\n", n, nrow(db_tl[[n]])))
}

sum(db_tl[['transaction_summary']]$transactions)
```

## Modification of instances in role-playing and role dimensions

The two *When* dimensions are role-playing and role dimensions. That is, they have the same instances and one can be obtained from the other by renaming attributes if necessary and of course the table.

```{r}
db_summary3 |>
  get_role_playing_dimension_names()

db_summary3 |>
  get_unique_attribute_values('when')

db_summary3 |>
  get_unique_attribute_values('when_paid')
```

Let us suppose that we are not interested in the specific hours of the operations, but rather the part of the day in which they were carried out. Therefore, we define the modifications on one of the dimensions *When*.

```{r}
db_summary4 <- db_summary3 |>
  replace_attribute_values(
    name = 'when',
    attributes = 'hour',
    old = c('05', '06', '07', '08', '09', '10', '11'),
    new = c('Morning')
  ) |>
  replace_attribute_values(
    name = 'when',
    attributes = 'hour',
    old = c('12', '13', '14', '15', '16'),
    new = c('Afternoon')
  ) |>
  replace_attribute_values(
    name = 'when',
    attributes = 'hour',
    old = c('17', '18', '19', '20'),
    new = c('Evening')
  ) |>
  replace_attribute_values(
    name = 'when',
    attributes = 'hour',
    old = c('21', '22', '23', '00', '01', '02', '03', '04'),
    new = c('Night')
  ) |>
  group_dimension_instances(name = 'when')
```

The result obtained in both dimensions is shown below. That is, the modifications have been propagated to the equivalent dimension.

```{r}
db_summary4 |>
  get_unique_attribute_values('when')

db_summary4 |>
  get_unique_attribute_values('when_paid')
```

We do the same check as before on the number of instances of the tables and we observe that in this case the number of records in the fact table has also been reduced as a result of the aggregation carried out.

```{r}
db_tl <- db_summary4 |>
  as_tibble_list()

for (n in names) {
  cat(sprintf("name: %s, %d rows\n", n, nrow(db_tl[[n]])))
}

sum(db_tl[['transaction_summary']]$transactions)
```


# Modifications of instances in a constellation

To define a constellation we need at least two star databases. We consider the other star database from the starting example. We look at the instances of the *what* dimension.

```{r}
db_finest |>
  get_unique_attribute_values('what', col_as_vector = "As a vector")
```

Suppose some instances of this dimension are also modified, as shown below.

```{r}
db_finest2 <- db_finest |>
  replace_attribute_values(
    name = 'what',
    attributes = 'product',
    old = c('Additivum', 'Autokozmetik'),
    new = c('Other')
  ) |>
  group_dimension_instances(name = 'what')

db_finest2 |>
  get_unique_attribute_values('what', col_as_vector = "As a vector")
```
 
We define the constellation and as a result the dimensions are integrated.

```{r}
ct <- constellation("CSS", list(db_summary4, db_finest2))

class(ct)

names <- ct |>
  get_table_names()

db_tl <- ct |>
  as_tibble_list()

for (n in names) {
  cat(sprintf("name: %s, %d rows\n", n, nrow(db_tl[[n]])))
}
```

## Transformations in just one star database

For the dimension *When*, transformations were defined in only one of the star databases. The role-playing and role dimensions of the two star databases are also integrated. As a result, modifications are carried over from one star database to the other. 

```{r}
ct |>
  get_role_playing_dimension_names()

ct |>
  get_unique_attribute_values('when_processed')
```

The result is shown for dimension *when_processed*, which is not part of the star database on which the modifications were defined but it is modified.


## Transformations in both star databases

In the case of the *what* dimension, transformations have been carried out in both star databases: They are combined taking the new operations from one database to another.

```{r}
ct |>
  get_unique_attribute_values('what', col_as_vector = "As a vector")
```

The result obtained is explained because in star database *db_summary* the new operations of star database *db_finest* have been applied and vice versa. In the star database *db_finest*, the *Autokosmet.* to *Autokozmetik* transformation has been applied after the *Autokozmetik* to *Other* transformation, so the *Autokozmetik* value has been generated.

This situation has been sought in this example. When defining a constellation we must review the resulting dimensions. In any case, we can continue to make changes to the constellation, as shown below.

```{r}
ct2 <- ct |>
  replace_attribute_values(
    name = 'what',
    attributes = 'product',
    old = c('Autokozmetik'),
    new = c('Other')
  ) |>
  group_dimension_instances(name = 'what')

ct2 |>
  get_unique_attribute_values('what', col_as_vector = "As a vector")
```
 
 
# Conclusions

This document shows how to perform modification operations on dimension instances in both star databases and constellations. The modifications take into account the definition of role-playing and role dimensions both in a single database and in the integration of several to form a constellation.

In the case of performing operations on common dimensions of several star databases, the new operations are carried from one to another star database. If there is a conflict, we may need to define new operations on the constellation.

Although it is usually advisable to carry out the modification operations as close as possible to the data sources, this mechanism gives us flexibility especially when integrating several star databases into a constellation.
