Dimensions and dimension attributes

Attributes considered by the designer as dimensions can be grouped taking into account the natural affinities between them. In particular, they can be grouped as they describe the “who, what, where, when, how and why” associated with the modelled business process. Two attributes share a natural affinity when they are only related in one context. When their relationships are determined by transactions or activities, they can occur in multiple contexts, if this occurs, they must be located in different dimensions.

In this way, a dimension is made up of a set of naturally related dimension attributes that describe the context of facts. Dimensions are used for two purposes: fact selection and fact grouping with the desired level of detail.

Additionally, in the dimensions hierarchies with levels and descriptors can be defined. More details can be found at Jensen, Pedersen, and Thomsen (2010). These concepts are not used in the current version of the package.

Facts and measures

A fact has a granularity, which is determined by the attributes of the dimensions that are considered at each moment. Thus, a measure in a fact has two components, the numerical property of the fact and an formula, frequently the SUM aggregation function, that allows combining several values of this measure to obtain a new value of the same measure with a coarser granularity (Jensen, Pedersen, and Thomsen 2010).

According to their behaviour to obtain a coarser granularity, three types of measures are distinguished: additive, semi-additive and non-additive. For additive measures, SUM is always a valid formula that maintains the meaning of the measure when the granularity changes. For semi-additive measures, there is no point in using SUM when changing the level of detail in any of the dimensions because the meaning of the measure changes, this frequently occurs in dimensions that represents time and measures representing inventory level. For non-additive measures, values cannot be combined across any dimension using SUM because the result obtained has a different meaning from the original measure (generally occurs with ratios, percentages or unit amounts such as unit cost or unit price).

The most useful measures are additive. If we have non-additive measures, they can generally be redefined from other additive measures.

Dimension tables

Dimension tables contain the context associated with business process measures. Although they can contain any type of data, numerical data is generally not used for dimension attributes because some query tools consider any numeric data as a measure.

Dimension attributes with NULL value are a source of problems when querying since DBMS and query tools sometimes handle them inconsistently, the result depends on the product. It is recommended to avoid the use of NULL and replace them with a descriptive text. In the case of dates, it is recommended to replace the NULL values with an arbitrary date in the very far future.

Fact table

At the centre of a star schema is the fact table. In addition to containing measures, the fact table includes foreign keys that refer to each of the surrogate keys in the dimension tables.

Multiple fact tables

It is frequent the need to have several fact tables for various reasons:

• We find measurements with different grain.

• There are measures that do not occur simultaneously, for example, when one occurs, we have no value for others and vice versa.

In reality it is about different business processes, each one has to have its own fact table but they have dimensions in common. This is known as a fact constellation which corresponds to the Kimball enterprise data warehouse bus architecture.

Conformed dimensions

When star schemas share a set of common dimensions, these dimensions are called conformed dimensions.

There are several possibilities to have conformed dimensions, the most obvious form is that the dimension tables share structure and content, that is, they are identical dimensions. This is the one considered in this version of the starschemar package.

Cleaning and conforming data

When data is loaded into a star schema, errors or inconsistencies can be discovered in some of them. In some cases, it is best to make corrections at the source of the data, in operational systems. Sometimes this is not possible and there is no other option but to modify the data before loading it into the star schema or even when it is already loaded.

Inconsistencies are often found in dimensions, when dimensions are integrated into one, for example to generate a role-playing dimension or conformed dimensions. Support for modifying dimension data is provided in the package.

Incremental refresh

When a star schema is built, an initial load is performed with all available data from a moment in time onwards.

Operational systems continue to operate and produce data. If we want to incorporate these data into the star schema, we have two possibilities:

• Carry out a new load with all the data available at that time.

• Perform an incremental refresh with only the new data.

In order to carry out this second option, the CDC (change data capture) system allows to exclusively obtain the new data produced.

In this package, it has been considered that we can obtain the new data, possibly mixed with updates to data already incorporated into the star schema, in order to carry out an incremental refresh of star schemas with them.

Data according to age range

To avoid having to write the name of the attributes of the table, with the following function we can have them in the form of a string. Thus, we can copy and paste each name as needed.

dput(colnames(mrs_age))
#> c("Reception Year", "Reception Week", "Reception Date", "Data Availability Year", 
#> "Data Availability Week", "Data Availability Date", "Year", "WEEK", 
#> "Week Ending Date", "REGION", "State", "City", "Age Range", "Deaths"
#> )

The definition of the dimensional model for the data considered is shown below.

library(tidyr)
library(starschemar)

dm_mrs_age <- dimensional_model() %>%
  define_fact(
    name = "mrs_age",
    measures = c(
      "Deaths"
    ),
    agg_functions = c(
      "SUM"
    ),
    nrow_agg = "nrow_agg"
  ) %>%
  define_dimension(
    name = "when",
    attributes = c(
      "Week Ending Date",
      "WEEK",
      "Year"
    )
  ) %>%
  define_dimension(
    name = "when_available",
    attributes = c(
      "Data Availability Date",
      "Data Availability Week",
      "Data Availability Year"
    )
  ) %>%
  define_dimension(
    name = "where",
    attributes = c(
      "REGION",
      "State",
      "City"
    )
  ) %>%
  define_dimension(
    name = "who",
    attributes = c(
      "Age Range"
    )
  )

In this case, all the elements have been explicitly defined, including aggregation functions and the name of an additional measure representing the number of rows aggregated, which is always included. Only data from two of the three possible time-related dimensions have been considered.

Data according to cause

In the case of data according to cause of death, the definition of the model is shown below.

dm_mrs_cause <- dimensional_model() %>%
  define_fact(
    name = "mrs_cause",
    measures = c(
      "Pneumonia and Influenza Deaths",
      "Other Deaths"
    ),
  ) %>%
  define_dimension(
    name = "when",
    attributes = c(
      "Week Ending Date",
      "WEEK",
      "Year"
    )
  ) %>%
  define_dimension(
    name = "when_received",
    attributes = c(
      "Reception Date",
      "Reception Week",
      "Reception Year"
    )
  ) %>%
  define_dimension(
    name = "when_available",
    attributes = c(
      "Data Availability Date",
      "Data Availability Week",
      "Data Availability Year"
    )
  ) %>%
  define_dimension(
    name = "where",
    attributes = c(
      "REGION",
      "State",
      "City"
    )
  )

If no aggregation function is indicated, by default, SUM is considered. Although not explicitly stated, it also includes by default the measure relative to the number of rows aggregated. In this case, the three dimensions related to the date have been defined.

Star schema definition

To define a star schema, we need a flat table and a dimensional model defined from it. To define a star schema, we need a flat table and a dimensional model defined from it. Once defined, we can apply format modification operations to it.

st_mrs_age <- star_schema(mrs_age, dm_mrs_age)

st_mrs_age <- st_mrs_age %>%
  role_playing_dimension(
    dim_names = c("when", "when_available"),
    name = "When Common",
    attributes = c("date", "week", "year")
  ) %>%
  snake_case() %>%
  character_dimensions(NA_replacement_value = "Unknown",
                       length_integers = list(week = 2))

st_mrs_cause <- star_schema(mrs_cause, dm_mrs_cause) %>%
  snake_case() %>%
  character_dimensions(
    NA_replacement_value = "Unknown",
    length_integers = list(
      week = 2,
      data_availability_week = 2,
      reception_week = 2
    )
  ) %>%
  role_playing_dimension(
    dim_names = c("when", "when_received", "when_available"),
    name = "when_common",
    attributes = c("date", "week", "year")
  )

Constellation definition

A constellation is defined from a list of star schemas, as shown below.

ct_mrs <- constellation(list(st_mrs_age, st_mrs_cause), name = "mrs")

All dimensions of the same name in star schemas must be compatible in structure and type of columns, and are defined as conformed dimensions. The conformed dimensions share all the instances of the original dimensions, this implies possible modifications in the surrogate keys that are transmitted to foreign keys of the component fact tables.

The tables of the obtained conformed dimensions are shown below.

It can be seen that the conformed dimensions are considered regardless of the type of dimension in the star schema. Definition of conformed dimensions does not imply any change in the definition of dimensions in star schemas: Role and role playing dimensions remain the same, only their rows may have changed.

Definition of updates

Using the following code, first of all, we get the names of the dimensions of a star schema, then, we get them by their name (using utils::View function we can see the rows). If there are several role dimensions, it is enough to consult one of them, updates defined on it will be propagated to the rest.

dim_names <- st_mrs_age %>%
    get_dimension_names()

where <- st_mrs_age %>%
  get_dimension("where")

# View(where)

when <- st_mrs_age %>%
  get_dimension("when")

# View(when)
# when[when$when_key %in% c(36, 37, 73), ]

who <- st_mrs_age %>%
  get_dimension("who")

# View(who)

Below is a selection of rows involved in update operations that are referred by their surrogate key.

Updates are defined on dimensions. Various update definition functions are available, as shown below.

updates_st_mrs_age <- record_update_set() %>%
  update_selection_general(
    dimension = where,
    columns_old = c("state", "city"),
    old_values = c("CT", "Bridgepor"),
    columns_new = c("city"),
    new_values = c("Bridgeport")
  ) %>%
  match_records(dimension = when,
                old = 37,
                new = 36) %>%
  update_record(
    dimension = when,
    old = 73,
    values = c("1962-02-17", "07", "1962")
  ) %>%
  update_selection(
    dimension = who,
    columns = c("age_range"),
    old_values = c("<1 year"),
    new_values = c("1: <1 year")
  ) %>%
  update_selection(
    dimension = who,
    columns = c("age_range"),
    old_values = c("1-24 years"),
    new_values = c("2: 1-24 years")
  ) %>%
  update_selection(
    dimension = who,
    columns = c("age_range"),
    old_values = c("25-44 years"),
    new_values = c("3: 25-44 years")
  ) %>%
  update_selection(
    dimension = who,
    columns = c("age_range"),
    old_values = c("45-64 years"),
    new_values = c("4: 45-64 years")
  ) %>%
  update_selection(
    dimension = who,
    columns = c("age_range"),
    old_values = c("65+ years"),
    new_values = c("5: 65+ years")
  )

Although in some of the functions the dimension surrogate key is referred to, updates only consider the values of the rest of the columns of the corresponding dimension. In this way, it is achieved that updates can be applied to equivalent dimensions of other star schemas, where the values of the surrogate key do not necessarily coincide with those of the dimension where updates were originally defined.

Updates application

Once updates are defined, they can be applied on the star schema from which they have been defined, as shown below.

st_mrs_age <- st_mrs_age %>%
  modify_dimension_records(updates_st_mrs_age)

The result obtained for the first star schema is shown below.

It can be seen that the row with the value “Bridgepor” in the city column has disappeared: it has been merged with the row with the correct value. This update has also been transmitted to the fact table. Although it is not seen in the tables, the same has happened with the dates that have been unified by the update.

The same updates can also be applied to other star schemas with dimensions in common with the original star schema, or to the shaped dimensions of a constellation, as shown below.

st_mrs_cause <- st_mrs_cause %>%
  modify_dimension_records(updates_st_mrs_age)

ct_mrs <- ct_mrs %>%
  modify_conformed_dimension_records(updates_st_mrs_age)

In the latter case, the ideal procedure is to apply particular updates to star schemas before forming the constellation and define custom updates for conformed dimensions, this is just an example. Functions are also available to obtain the names of conformed dimensions and the dimensions themselves. Updates on these dimensions are propagated to the corresponding dimensions of the component star schemas.

Data according to age range

Below you can see the function that groups the transformations defined for the data according to the age range.

mrs_age_definition <- function(ft, dm, updates) {
  star_schema(ft, dm) %>%
    role_playing_dimension(
      dim_names = c("when", "when_available"),
      name = "When Common",
      attributes = c("date", "week", "year")
    ) %>%
    snake_case() %>%
    character_dimensions(NA_replacement_value = "Unknown",
                         length_integers = list(week = 2)) %>%
    modify_dimension_records(updates)
}

We apply this function to new data sets, as shown below.

st_mrs_age_w10 <-
  mrs_age_definition(mrs_age_w10, dm_mrs_age, updates_st_mrs_age)

st_mrs_age_w11 <-
  mrs_age_definition(mrs_age_w11, dm_mrs_age, updates_st_mrs_age)

Once we have the data in the same format, we can apply the incremental refresh to the original star schema, as follows.

st_mrs_age <- st_mrs_age %>%
  incremental_refresh_star_schema(st_mrs_age_w10, existing = "replace") %>%
  incremental_refresh_star_schema(st_mrs_age_w11, existing = "replace")

In this case, it has been assumed that if data from previous periods appears among the new data, the new data has to replace the previous data (value “replace” in existing parameter).

If the star schema has been integrated into a constellation, the incremental refresh can be performed on it, as follows.

ct_mrs <- ct_mrs %>%
  incremental_refresh_constellation(st_mrs_age_w10, existing = "replace") %>%
  incremental_refresh_constellation(st_mrs_age_w11, existing = "replace")

In this case, the corresponding star schema, the conformed dimensions and all the star schemas that share them are updated.

Data according to cause

Similar to how it has been done for age data, it can be done for cause data, as shown below.

mrs_cause_definition <- function(ft, dm, updates) {
  star_schema(ft, dm) %>%
    snake_case() %>%
    character_dimensions(
      NA_replacement_value = "Unknown",
      length_integers = list(
        week = 2,
        data_availability_week = 2,
        reception_week = 2
      )
    ) %>%
    role_playing_dimension(
      dim_names = c("when", "when_received", "when_available"),
      name = "when_common",
      attributes = c("date", "week", "year")
    ) %>%
    modify_dimension_records(updates)
}

st_mrs_cause_w10 <-
  mrs_cause_definition(mrs_cause_w10, dm_mrs_cause, updates_st_mrs_age)

st_mrs_cause_w11 <-
  mrs_cause_definition(mrs_cause_w11, dm_mrs_cause, updates_st_mrs_age)

st_mrs_cause <- st_mrs_cause %>%
  incremental_refresh_star_schema(st_mrs_cause_w10, existing = "group") %>%
  incremental_refresh_star_schema(st_mrs_cause_w11, existing = "group")

ct_mrs <- ct_mrs %>%
  incremental_refresh_constellation(st_mrs_cause_w10, existing = "group") %>%
  incremental_refresh_constellation(st_mrs_cause_w11, existing = "group")

In this case, the previously existing data is treated differently than it was in the previous case, now what is done is grouping it using the aggregation functions, assuming it is additional data that has not been entered before (value “group” in existing parameter).

Star schema

Various export possibilities are offered. Specifically, for a star schema one of them is to export the data as a flat table. The main difference from the initial data is that we have cleaned and conformed it. This operation is offered for completeness. To work only with flat tables, this package is not suitable.

To work with databases, it is useful to be able to export a star schema as a list of tibble with dimension and fact tables, as shown below.

tl <- st_mrs_age %>%
  star_schema_as_tibble_list()

Optionally, the export function allows the role playing dimensions to be included.

Constellation

To export constellation data, as well as a tibble list, the multistar format may be interesting, where you have a list of tibble for fact tables and another for dimension tables.

ms <- ct_mrs %>%
  constellation_as_multistar()

Star schema definition

A star schema is defined from a flat table and a dimensional model definition. Once defined, a star schema can be transformed by defining role playing dimensions, changing the writing style of element names or the type of dimension attributes. These operations are carried out through the following functions:

• star_schema(): Creates a star_schema object from a flat table (implemented by a tibble) and a dimensional_model object. Example:

st <- star_schema(mrs_age, dm_mrs_age)

• role_playing_dimension(): Given a list of star_schema dimension names, all with the same structure, a role playing dimension with the indicated name and attributes is generated. The original dimensions become role dimensions defined from the new role playing dimension. Example:

st <- star_schema(mrs_age, dm_mrs_age) %>%
  role_playing_dimension(
    dim_names = c("when", "when_available"),
    name = "When Common",
    attributes = c("Date", "Week", "Year")
  )

• snake_case(): Transform fact, dimension, measurement, and attribute names according to the snake case style. Example:

st <- star_schema(mrs_age, dm_mrs_age) %>%
  snake_case()

• character_dimensions(): Transforms numeric type attributes of dimensions into character type. In a star_schema numerical data are measurements that are situated in the facts. Numerical data in dimensions are usually codes, day, week, month or year numbers. There are tools that consider any numerical data to be a measurement, for this reason it is appropriate to transform the numerical data of dimensions into character data. It also allows indicating the literal to be used in case the numerical value is not defined. Example:

st <- star_schema(mrs_age, dm_mrs_age) %>%
  character_dimensions()

Constellation definition

Based on various star schemas, a constellation can be defined in which star schemas share common dimensions. Dimensions with the same name must be shared. It is defined by the following function:

• constellation(): Creates a constellation object from a list of star_schema objects. All dimensions with the same name in the star schemas have to be conformable. Example:

ct <- constellation(list(st_mrs_age, st_mrs_cause), name = "mrs")

Obtaining dimensions

We can obtain dimensions from a star schema or conformed dimensions from a fact constellation. Available functions in both cases are similar.

dn <- st_mrs_age %>%
  get_dimension_names()

where <- st_mrs_age %>%
  get_dimension("where")

dn <- ct_mrs %>%
  get_conformed_dimension_names()

when <- ct_mrs %>%
  get_conformed_dimension("when")

Definition of updates

Modifications are defined on dimension rows in various ways based exclusively on the values of the dimension fields. Although the surrogate key intervenes in the definition, the result, internally, does not depend on it so that it can be applied more generally in other star schemas.

• record_update_set(): A record_update_set object is created. Stores updates on dimension records. Each update is made up of a dimension name, an old value set, and a new value set. Example:

updates <- record_update_set()

• match_records(): For a dimension, given the primary key of two records, it adds an update to the set of updates that modifies the combination of values of the rest of attributes of the first record so that they become the same as those of the second. Example:

updates <- record_update_set() %>%
  match_records(dimension = where,
                old = 1,
                new = 2)

• update_record(): For a dimension, given the primary key of one record, it adds an update to the set of updates that modifies the combination of values of the rest of attributes of the selected record so that they become those given. Example:

updates <- record_update_set() %>%
  update_record(
    dimension = where,
    old = 1,
    values = c("1", "CT", "Bridgeport")
  )

• update_selection(): For a dimension, given a vector of column names, a vector of old values and a vector of new values, it adds an update to the set of updates that modifies all the records that have the combination of old values in the columns with the new values in those same columns. Example:

updates <- record_update_set() %>%
  update_selection(
    dimension = where,
    columns = c("city"),
    old_values = c("Bridgepor"),
    new_values = c("Bridgeport")
  )

• update_selection_general(): For a dimension, given a vector of column names, a vector of old values for those columns, another vector column names, and a vector of new values for those columns, it adds an update to the set of updates that modifies all the records that have the combination of old values in the first column vector with the new values in the second column vector. Example:

updates <- record_update_set() %>%
  update_selection_general(
    dimension = where,
    columns_old = c("state", "city"),
    old_values = c("CT", "Bridgepor"),
    columns_new = c("city"),
    new_values = c("Bridgeport")
  )

Updates application

Defined updates can be applied on a star schema or on the conformed dimension of a fact constellation.

st <- st_mrs_age %>%
  modify_dimension_records(updates_st_mrs_age)

ct <- ct_mrs %>%
  modify_conformed_dimension_records(updates_st_mrs_age)

Star schema

• incremental_refresh_star_schema(): Incrementally refresh a star schema with the content of a new one that is integrated into the first. Once the dimensions are integrated, if there are records in the fact table whose keys match the new ones, new ones can be ignored, they can be replaced by new ones, all of them can be grouped using the aggregation functions, or they can be deleted. Therefore, the possible values of the existing parameter are: “ignore”, “replace”, “group” or “delete”. Example:

st <- st_mrs_age %>%
  incremental_refresh_star_schema(st_mrs_age_w10, existing = "replace")

Constellation

• incremental_refresh_constellation(): Incrementally refresh a star schema in a constellation with the content of a new star schema that is integrated into the first. Example:

ct <- ct_mrs %>%
  incremental_refresh_constellation(st_mrs_age_w10, existing = "replace")

Star schema

• star_schema_as_flat_table(): We can again obtain a flat table, implemented using a tibble, from a star schema. Example:

ft <- st_mrs_age %>%
  star_schema_as_flat_table()

• star_schema_as_multistar(): We can obtain a multistar. A multistar only distinguishes between general and conformed dimensions, each dimension has its own data. It can contain multiple fact tables. Example:

ms <- st_mrs_age %>%
  star_schema_as_multistar()

• star_schema_as_tibble_list(): We can obtain a tibble list with them. Role playing dimensions can be optionally included. Example:

tl <- st_mrs_age %>%
  star_schema_as_tibble_list(include_role_playing = TRUE)

Constellation

• constellation_as_multistar(): We can obtain a multistar. A multistar only distinguishes between general and conformed dimensions, each dimension has its own data. It can contain multiple fact tables. Example:

ms <- ct_mrs %>%
  constellation_as_multistar()

• constellation_as_tibble_list(): We can obtain a tibble list with them. Role playing dimensions can be optionally included. Example:

tl <- ct_mrs %>%
  constellation_as_tibble_list(include_role_playing = TRUE)