---
title: "`wbids` Data Model"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{`wbids` Data Model}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

## Overview

The `wbids` package uses a relational data model with five interconnected tables. In our design, we primarily have R users in mind, particularly those who rely heavily on the popular data manipulation packages `dplyr` and `data.table`. For these users, having consistent and descriptive primary key column names across different tables (e.g., `entity_id`, `series_id`) simplifies writing joins across tables, reduces the risk of column name conflicts, and avoids ambiguity. We hence deliberately deviate from a common practice in data modeling to omit the entity prefix from a entity table (e.g. `entity_` in the entities table).

## Entity Relationships

The tables are related as follows:

- **ENTITIES** → DEBT_STATISTICS: One entity can have many debt statistics records
- **SERIES** → DEBT_STATISTICS: One series can have many debt statistics records
- **SERIES** → SERIES_TOPICS: One series can have many topic associations
- **COUNTERPARTS** → DEBT_STATISTICS: One counterpart can have many debt statistics records

The **DEBT_STATISTICS** table is the central fact table, linking entities, series, and counterparts through foreign keys (`entity_id`, `series_id`, `counterpart_id`).

## Table Details

### Entities

| Column name | Description | Example value |
|-----------------|-------------------------------------|-----------------|
| entity_id | ISO 3166-1 alpha-3 code of the entity | ZMB |
| entity_name | Standardized name of the entity | Zambia |
| entity_iso2code | ISO 3166-1 alpha-2 code of the entity | ZM |
| entity_type | Type of entity (e.g., country, region) | Country |
| capital_city | Capital city of the entity | Lusaka |
| region_id | Unique identifier for the region | SSF |
| region_iso2code | ISO 3166-1 alpha-2 code of the region | ZG |
| region_name | Name of the region | Sub-Saharan Africa |
| admin_region_id | Unique identifier for the administrative region | SSA |
| admin_region_iso2code | Unique identifier for the administrative region | ZF |
| admin_region_name | Name of the administrative region | Sub-Saharan Africa (excluding high income) |
| lending_type_id | Unique identifier for the lending type | IDX |
| lending_type_iso2code | ISO code of the lending type | XI |
| lending_type_name | Name of the lending type | IDA |

### Counterparts

| Column name | Description | Example value |
|-----------------|-------------------------------------|-----------------|
| counterpart_id | Unique identifier for the counterpart | 730 |
| counterpart_name | Standardized name of the counterpart | China |
| counterpart_iso2code | ISO 3166-1 alpha-2 code of the counterpart | CN |
| counterpart_iso3code | ISO 3166-1 alpha-3 code of the counterpart | CHN |
| counterpart_type | Type of counterpart (e.g., institution, country, region) | Country |

### Series

| Column name | Description | Example value |
|-----------------|--------------------|----------------------------------|
| series_id | Unique identifier for the data series | DT.DOD.DPPG.CD |
| series_name | Name of the series | External debt stocks, public and publicly guaranteed (PPG) (DOD, current US\$) |
| source_id | Unique identifier for the data source | 2 |
| source_note | Note about the data source | Public and publicly guaranteed debt comprises long-term external obligations of public debtors, including the national government,  Public Corporations, State Owned Enterprises, Development Banks and Other Mixed Enterprises, political subdivisions (or an agency of either), autonomous public bodies, and external obligations of private debtors that are guaranteed for repayment by a public entity. Data are in current U.S. dollars. |
| source_organization | Organization responsible for the data series | World Bank, International Debt Statistics. |

### Series Topics

| Column name | Description | Example value |
|----------------|--------------------------------|----------------|
| series_id      | Identifier for the series (FK) | DT.DOD.DPPG.CD |
| topic_id       | Identifier for the topic       | 3              |
| topic_name     | Name of the topic              | Economy & Growth |

### Debt Statistics

| Column name | Description | Example value  |
|----------------|--------------------------------|----------------|
| series_id      | Identifier for the series      | DT.DOD.DPPG.CD |
| entity_id   | Identifier for the entity   | ZMB            |
| counterpart_id | Identifier for the counterpart | 061.           |
| year           | Year of the data point         | 2020           |
| value          | Value of the data point        | 4298957000     |

## Assignment of Entity and Counterpart Types

The original World Bank IDS data includes a 'country' field, containing both countries and regions, and a 'counterpart-area' field, which may include countries, regions, and institutions. In our data model, these fields are renamed to 'entity' and 'counterpart' to clarify the types of entities in each column.

We also introduce corresponding type columns that specify whether a entity is a country (e.g., "Aruba") or a region (e.g., "Africa Eastern and Southern"), and whether a counterpart is a country, region, or a special category (e.g., "Global IFIs", "Global MDBs"). Each counterpart is represented in the entity table if it is a country or region, ensuring consistency across both tables.

## Harmonization of Entity and Counterpart Names

In some cases, the  IDS data provides different names for entities that appear both in the 'counterpart-area' and the 'country' data. We use the entity names whenever they are available and drop counterpart names with different wording. For instance, if the original data features "Cote D`Ivoire, Republic Of" in the counterpart table, but the country name is "Cote d'Ivoire", then we overwrite the former with the latter.