Methodology, Key Considerations, and FAQs

The purpose of the Methodology, Key Considerations, and FAQs is to address questions that users have regarding the correlationfunnel methodology and to highlight key considerations before or after various steps in correlationfunnel process.

Libraries used.

library(correlationfunnel)
library(dplyr)
library(tibble)
library(ggplot2)
library(stringr)
library(forcats)
library(knitr)

Methodology

The method used is to perform binarization, the process of converting numeric and categorical variables to binary variables, and then to perform correlation analysis (Pearson method is used by default as the corelate() function wraps stats::cor()). This method lends itself well to quickly understanding relationships in data.

The Pearson correlation coefficient (see phi coefficient) measures the linear correlation between two dichotomous (binary) variables.

\[\phi = \frac{f_{11}*f_{00}-f_{01}*f_{10}}{\sqrt{(f_{11}+f_{01})*(f_{10}+f_{00})*(f_{00}+f_{01})*(f_{10}+f_{11})}}\]

Where from a Contingency Table,

y=0 y=1
x=0 f00 f01
x=1 f10 f11

Use of this method and tradeoffs are discussed at length herein and in L. Duan et al., 2014.1

Key Considerations

We recommend using the following best practices to get high performance with correlationfunnel:

Prior to Binarization Step

  1. Clean the Data - Make sure categorical data and numeric data are cleaned to take care of any errors and missing values.

  2. Perform Feature Engineering - Make sure useful information any text fields (e.g. description fields). Make sure date fields are converted to categorical data like day of the week. Use

  3. Discard Features Known to Have No Predictive Value - Example ID fields

  4. Discard ALL Non-Numeric and Non-Categorial Data - Remove date or datetime features, generic discription fields.

  5. Sample Data if Large - Performance could become slow on large data sets (Many Columns, Many Rows, and Many Category Levels within Categorical Columns).
    • Use the thresh_infreq to limit the number of infrequently appearing categories
    • Sample the data to reduce the row number.
    • Run the Correlation Funnel, and select the best features.
    • Increase the rows with the best feature selections.
    • Re-run the Correlation Funnel on the shorter-width (columns), larger-height (rows) dataset.

During Binarization

The binarize() function performs the transformation from numeric/categorical data to binary data. Use the following parameters to ensure that the correlation step goes well.

  1. Specify Number of Bins (Numeric Data) - Use n_bins to control the number of bins that numeric data is binned into. Too many and data gets overly discrete. Too few and trend can be missed. Typically 4 or 5 bins is adequate.

  2. Eliminate Infrequent Categories (Categorical Data) - Use thresh_infreq to control the appearance of infrequent categories. This will speed up processing of the correlation and prevent dimensionality (width of columns) getting out of control. Typically 0.01 threshold is adequate.

Prior to Correlation Step

  1. Address Data Imbalance - The correlation analysis is susceptible to data imbalance. Try reducing the number of majority class rows to get a proportion of 75% to 25% majority to minority class.

After Plotting the Correlation Funnel

  1. Garbage In, Garbage Out - Using the correlationfunnel package on data that has little relationship will not yield good results. If you are not getting good results by following the afformentioned “Key Considerations”, then your data may have little relationship. This is useful and a sign that you may need to collect better data.

  2. Something Bad Happened or You Think Something Good Should Have Happened - File an error on GitHub if you have a question and you believe that correlationfunnel should be reporting something differently and/or has an error.

FAQs

1. How does the Correlation Funnel Find Relationships in Numeric Data?

The approach to numeric data is to bin. This works well when non-linear relationships are present at the expense of a slight loss in linear relationship identification. We’ll see examples using synthetic data to illustrate this point.

1.1 Linear Relationships

Let’s make some sample data for Sales versus a highly correlated Macroeconomic Predictor.

We can see that the correlation between the two features is approximately 0.9.

What happens when we binarize()? What we want to know is which segment of the Macroeconomic Indicator is most correlated with the highest sales bin.

We can see that the best relationship between the highest sales bin is the highest macroeconomic indicator bin. The magnitude of the correlation is lower, but it’s still relatively high at approximately 0.8.

When we visualize with plot_correlation_funnel(), the macroeconomic predictor trend shows up indicating the highest macroeconomic indicator bin is highly correlated with the highest sales bin.

1.2 Non-Linear Relationships

The real beauty of the binning process is when nonlinear trends are present. Let’s again make some synthetic data this time between sales and age of customer.

We can see that the age range between 30 and 37 has much larger sales than the other age ranges. However, a linear trendline is flat indicating essentially no linear relationship.

The correlation analysis on un-binned data similarly expresses the magnitude of the relationship at approximately 0.

However, when we bin the data, the relationship is exposed. The bin age 31-36 has a 0.25 correlation, which is quite high for real data. This indicates predictive power. Below 25 at -0.18 is negatively correlated, and likewise above 46 is negatively correlated. This tells the story that the 30-36 age range is the most likely group to purchase products.

We can confirm the relationship visually using plot_correlation_funnel().

2. What About Skewed Numeric Data?

Highly skewed numeric data is tricky because the binning strategy does not conform well to the traditional quantile() approach for cutting data into bins. One bin tends to dominate.

To get around this issue, the binarize() function attempts to cut using the n_bins value provided by you.

When data becomes highly skewed, meaning one numeric value dominates so much that virtually all values are this value, the cuts are iteratively reduced until only 2 bins are left.

At that point the algorithm converts the values to factors, and anything that is not in the majority class gets a class of “OTHER” (or the value you provide as name_infreq).

We’ll show this process using skewed features in the marketing_campaign_tbl data that ships with correlationfunnel.

2.2. Highly Skewed Data

At some point, binning becomes impossible because only 2 values exist. Rather than drop the feature, binarize() converts it to a factor() and the one-hot encoding process takes over using the thresh_infreq argument for converting any low frequency factors to a generic “OTHER” category.

We can see that the “PDAYS” feature is highly skewed with almost all values are -1.

We can see the high skew visually.

We can see that binarize() converts this feature to categorical data then the categorical threshold takes over. Just specify the proportion of low frequency categories to retain.

3. How does the Correlation Funnel Work With Categorical Data?

We’ve actually already seen this. The binarization processed when performed on numeric data converts the data to bins and then one-hot encodes the result. The bins are categories.

The process for categorical data works the same as for numeric (minus the binning process).

Let’s take a categorical feature from our marketing_campaign_tbl.

3.1 One-Hot Encoding vs Dummy Encoding

The binarize() function uses One-Hot Encoding by default. The Dummy Encoding Method has a key flaw for Correlation Analysis in that it does not show all of the categorical levels. This affects the Correlation Funnel Plot causing loss of potentially highly correlated bins in the visualization.

One-Hot Encoding is the process of converting a categorical feature into a series of binary features. The default in binarize() is to perform one-hot encoding, which returns the number of columns equal to the number of levels in the category. This creates a series of binary flags to use in the Correlation Analysis.

Another popular method in statistical analysis is Dummy Encoding. The only real difference is that the number of columns are equal to the number of levels minus 1. When zeros are present in each of the new columns (secondary, tertiary, unknown) such as in Row 9, the value means not secondary, tertiary, or unknown, which in turn means primary.

3.2 Reducing Dimensionality (Preventing Irrelevant Factor Levels)

The binarize() function uses a parameter thresh_infreq = 0.01 and name_infreq = "OTHER" to keep infrequent categories that add little value to the analysis from unnecessarily increasing dimensionality.

Dimensionality Reduction is important for the correlation analysis. Categorical data can quickly get out of hand causing the width (number of columns) to increase, which is known as High Dimensionality. The One-Hot Encoding process will add a lot of features if not kept in check.

High Dimensionality is a two-fold problem. First, having many features adds to the computation time to analyze data. This is particularly painful on large data sets (5M rows+). Second, adding infrequent features (low occurrance) typically adds little predictive value to the modeling process.

To prevent this High Dimensionality situation, the binarize() function contains a thresh_infreq argument that lumps infrequent categories together based on a threshold of proportion within the rows of the data. For example, a thresh_infreq = 0.01 lumps any factors present in less than 1% of the data into a new category defined by name_infreq = "-OTHER" (you can change this name for the lumped category).

Let’s examine the “JOB” categorical feature from the marketing_campaign_tbl, which has 12 levels, one for the category of Job that the customer falls into. We can se that using binarize() creates 12 new features.

Examining these features, not all are present frequenly. We can see that several features appear only 5% or less of the time.

We can reduce the dimensionality of this feature by increasing the grouping to 6% using thresh_infreq = 0.06. Now the features with presence less than 6% are grouped into an “-OTHER” category.

3.3 Assessing Correlations With Categorical Data

Once the data has been One-Hot Encoded, Correlation Analysis proceeds as normal.

References

  1. [L. Duan et al., 2014] Lian Duan, W. Nick Street, Yanchi Liu, Songhua Xu, and Brook Wu. 2014. Selecting the right correlation measure for binary data. ACM Trans. Knowl. Discov. Data 9, 2, Article 13 (September 2014), 28 pages. DOI: http://dx.doi.org/10.1145/2637484↩