Analyzing churn with chaid

Build the model

Now that we have the data organized the way we want it we can let CHAID do its’ thing and tell us what we want to know. What combination of our 13 predictor variables best explain or predict why our customers are leaving us (or as I mentioned before which customers are most likely to stay). The way it does that is devilishly simple and elegant. For all 13 predictors it runs a \(\chi^2\) test of independence (a.k.a. association) between the predictor and our outcome churn. Unlike some other tree models it can be a multi-way split. It will compute a p value for all these possible splits (see note above about ordinal versus nominal splits) and choose the split out of all possible splits with the smallest Bonferroni adjusted p value. That is the simplest explanation please see the ?chaid help page for complete details.

Just to break up the long passages of text and because I’m a visual learner let’s make a mosaic plot using ggmosaic of CUSCALL vs CHURN. We’ll even do a little magic to display the percentage of churn within the categories of CUSCALL. chaid will test not only this 4 x 2 table but also see if combining any adjacent categories is a better split.

p <- 
  ggplot(data = churn) + 
  geom_mosaic(aes(x = product(CUSCALL), fill = CHURN))

xxx <- ggplot_build(p)$data[[1]]

XXX <- xxx %>% 
  group_by_at(vars(ends_with("__CUSCALL"))) %>% 
  mutate(NN = sum(.wt)) %>% 
  mutate(pct = paste0(round(.wt/NN*100, 1), "%")) %>% 
  select(-(xmin:weight))

p + geom_text(data = xxx, 
              aes(x = (xmin + xmax)/2, 
                  y = (ymin + ymax)/2, 
                  label = XXX$pct)) + 
  labs(y = NULL, 
       x = "Number of Customer Calls",
       title = "Amount of Churn by # of Customer Calls") +
  scale_y_continuous(labels = scales::label_percent(accuracy = 1.0),
                     breaks = seq(from = 0, 
                                  to = 1, 
                                  by = 0.10),
                     minor_breaks = seq(from = 0.05, 
                                        to = 0.95, 
                                        by = 0.10))

Clearly, visually, something is going on but we’ll let the algorithm decide what the best splits are. It will keep on going until it runs out of significant splits or some other criteria we set such as the size of the bins or the number of levels we want in the model.

Since we’re pretending this is our first time through the data, and since we want clear, easy to understand, recommendations to give our business leaders to act on, we’re going to tell chaid to limit itself to just three “levels” of prediction. If you want a better understanding of what you can change to control the chaid model I suggest you go back to one of my earlier posts here.

Let’s build a “solution” and put it in an object called solution and then print and plot it using the built-in methods from partykit.

solution <- CHAID::chaid(CHURN ~ ., 
                         data = churn,
                         control = chaid_control(maxheight = 3))

print(solution)

## 
## Model formula:
## CHURN ~ ACLENGTH + INTPLAN + DATAPLAN + DATAGB + OMMIN + OMCALL + 
##     OTMIN + OTCALL + NGMIN + NGCALL + IMIN + ICALL + CUSCALL
## 
## Fitted party:
## [1] root
## |   [2] INTPLAN in No
## |   |   [3] CUSCALL in 0
## |   |   |   [4] OMMIN in Low: No (n = 206, err = 3.4%)
## |   |   |   [5] OMMIN in MedLow, Medium, MedHigh: No (n = 593, err = 7.4%)
## |   |   |   [6] OMMIN in High: No (n = 179, err = 15.6%)
## |   |   [7] CUSCALL in 1
## |   |   |   [8] OMMIN in Low, MedLow: No (n = 577, err = 6.4%)
## |   |   |   [9] OMMIN in Medium, MedHigh: No (n = 597, err = 10.9%)
## |   |   |   [10] OMMIN in High: No (n = 330, err = 19.1%)
## |   |   [11] CUSCALL in 2
## |   |   |   [12] OMMIN in Low, MedLow: No (n = 466, err = 8.4%)
## |   |   |   [13] OMMIN in Medium, MedHigh, High: No (n = 679, err = 19.0%)
## |   |   [14] CUSCALL in More than 2
## |   |   |   [15] OMMIN in Low, MedLow, Medium: No (n = 516, err = 20.0%)
## |   |   |   [16] OMMIN in MedHigh, High: No (n = 334, err = 36.8%)
## |   [17] INTPLAN in Yes
## |   |   [18] OMMIN in Low, MedLow
## |   |   |   [19] OMCALL in Low, MedLow, Medium: No (n = 186, err = 46.2%)
## |   |   |   [20] OMCALL in MedHigh, High: Yes (n = 32, err = 21.9%)
## |   |   [21] OMMIN in Medium, MedHigh, High
## |   |   |   [22] IMIN in Low, MedLow, Medium: No (n = 33, err = 45.5%)
## |   |   |   [23] IMIN in MedHigh, High: Yes (n = 272, err = 25.0%)
## 
## Number of inner nodes:     9
## Number of terminal nodes: 14

plot(solution,
  main = "churn dataset, maxheight = 3",
  gp = gpar(
    lty = "solid",
    lwd = 2,
    fontsize = 8
  ))

With an overall churn rate of about 20% we can see that even this simple chaid model gives us a much clearer picture of where our risks and opportunities lie. (A reminder that you can review how to interpret the print and plot output here.) We can identify hundreds of customers who pose little risk of changing carriers on the left side of the plot and conversely several high risk hotspots towards the right with the prediction for churn approaching 80%!

Makes sense at this point to check the accuracy of our model. if we were primarily interested in squeezing as much accuracy as possible we might follow the approach I demonstrate here but for a very simple first step I’m happy to sacrifice details and the risk of over-fitting for the simplicity of this design and the ease with which we can make suggestions for changing business practice.

caret::confusionMatrix(predict(solution), churn$CHURN)

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   No  Yes
##        No  3957  739
##        Yes   75  229
##                                           
##                Accuracy : 0.8372          
##                  95% CI : (0.8267, 0.8473)
##     No Information Rate : 0.8064          
##     P-Value [Acc > NIR] : 1.013e-08       
##                                           
##                   Kappa : 0.2948          
##                                           
##  Mcnemar's Test P-Value : < 2.2e-16       
##                                           
##             Sensitivity : 0.9814          
##             Specificity : 0.2366          
##          Pos Pred Value : 0.8426          
##          Neg Pred Value : 0.7533          
##              Prevalence : 0.8064          
##          Detection Rate : 0.7914          
##    Detection Prevalence : 0.9392          
##       Balanced Accuracy : 0.6090          
##                                           
##        'Positive' Class : No              
## 

As with the original blog post we can clearly tell our management team that additional focus is needed on our clients who have international calling plans.

More with what we have

I have to give partykit credit, the print and plot methods pack a lot of information into some efficient space. But they also leave me wanting for information, especially with regards to the “inner nodes”. To be fair, part of the problem is that chaid is a bit old and doesn’t take maximum advantage of partykit but regardless I’m a big fan of getting back an object I can do more analysis on.

That’s what I told myself when I started writing a function I called chaid_table(). It takes our solution object and converts it to a tibble that is chock full of information about our analysis/model.

A quick look at the first 5 rows should give you an idea of what’s in there. Hopefully nodeID, parent, NodeN, No, and Yes are obvious. Note that “no” and “yes” are actually pulled from the levels of the outcome variable so it will match your data. “ruletext” is a plain English summary of the complete rule to arrive at this node. “split.variable” is the variable that will be used to split the current node and produce child nodes.

“chisq” is obvious, “df” is the degrees of freedom, “adjustedp” is the p value after the bonferroni correction while “rawpvalue” is the uncorrected value. The split rule columns are the R code that would produce the split.

review_me <- CGPfunctions::chaid_table(solution)

kable(review_me[1:5, ])

The best part IMHO about this tibble format is that you can ask the data lots of additional questions in a dplyr pipeline. here are two obvious ones:

1. What percentage of customers are leaving if they have an international plan versus don’t? (14% versus 63%)

2. Can you provide an ordered list of where our churns are most likely to occur? (of course – allows us to make good business decisions. For example while node # 20 has the most churn at 78% there’s only 30 some people in that node while #23 has slightly less churn and a lot more people to influence.)

# Question #1 
review_me %>%
  select(nodeID:ruletext) %>%
  mutate(pctLeaving = Yes/NodeN * 100) %>% 
  filter(parent == 1) %>%
  kable(digits = 1, caption = "Question #1 answer")

# Question #2  
review_me %>%
  select(nodeID:split.variable) %>%
  mutate(pctLeaving = Yes/NodeN * 100) %>% 
  filter(is.na(split.variable)) %>%
  select(-parent, -split.variable) %>%
  arrange(desc(pctLeaving)) %>%
  kable(digits = 1, caption = "Question #2 answer")

Those are just a sample of what you can do with the data in a tibble. Feel free to experiment.

A plot is worth a thousand words

I’m a huge fan of displaying data graphically any time it can be done. I find it helps to drive home your messaging from data science projects. The other new function I added to my package recently is PlotXTabs2() which is built to display bivariate crosstabs. I borrowed heavily from ggstatsplot to allow you to optionally include statistical information.

Let’s use it to display information about the relationship between churn and our clients international plan. First some simple plots just displaying different x and y axis orientation and with and without pipelining. There are too many options to talk about here so please refer to the online pages.. My next project is to bring mosaic plots to the function.

CGPfunctions::PlotXTabs2(churn, 
                         CHURN, 
                         INTPLAN, 
                         bf.display = "sensible")

churn %>% CGPfunctions::PlotXTabs2(INTPLAN, 
                                   CHURN, 
                                   bf.display = "sensible")

The ability to pipe from the data to the graph is immensely useful for displaying practical data. The two final plots will focus on the second level of the model, where our customers have no international calling plan. Once again we’ll display the same data in reversed x/y order and in the second plot since there are more levels we’ll select a palette that is more friendly to users with potential sight challenges.

churn %>% 
  filter(INTPLAN == "No") %>% 
  CGPfunctions::PlotXTabs2(CUSCALL, 
                           CHURN, 
                           bf.display = "sensible")

churn %>% 
  filter(INTPLAN == "No") %>% 
  CGPfunctions::PlotXTabs2(CHURN, 
                           CUSCALL, 
                           bf.display = "sensible")

A final look at why it matters

So I wanted to end this post by reinforcing why chaid can be a useful tool even if it isn’t the newest algorithm or the coolest. Note that the accuracy wasn’t quite as good as some other methods but it is a fantastic tool for initial modelling. Note that it is also very robust and gives you answers that are easily explained to business leaders.

Done

Hope you enjoyed the post. Comments always welcomed. Especially please let me know if you actually use the tools and find them useful.

Chuck

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License