Same data but greater insight: How one (secret) method yields more clarity

Bringing digital to life 23 February 2018 Thomas Lehman Jensen

The amount of data is exploding, and companies collect increasingly more data about their customers. But more data alone doesn't spell success. Success lies in getting greater insight about your stakeholders and that's something you can get without needing more data. Learn how in this expert column.

Expert columns
10 min

Big data, digitalisation, artificial intelligence are all terms in wide use these days. Regardless of which term is used, data is an inherent part of the terminology. But what’s relevant is when you convert this data into insight. That is how you can make a real difference. 

This article zooms in on a key stakeholder; customers. By this, the article provides you with a method to analyse purchase data more effectively, so you can get greater insight without needing to use more data. 

More of the right customers 

Companies have different strategies for generating their economic results. Many companies find it advantageous if customers make a certain minimum amount of purchases. When a customer makes many purchases, the overhead costs for that customer get covered and this makes earnings on the customer extra positive. This is the kind of customer that companies would like more of. 

Following one simple example we are about to see how a non-parametric regression analysis can provide insight that you would not immediately get through an ordinary regression analysis. First, let me briefly explain the key vocabulary: 

Regression analysis is a way to investigate the coherence between a dependent variable and other specifically independent variables. 

A non-parametric regression analysis refers to a category of regression analysis that does not require specific statistical distributions, which then gives the data ‘another chance to speak’. (chance to be used). This analysis is a secret to many, for others it perhaps needs some un-dusting. 

Deeper insight with the same data

Now, looking at the example: Imagine that you would like to investigate the connection between quantity of purchases and customer experience. This quantity is measured as the customer’s purchases within a certain period of time. For convenience, the customer experience is measured by two outcomes, detractors and attractors. 

The detractors indicate that the customer did not receive the kind of customer experience the company wanted to provide. The attractors indicate that the customer got exactly the kind of customer experience that the company wanted to provide. 

Table 1.

Results using traditional methods 

The results in Table 1 show that detractors make fewer purchases. On the average, detractors only make 94 purchases – and that is 34 fewer than the attractors. 

The difference of 34 purchases could also be identified through an ordinary regression analysis. In Table , you can see the results of a regression analysis where the number of purchases is regressed according to customer experiences. As shown, the number of purchases by the customer group that had disappointing customer experiences can be calculated as 128 - 34 = 94. 

Now we could end this analysis by concluding that it is important for a company to provide a good customer experience. The results are clear that there is an effect in the form of a loss of 34 purchases, if a company does not provide good customer experiences. But we can get even more out of this data.

Table 2. Descriptive analysis - Average.

Greater insight through a lesser known method 

With an ordinary regression analysis, we have reduced the question of effect to a question about finding a figure for the effect of the customer experience for an average amount of purchases. This answers the question: What is our expectation of the average amount of purchases for a group of customers who have had a positive customer experience? 

But can we expect effects of the same magnitude, if we look for other characteristics related to distribution in the number of purchases than just the quartile average? For example, would we see the same effect for 34 purchases, if we are interested in the upper quartile in number of purchases? For this, we need a non-parametric regression analysis.
Let us pursue the idea that the effects can differ, by looking at the percentiles for the number of purchases. This is shown in Table 3. 

Table 3.

In the table above, you can see that half of all customers made up to 109 purchases, while the other half made 109 or more purchases. This is shown by the median. In addition, we can see that the difference between the detractor and the attractor in the median value is 26, which is lower than the previously mentioned 34. 

The effect on the upper quartile shows a difference of 45 between the detractor and attractor, which is thus larger than 34. So, the effect is then 32% higher. This means that the ordinary regression analysis will underestimate the effect of good customer experiences towards increasing purchases, when the customer already makes lots of purchases. 

By creating positive customer experiences, this activates the top level, especially in this example, with an increase from 160 to 207. This is particularly interesting, if the customer is first profitable when reaching 150 purchases. 

In this way, using the same data but a different method can uncover more effects in the number of purchases. We no longer need to focus only on average results.

Use the right methods and avoid drowning in data 

The amount of data is growing at an explosive rate. This makes it more important than ever to avoid drowning in data. Different statistical methods can all contribute in different ways to help derive relevant insight.

In this article, we have focused on the “classical” regression analysis and the lesser-used non-parametrical regression analysis. These ensure that we do not derive too simple conclusions about the effects. The story about derived effects is enhanced through a simple method that gives far more nuanced insight into the data correlations, which may otherwise be very difficult to spot when looking at the data material.

Greater insight into critical KPI’s 

Just as an average measurement gives us an incomplete description of a distribution, a regression analysis provides an incomplete description of the effect. 

The non-parametrical regression is especially relevant when you want an understanding of what drives and influences the critical KPIs. People normally strive for high values on their KPIs and seldom for an average (read: mediocre) value. So, the non-parametrical regression contributes with a much clearer tale about the real value, without the need for more data.