Making Sense of KPI Changes | Towards Data Science

, we’re often monitoring metrics. Very often, metrics change. And once they do, it’s our job to determine what’s happening: why did the conversion fee immediately drop, or what’s driving constant income development?

I began my journey in information analytics as a Kpi analyst. For nearly three years, I’d been doing root trigger evaluation and KPI deep dives practically full-time. Even after shifting to product analytics, I’m nonetheless repeatedly investigating the KPI shifts. You would say I’ve turn into fairly the skilled analytics detective.

The cornerstone of Root Cause Analysis is often slicing and dicing the info. Most frequently, determining what segments are driving the change provides you with a clue to the basis causes. So, on this article, I want to share a framework for estimating how totally different segments contribute to adjustments in your key metric. We’ll put collectively a set of capabilities to slice and cube our information and establish the primary drivers behind the metric’s adjustments.

Nonetheless, in actual life, earlier than leaping into information crunching, it’s essential to grasp the context: 

• Is the info full, and may we evaluate current durations to earlier ones?
• Are there any long-term developments and recognized seasonal results we’ve seen up to now?
• Have we launched something lately, or are we conscious of any exterior occasions affecting our metrics, similar to a competitor’s advertising marketing campaign or forex fluctuations?

I’ve mentioned such nuances in additional element in my earlier article, “Root Cause Analysis 101”.

KPI change framework

We encounter totally different metrics, and analysing their adjustments requires totally different approaches. Let’s begin by defining the 2 sorts of metrics we will likely be working with:

• Easy metrics signify a single measure, for instance, whole income or the variety of energetic customers. Regardless of their simplicity, they’re typically utilized in product analytics. One of many widespread examples is the North Star metrics. Good North Star metric estimates the overall worth obtained by prospects. For instance, AirBnB would possibly use nights booked, and WhatsApp would possibly observe messages despatched. Each are easy metrics.

You’ll be able to be taught extra about North Star Metrics from the Amplitude Playbook.

• Nonetheless, we are able to’t keep away from utilizing compound or ratio metrics, like conversion fee or common income per person (ARPU). Such metrics assist us observe our product efficiency extra exactly and isolate the affect of particular adjustments. For instance, think about your crew is engaged on enhancing the registration web page. They will probably observe the variety of registered prospects as their main KPI, however it is perhaps extremely affected by exterior elements (i.e., a advertising marketing campaign driving extra visitors). A greater metric for this case can be a conversion fee from touchdown on a registration web page to finishing it.

We’ll use a fictional instance to learn to strategy root trigger evaluation for several types of metrics. Think about we’re engaged on an e-commerce product, and our crew is concentrated on two principal KPIs: 

• whole income (a easy metric),
• conversion to buy — the ratio of customers who made a purchase order to the overall variety of customers (a ratio metric).

We’ll use artificial datasets to have a look at doable situations of metrics’ adjustments. Now it’s time to maneuver on and see what’s happening with the income.

Evaluation: easy metrics

Let’s begin easy and dig into the income adjustments. As traditional, step one is to load a dataset. Our information has two dimensions: nation and maturity (whether or not a buyer is new or present). Moreover, now we have three totally different situations to check our framework beneath numerous situations.

import pandas as pd
df = pd.read_csv('absolute_metrics_example.csv', sep = 't')
df.head()

The principle purpose of our evaluation is to find out how every phase contributes to the change in our top-line metric. Let’s break it down. We’ll write a bunch of formulation. However don’t fear, it received’t require any data past fundamental arithmetic.

To start with, it’s useful to see how the metric modified in every phase, each in absolute and relative numbers. 

[textbf{difference}^{textsf{i}} = textbf{metric}_{textsf{before}}^textsf{i} – textbf{metric}_{textsf{after}}^textsf{i}
textbf{difference_rate}^{textsf{i}} = frac{textbf{difference}^{textsf{i}}}{textbf{metric}_{textsf{before}}^textsf{i}}]

The following step is to have a look at it holistically and see how every phase contributed to the general change within the metric. We’ll calculate the affect because the share of the overall distinction.

[textbf{impact}^{textsf{i}} = frac{textbf{difference}^{textsf{i}}}{sum_{textsf{i}}{textbf{difference}^{textsf{i}}}}]

That already provides us some invaluable insights. Nonetheless, to grasp whether or not any phase is behaving unusually and requires particular consideration, it’s helpful to match the phase’s contribution to the metric change with its preliminary share of the metric. 

Right here’s the reasoning. If the phase makes up 90% of our metric, then it’s anticipated for it to contribute 85–95% of the change. But when a phase that accounts for less than 10% finally ends up contributing 90% of the change, that’s undoubtedly an anomaly.

To calculate it, we’ll merely normalise every phase’s contribution to the metric by the preliminary phase measurement. 

[textbf{segment_share}_{textsf{before}}^textsf{i} = frac{textbf{metric}_{textsf{before}}^textsf{i}}{sum_{textsf{i}}{textbf{metric}_{textsf{before}}^textsf{i}}}
textbf{impact_normalised}^textsf{i} = frac{textbf{impact}^{textsf{i}}}{textbf{segment_share}_{textsf{before}}^textsf{i}}]

That’s it for the formulation. Now, let’s write the code and see this strategy in follow. It is going to be simpler to grasp the way it works by way of sensible examples. 

def calculate_simple_growth_metrics(stats_df):
  # Calculating total stats
  earlier than = stats_df.earlier than.sum()
  after = stats_df.after.sum()
  print('Metric change: %.2f -> %.2f (%.2f%%)' % (earlier than, after, 100*(after - earlier than)/earlier than))

  # Estimating affect of every phase
  stats_df['difference'] = stats_df.after - stats_df.earlier than
  stats_df['difference_rate'] = (100*stats_df.distinction/stats_df.earlier than)
    .map(lambda x: spherical(x, 2))
  stats_df['impact'] = (100*stats_df.distinction / stats_df.distinction.sum())
    .map(lambda x: spherical(x, 2))
  stats_df['segment_share_before'] = (100* stats_df.earlier than / stats_df.earlier than.sum())
    .map(lambda x: spherical(x, 2))
  stats_df['impact_norm'] = (stats_df.affect/stats_df.segment_share_before)
    .map(lambda x: spherical(x, 2))

  # Creating visualisations
  create_parallel_coordinates_chart(stats_df.reset_index(), stats_df.index.title)
  create_share_vs_impact_chart(stats_df.reset_index(), stats_df.index.title, 'segment_share_before', 'affect')
  
  return stats_df.sort_values('impact_norm', ascending = False)

I consider that visualisations are an important a part of any information storytelling as visualisations assist viewers grasp insights extra shortly and intuitively. That’s why I’ve included a few charts in our perform: 

• A parallel coordinates chart to indicate how the metric modified in every slice — this visualisation will assist us see essentially the most important drivers in absolute phrases.
• A scatter plot to match every phase’s affect on the KPI with the phase’s preliminary measurement. This chart helps spot anomalies — segments whose affect on the KPI is disproportionately massive or small.

You could find the entire code for the visualisations on GitHub.

Now that now we have all of the instruments in place to analyse income information, let’s see how our framework performs in several situations.

State of affairs 1: Income dropped equally throughout all segments

Let’s begin with the primary situation. The evaluation may be very easy — we simply must name the perform outlined above.

calculate_simple_growth_metrics(
  df.groupby('nation')[['revenue_before', 'revenue_after_scenario_1']].sum()
    .sort_values('revenue_before', ascending = False).rename(
        columns = {'revenue_after_scenario_1': 'after', 
          'revenue_before': 'earlier than'}
    )
)

Within the output, we’ll get a desk with detailed stats.

Nonetheless, in my view, visualisations are extra informative. It’s apparent that income dropped by 30–40% in all international locations, and there are not any anomalies. 

State of affairs 2: A number of segments drove the change

Let’s take a look at one other situation by calling the identical perform.

calculate_simple_growth_metrics(
  df.groupby('nation')[['revenue_before', 'revenue_after_scenario_2']].sum()
    .sort_values('revenue_before', ascending = False).rename(
        columns = {'revenue_after_scenario_2': 'after', 
          'revenue_before': 'earlier than'}
    )
)

We are able to see the largest drop in each absolute and relative numbers in France. It’s undoubtedly an anomaly because it accounts for 99.9% of the overall metric change. We are able to simply spot this in our visualisations.

Additionally, it’s value going again to the primary instance. We seemed on the metric cut up by nation and located no particular segments driving adjustments. However digging a little bit bit deeper would possibly assist us perceive what’s happening. Let’s strive including one other layer and have a look at nation and maturity.

df['segment'] = df.nation + ' - ' + df.maturity 
calculate_simple_growth_metrics(
    df.groupby(['segment'])[['revenue_before', 'revenue_after_scenario_1']].sum()
        .sort_values('revenue_before', ascending = False).rename(
            columns = {'revenue_after_scenario_1': 'after', 'revenue_before': 'earlier than'}
        )
)

Now, we are able to see that the change is usually pushed by new customers throughout the international locations. These charts clearly spotlight points with the brand new buyer expertise and offer you a transparent path for additional investigation.

State of affairs 3: Quantity shifting between segments

Lastly, let’s discover the final situation for income.

calculate_simple_growth_metrics(
    df.groupby(['segment'])[['revenue_before', 'revenue_after_scenario_3']].sum()
        .sort_values('revenue_before', ascending = False).rename(
            columns = {'revenue_after_scenario_3': 'after', 'revenue_before': 'earlier than'}
        )
)

We are able to clearly see that France is the largest anomaly — income in France has dropped, and this alteration is correlated with the top-line income drop. Nonetheless, there may be one other excellent phase — Spain. In Spain, income has elevated considerably.

This sample raises a suspicion that a number of the income from France may need shifted to Spain. Nonetheless, we nonetheless see a decline within the top-line metric, so it’s value additional investigation. Virtually, this case may very well be attributable to information points, logging errors or service unavailability in some areas (so prospects have to make use of VPNs and seem with a unique nation in our logs).

We’ve checked out a bunch of various examples, and our framework helped us discover the primary drivers of change. I hope it’s now clear the right way to conduct root trigger evaluation with easy metrics, and we’re prepared to maneuver on to ratio metrics.

Evaluation: ratio metrics

Product metrics are sometimes ratios like common income per buyer or conversion. Let’s see how we are able to break down adjustments in such a metrics. In our case, we’ll have a look at conversion. 

There are two sorts of results to contemplate when analysing ratio metrics: 

• Change inside a phase, for instance, if buyer conversion in France drops, the general conversion may even drop.
• Change within the combine, for instance, if the share of latest prospects will increase, and new customers usually convert at a decrease fee, this shift within the combine also can result in a drop within the total conversion fee.

To grasp what’s happening, we’d like to have the ability to distinguish these results. As soon as once more, we’ll write a bunch of formulation to interrupt down and quantify every kind of affect. 

Let’s begin by defining some helpful variables.

[
textbf{c}_{textsf{before}}^{textsf{i}}, textbf{c}_{textsf{after}}^{textsf{i}} – textsf{converted users}
textbf{C}_{textsf{before}}^{textsf{total}} = sum_{textsf{i}}{textbf{c}_{textsf{before}}^{textsf{i}}}
textbf{C}_{textsf{after}}^{textsf{total}} = sum_{textsf{i}}{textbf{c}_{textsf{after}}^{textsf{i}}}
textbf{t}_{textsf{before}}^{textsf{i}}, textbf{t}_{textsf{after}}^{textsf{i}} – textsf{total users}
textbf{T}_{textsf{before}}^{textsf{total}} = sum_{textsf{i}}{textbf{t}_{textsf{before}}^{textsf{i}}}
textbf{T}_{textsf{after}}^{textsf{total}} = sum_{textsf{i}}{textbf{t}_{textsf{after}}^{textsf{i}}}
]

Subsequent, let’s discuss concerning the affect of the change in combine. To isolate this impact, we’ll estimate how the general conversion fee would change if conversion charges inside all segments remained fixed, and absolutely the numbers for each transformed and whole customers in all different segments stayed fastened. The one variables we’ll change are the overall and transformed variety of customers in phase i. We’ll regulate it to replicate its new share within the total inhabitants.

Let’s begin by calculating how the overall variety of customers in our phase wants to alter to match the goal phase share. 

[
frac{textbf{t}_{textsf{after}}^{textsf{i}}}{textbf{T}_{textsf{after}}^{textsf{total}}} = frac{textbf{t}_{textsf{before}}^{textsf{i}} + deltatextbf{t}^{textsf{i}}}{textbf{T}_{textsf{before}}^{textsf{total}}+ deltatextbf{t}^{textsf{i}}}
deltatextbf{t}^{textsf{i}} = frac{textbf{T}_{textsf{before}}^{textsf{total}} * textbf{t}_{textsf{after}}^{textsf{i}} – textbf{T}_{textsf{after}}^{textsf{total}} * textbf{t}_{textsf{before}}^{textsf{i}}}{textbf{T}_{textsf{after}}^{textsf{total}} – textbf{t}_{textsf{after}}^{textsf{i}}}
]

Now, we are able to estimate the change in combine affect utilizing the next formulation. 

[
textbf{change in mix impact} = frac{textbf{C}_{textsf{before}}^{textsf{total}} + deltatextbf{t}^{textsf{i}} * frac{textbf{c}_{textsf{before}}^{textsf{i}}}{textbf{t}_{textsf{before}}^{textsf{i}}}}{textbf{T}_{textsf{before}}^{textsf{total}} + deltatextbf{t}^{textsf{i}}} – frac{textbf{C}_{textsf{before}}^{textsf{total}}}{textbf{T}_{textsf{before}}^{textsf{total}}}
]

The following step is to estimate the affect of the conversion fee change inside phase i. To isolate this impact, we’ll hold the overall variety of prospects and transformed prospects in all different segments fastened. We’ll solely change the variety of transformed customers in phase i to match the goal conversion fee at a brand new level. 

[
textbf{change within segment impact} = frac{textbf{C}_{textsf{before}}^{textsf{total}} + textbf{t}_{textsf{before}}^{textsf{i}} * frac{textbf{c}_{textsf{after}}^{textsf{i}}}{textbf{t}_{textsf{after}}^{textsf{i}}} – textbf{c}_{textsf{before}}^{textsf{i}}}{textbf{T}_{textsf{before}}^{textsf{total}}} – frac{textbf{C}_{textsf{before}}^{textsf{total}}}{textbf{T}_{textsf{before}}^{textsf{total}}} = frac{textbf{t}_{textsf{before}}^{textsf{i}} * textbf{c}_{textsf{after}}^{textsf{i}} – textbf{t}_{textsf{after}}^{textsf{i}} * textbf{c}_{textsf{before}}^{textsf{i}}}{textbf{T}_{textsf{before}}^{textsf{total}} * textbf{t}_{textsf{after}}^{textsf{i}}}
]

We are able to’t merely sum the several types of results as a result of their relationship shouldn’t be linear. That’s why we additionally must estimate the mixed affect for the phase. It will mix the 2 formulation above, assuming that we’ll match each the brand new conversion fee inside phase i and the brand new phase share.

[
textbf{total segment change} = frac{textbf{C}_{textsf{before}}^{textsf{total}} – textbf{c}_{textsf{before}}^{textsf{i}} + (textbf{t}_{textsf{before}}^{textsf{i}} + deltatextbf{t}^{textsf{i}}) * frac{textbf{c}_{textsf{after}}^{textsf{i}}}{textbf{t}_{textsf{after}}^{textsf{i}}}}{textbf{T}_{textsf{before}}^{textsf{total}} + deltatextbf{t}^{textsf{i}}} – frac{textbf{C}_{textsf{before}}^{textsf{total}}}{textbf{T}_{textsf{before}}^{textsf{total}}}
]

It’s value noting that these impact estimations will not be 100% correct (i.e. we are able to’t sum them up instantly). Nonetheless, they’re exact sufficient to make choices and establish the primary drivers of the change.

The following step is to place the whole lot into code. We’ll once more leverage visualisations: correlation and parallel coordinates charts that we’ve already used for easy metrics, together with a few waterfall charts to interrupt down affect by segments.

def calculate_conversion_effects(df, dimension, numerator_field1, denominator_field1, 
                       numerator_field2, denominator_field2):
  cmp_df = df.groupby(dimension)[[numerator_field1, denominator_field1, numerator_field2, denominator_field2]].sum()
  cmp_df = cmp_df.rename(columns = {
      numerator_field1: 'c1', 
      numerator_field2: 'c2',
      denominator_field1: 't1', 
      denominator_field2: 't2'
  })
    
  cmp_df['conversion_before'] = cmp_df['c1']/cmp_df['t1']
  cmp_df['conversion_after'] = cmp_df['c2']/cmp_df['t2']
  
  C1 = cmp_df['c1'].sum()
  T1 = cmp_df['t1'].sum()
  C2 = cmp_df['c2'].sum()
  T2 = cmp_df['t2'].sum()

  print('conversion earlier than = %.2f' % (100*C1/T1))
  print('conversion after = %.2f' % (100*C2/T2))
  print('whole conversion change = %.2f' % (100*(C2/T2 - C1/T1)))
  
  cmp_df['dt'] = (T1*cmp_df.t2 - T2*cmp_df.t1)/(T2 - cmp_df.t2)
  cmp_df['total_effect'] = (C1 - cmp_df.c1 + (cmp_df.t1 + cmp_df.dt)*cmp_df.conversion_after)/(T1 + cmp_df.dt) - C1/T1
  cmp_df['mix_change_effect'] = (C1 + cmp_df.dt*cmp_df.conversion_before)/(T1 + cmp_df.dt) - C1/T1
  cmp_df['conversion_change_effect'] = (cmp_df.t1*cmp_df.c2 - cmp_df.t2*cmp_df.c1)/(T1 * cmp_df.t2)
  
  for col in ['total_effect', 'mix_change_effect', 'conversion_change_effect', 'conversion_before', 'conversion_after']:
      cmp_df[col] = 100*cmp_df[col]
        
  cmp_df['conversion_diff'] = cmp_df.conversion_after - cmp_df.conversion_before
  cmp_df['before_segment_share'] = 100*cmp_df.t1/T1
  cmp_df['after_segment_share'] = 100*cmp_df.t2/T2
  for p in ['before_segment_share', 'after_segment_share', 'conversion_before', 'conversion_after', 'conversion_diff',
                   'total_effect', 'mix_change_effect', 'conversion_change_effect']:
      cmp_df[p] = cmp_df[p].map(lambda x: spherical(x, 2))
  cmp_df['total_effect_share'] = 100*cmp_df.total_effect/(100*(C2/T2 - C1/T1))
  cmp_df['impact_norm'] = cmp_df.total_effect_share/cmp_df.before_segment_share

  # creating visualisations
  create_share_vs_impact_chart(cmp_df.reset_index(), dimension, 'before_segment_share', 'total_effect_share')
  cmp_df = cmp_df[['t1', 't2', 'before_segment_share', 'after_segment_share', 'conversion_before', 'conversion_after', 'conversion_diff',
                   'total_effect', 'mix_change_effect', 'conversion_change_effect', 'total_effect_share']]

  plot_conversion_waterfall(
      100*C1/T1, 100*C2/T2, cmp_df[['total_effect']].rename(columns = {'total_effect': 'impact'})
  )

  # placing collectively results cut up by change of combine and conversion change
  tmp = []
  for rec in cmp_df.reset_index().to_dict('data'): 
    tmp.append(
      {
          'phase': rec[dimension] + ' - change of combine',
          'impact': rec['mix_change_effect']
      }
    )
    tmp.append(
      {
        'phase': rec[dimension] + ' - conversion change',
        'impact': rec['conversion_change_effect']
      }
    )
  effects_det_df = pd.DataFrame(tmp)
  effects_det_df['effect_abs'] = effects_det_df.impact.map(lambda x: abs(x))
  effects_det_df = effects_det_df.sort_values('effect_abs', ascending = False) 
  top_effects_det_df = effects_det_df.head(5).drop('effect_abs', axis = 1)
  plot_conversion_waterfall(
    100*C1/T1, 100*C2/T2, top_effects_det_df.set_index('phase'),
    add_other = True
  )

  create_parallel_coordinates_chart(cmp_df.reset_index(), dimension, before_field='before_segment_share', 
    after_field='after_segment_share', impact_norm_field = 'impact_norm', 
    metric_name = 'share of phase', show_mean = False)
  create_parallel_coordinates_chart(cmp_df.reset_index(), dimension, before_field='conversion_before', 
    after_field='conversion_after', impact_norm_field = 'impact_norm', 
    metric_name = 'conversion', show_mean = False)

  return cmp_df.rename(columns = {'t1': 'total_before', 't2': 'total_after'})

With that, we’re accomplished with the speculation and able to apply this framework in follow. We’ll load one other dataset that features a few situations.

conv_df = pd.read_csv('conversion_metrics_example.csv', sep = 't')
conv_df.head()

State of affairs 1: Uniform conversion uplift

We’ll once more simply name the perform above and analyse the outcomes.

calculate_conversion_effects(
    conv_df, 'nation', 'converted_users_before', 'users_before', 
    'converted_users_after_scenario_1', 'users_after_scenario_1',
)

The primary situation is fairly easy: conversion has elevated in all international locations by 4–7% factors, ensuing within the top-line conversion improve as nicely. 

We are able to see that there are not any anomalies in segments: the affect is correlated with the phase share, and conversion has elevated uniformly throughout all international locations. 

We are able to have a look at the waterfall charts to see the change cut up by international locations and sorts of results. Despite the fact that impact estimations will not be additive, we are able to nonetheless use them to match the impacts of various slices.

The prompt framework has been fairly useful. We had been capable of shortly work out what’s happening with the metrics.

State of affairs 2: Simpson’s paradox

Let’s check out a barely trickier case.

calculate_conversion_effects(
    conv_df, 'nation', 'converted_users_before', 'users_before', 
    'converted_users_after_scenario_2', 'users_after_scenario_2',
)

The story is extra difficult right here: 

• The share of UK customers has elevated whereas conversion on this phase has dropped considerably, from 74.9% to 34.8%.
• In all different international locations, conversion has elevated by 8–11% factors.

Unsurprisingly, the conversion change within the UK is the largest driver of the top-line metric decline.

Right here we are able to see an instance of non-linearity: 10% of results will not be defined by the present cut up. Let’s dig one stage deeper and add a maturity dimension. This reveals the true story: 

• Conversion has really elevated uniformly by round 10% factors in all segments, but the top-line metric has nonetheless dropped. 
• The principle cause is the rise within the share of latest customers within the UK, as these prospects have a considerably decrease conversion fee than common.

Right here is the cut up of results by segments.

This counterintuitive impact known as Simpson’s paradox. A basic instance of Simpson’s paradox comes from a 1973 examine on graduate faculty admissions at Berkeley. At first, it appeared like males had the next probability of getting in than ladies. Nonetheless, once they seemed on the departments individuals had been making use of to, it turned out ladies had been making use of to extra aggressive departments with decrease admission charges, whereas males tended to use to much less aggressive ones. Once they added division as a confounder, the info really confirmed a small however important bias in favour of ladies.

As all the time, visualisation can provide you a little bit of instinct on how this paradox works.

That’s it. We’ve realized the right way to break down the adjustments in ratio metrics.

You could find the entire code and information on GitHub.

Abstract

It’s been a protracted journey, so let’s shortly recap what we’ve lined on this article:

• We’ve recognized two main sorts of metrics: easy metrics (like income or variety of customers) and ratio metrics (like conversion fee or ARPU). 
• For every metric kind, we’ve realized the right way to break down the adjustments and establish the primary drivers. We’ve put collectively a set of capabilities that may show you how to discover the solutions with simply a few perform calls.

With this sensible framework, you’re now totally outfitted to conduct root trigger evaluation for any metric. Nonetheless, there may be nonetheless room for enchancment in our answer. In my subsequent article, I’ll discover the right way to construct an LLM agent that can do the entire evaluation and abstract for us. Keep tuned!

Thank you numerous for studying this text. I hope this text was insightful for you.