Mining Rules from Data | Towards Data Science

with merchandise, we’d face a have to introduce some “guidelines”. Let me clarify what I imply by “guidelines” in sensible examples: 

• Think about that we’re seeing a large wave of fraud in our product, and we need to limit onboarding for a specific phase of shoppers to decrease this threat. For instance, we came upon that almost all of fraudsters had particular consumer brokers and IP addresses from sure international locations. 
• Another choice is to ship coupons to prospects to make use of in our on-line store. Nonetheless, we want to deal with solely prospects who’re more likely to churn since loyal customers will return to the product anyway. We would determine that essentially the most possible group is prospects who joined lower than a 12 months in the past and decreased their spending by 30%+ final month. 
• Transactional companies usually have a phase of shoppers the place they’re dropping cash. For instance, a financial institution buyer handed the verification and commonly reached out to buyer assist (so generated onboarding and servicing prices) whereas doing virtually no transactions (so not producing any income). The financial institution would possibly introduce a small month-to-month subscription payment for patrons with lower than 1000$ of their account since they’re seemingly non-profitable.

After all, in all these instances, we’d have used a fancy Machine Studying mannequin that may take note of all of the elements and predict the chance (both of a buyer being a fraudster or churning). Nonetheless, beneath some circumstances, we’d favor only a set of static guidelines for the next causes:  

• The pace and complexity of implementation. Deploying an ML mannequin in manufacturing takes effort and time. In case you are experiencing a fraud wave proper now, it could be extra possible to go reside with a set of static guidelines that may be applied shortly after which work on a complete answer. 
• Interpretability. ML fashions are black containers. Despite the fact that we’d be capable of perceive at a excessive stage how they work and what options are crucial ones, it’s difficult to clarify them to prospects. Within the instance of subscription charges for non-profitable prospects, it’s vital to share a set of clear guidelines with prospects in order that they will perceive the pricing. 
• Compliance. Some industries, like finance or healthcare, would possibly require auditable and rule-based choices to fulfill compliance necessities.

On this article, I need to present you ways we are able to resolve enterprise issues utilizing such guidelines. We’ll take a sensible instance and go actually deep into this matter:

• we are going to talk about which fashions we are able to use to mine such guidelines from information,
• we are going to construct a Decision Tree Classifier from scratch to study the way it works,
• we are going to match the sklearn Resolution Tree Classifier mannequin to extract the foundations from the information,
• we are going to learn to parse the Resolution Tree construction to get the ensuing segments,
• lastly, we are going to discover completely different choices for class encoding, because the sklearn implementation doesn’t assist categorical variables.

We’ve got plenty of subjects to cowl, so let’s leap into it.

Case

As typical, it’s simpler to study one thing with a sensible instance. So, let’s begin by discussing the duty we might be fixing on this article. 

We’ll work with the Bank Marketing dataset (). This dataset comprises information in regards to the direct advertising and marketing campaigns of a Portuguese banking establishment. For every buyer, we all know a bunch of options and whether or not they subscribed to a time period deposit (our goal). 

Our enterprise objective is to maximise the variety of conversions (subscriptions) with restricted operational assets. So, we are able to’t name the entire consumer base, and we need to attain one of the best consequence with the assets now we have.

Step one is to take a look at the information. So, let’s load the information set.

import pandas as pd
pd.set_option('show.max_colwidth', 5000)
pd.set_option('show.float_format', lambda x: '%.2f' % x)

df = pd.read_csv('bank-full.csv', sep = ';')
df = df.drop(['duration', 'campaign'], axis = 1)
# eliminated columns associated to the present advertising and marketing marketing campaign, 
# since they introduce information leakage

df.head()

We all know quite a bit in regards to the prospects, together with private information (similar to job kind or marital standing) and their earlier behaviour (similar to whether or not they have a mortgage or their common yearly steadiness).

The following step is to pick out a machine-learning mannequin. There are two courses of fashions which might be often used once we want one thing simply interpretable:

• determination bushes,
• linear or logistic regression.

Each choices are possible and may give us good fashions that may be simply applied and interpreted. Nonetheless, on this article, I want to follow the choice tree mannequin as a result of it produces precise guidelines, whereas logistic regression will give us chance as a weighted sum of options.

Knowledge Preprocessing 

As we’ve seen within the information, there are many categorical variables (similar to training or marital standing). Sadly, the sklearn determination tree implementation can’t deal with categorical information, so we have to do some preprocessing.

Let’s begin by reworking sure/no flags into integers. 

for p in ['default', 'housing', 'loan', 'y']:
    df[p] = df[p].map(lambda x: 1 if x == 'sure' else 0)

The following step is to rework the month variable. We will use one-hot encoding for months, introducing flags like month_jan , month_feb , and many others. Nonetheless, there could be seasonal results, and I believe it might be extra affordable to transform months into integers following their order. 

month_map = {
    'jan': 1, 'feb': 2, 'mar': 3, 'apr': 4, 'might': 5, 'jun': 6, 
    'jul': 7, 'aug': 8, 'sep': 9, 'oct': 10, 'nov': 11, 'dec': 12
}
# I saved 5 minutes by asking ChatGPT to do that mapping

df['month'] = df.month.map(lambda x: month_map[x] if x in month_map else x)

For all different categorical variables, let’s use one-hot encoding. We’ll talk about completely different methods for class encoding later, however for now, let’s follow the default method.

The best solution to do one-hot encoding is to leverage get_dummies function in pandas.

fin_df = pd.get_dummies(
  df, columns=['job', 'marital', 'education', 'poutcome', 'contact'], 
  dtype = int, # to transform to flags 0/1
  drop_first = False # to maintain all doable values
)

This perform transforms every categorical variable right into a separate 1/0 column for every doable. We will see the way it works for poutcome column. 

fin_df.merge(df[['id', 'poutcome']])
    .groupby(['poutcome', 'poutcome_unknown', 'poutcome_failure', 
      'poutcome_other', 'poutcome_success'], as_index = False).y.rely()
    .rename(columns = {'y': 'instances'})
    .sort_values('instances', ascending = False)

Our information is now prepared, and it’s time to debate how determination tree classifiers work.

Resolution Tree Classifier: Idea

On this part, we’ll discover the speculation behind the Resolution Tree Classifier and construct the algorithm from scratch. Should you’re extra keen on a sensible instance, be happy to skip forward to the subsequent half.

The best solution to perceive the choice tree mannequin is to take a look at an instance. So, let’s construct a easy mannequin based mostly on our information. We’ll use DecisionTreeClassifier from sklearn. 

feature_names = fin_df.drop(['y'], axis = 1).columns
mannequin = sklearn.tree.DecisionTreeClassifier(
  max_depth = 2, min_samples_leaf = 1000)
mannequin.match(fin_df[feature_names], fin_df['y'])

The following step is to visualise the tree.

dot_data = sklearn.tree.export_graphviz(
    mannequin, out_file=None, feature_names = feature_names, crammed = True, 
    proportion = True, precision = 2 
    # to point out shares of courses as a substitute of absolute numbers
)

graph = graphviz.Supply(dot_data)
graph

So, we are able to see that the mannequin is easy. It’s a set of binary splits that we are able to use as heuristics. 

Let’s determine how the classifier works beneath the hood. As typical, one of the simplest ways to know the mannequin is to construct the logic from scratch. 

The cornerstone of any downside is the optimisation perform. By default, within the determination tree classifier, we’re optimising the Gini coefficient. Think about getting one random merchandise from the pattern after which the opposite. The Gini coefficient would equal the chance of the scenario when these things are from completely different courses. So, our objective might be minimising the Gini coefficient. 

Within the case of simply two courses (like in our instance, the place advertising and marketing intervention was both profitable or not), the Gini coefficient is outlined simply by one parameter p , the place p is the chance of getting an merchandise from one of many courses. Right here’s the formulation:

[textbf{gini}(textsf{p}) = 1 – textsf{p}^2 – (1 – textsf{p})^2 = 2 * textsf{p} * (1 – textsf{p}) ]

If our classification is right and we’re capable of separate the courses completely, then the Gini coefficient might be equal to 0. The worst-case situation is when p = 0.5 , then the Gini coefficient can also be equal to 0.5.

With the formulation above, we are able to calculate the Gini coefficient for every leaf of the tree. To calculate the Gini coefficient for the entire tree, we have to mix the Gini coefficients of binary splits. For that, we are able to simply get a weighted sum:

[textbf{gini}_{textsf{total}} = textbf{gini}_{textsf{left}} * frac{textbf{n}_{textsf{left}}}{textbf{n}_{textsf{left}} + textbf{n}_{textsf{right}}} + textbf{gini}_{textsf{right}} * frac{textbf{n}_{textsf{right}}}{textbf{n}_{textsf{left}} + textbf{n}_{textsf{right}}}]

Now that we all know what worth we’re optimising, we solely have to outline all doable binary splits, iterate by means of them and select the best choice. 

Defining all doable binary splits can also be fairly easy. We will do it one after the other for every parameter, type doable values, and choose up thresholds between them. For instance, for months (integer from 1 to 12). 

Let’s attempt to code it and see whether or not we are going to come to the identical outcome. First, we are going to outline features that calculate the Gini coefficient for one dataset and the mixture.

def get_gini(df):
    p = df.y.imply()
    return 2*p*(1-p)

print(get_gini(fin_df)) 
# 0.2065
# near what we see on the root node of Resolution Tree

def get_gini_comb(df1, df2):
    n1 = df1.form[0]
    n2 = df2.form[0]

    gini1 = get_gini(df1)
    gini2 = get_gini(df2)
    return (gini1*n1 + gini2*n2)/(n1 + n2)

The following step is to get all doable thresholds for one parameter and calculate their Gini coefficients. 

import tqdm
def optimise_one_parameter(df, param):
    tmp = []
    possible_values = listing(sorted(df[param].distinctive()))
    print(param)

    for i in tqdm.tqdm(vary(1, len(possible_values))): 
        threshold = (possible_values[i-1] + possible_values[i])/2
        gini = get_gini_comb(df[df[param] <= threshold], 
          df[df[param] > threshold])
        tmp.append(
            {'param': param, 
            'threshold': threshold, 
            'gini': gini, 
            'sizes': (df[df[param] <= threshold].form[0], df[df[param] > threshold].form[0]))
            }
        )
    return pd.DataFrame(tmp)

The ultimate step is to iterate by means of all options and calculate all doable splits. 

tmp_dfs = []
for characteristic in feature_names:
    tmp_dfs.append(optimise_one_parameter(fin_df, characteristic))
opt_df = pd.concat(tmp_dfs)
opt_df.sort_values('gini', asceding = True).head(5)

Great, we’ve obtained the identical outcome as in our DecisionTreeClassifier mannequin. The optimum cut up is whether or not poutcome = success or not. We’ve lowered the Gini coefficient from 0.2065 to 0.1872. 

To proceed constructing the tree, we have to repeat the method recursively. For instance, happening for the poutcome_success <= 0.5 department:

tmp_dfs = []
for characteristic in feature_names:
    tmp_dfs.append(optimise_one_parameter(
      fin_df[fin_df.poutcome_success <= 0.5], characteristic))

opt_df = pd.concat(tmp_dfs)
opt_df.sort_values('gini', ascending = True).head(5)

The one query we nonetheless want to debate is the stopping standards. In our preliminary instance, we’ve used two circumstances:

• max_depth = 2 — it simply limits the utmost depth of the tree, 
• min_samples_leaf = 1000 prevents us from getting leaf nodes with lower than 1K samples. Due to this situation, we’ve chosen a binary cut up by contact_unknown regardless that age led to a decrease Gini coefficient.

Additionally, I often restrict the min_impurity_decrease that stop us from going additional if the positive factors are too small. By positive factors, we imply the lower of the Gini coefficient.

So, we’ve understood how the Resolution Tree Classifier works, and now it’s time to make use of it in observe.

Should you’re to see how Resolution Tree Regressor works in all element, you’ll be able to look it up in my previous article.

Resolution Timber: observe

We’ve already constructed a easy tree mannequin with two layers, but it surely’s undoubtedly not sufficient because it’s too easy to get all of the insights from the information. Let’s prepare one other Resolution Tree by limiting the variety of samples in leaves and reducing impurity (discount of Gini coefficient). 

mannequin = sklearn.tree.DecisionTreeClassifier(
  min_samples_leaf = 1000, min_impurity_decrease=0.001)
mannequin.match(fin_df[features], fin_df['y'])

dot_data = sklearn.tree.export_graphviz(
    mannequin, out_file=None, feature_names = options, crammed = True, 
    proportion = True, precision=2, impurity = True)

graph = graphviz.Supply(dot_data)

# saving graph to png file
png_bytes = graph.pipe(format='png')
with open('decision_tree.png','wb') as f:
    f.write(png_bytes)

That’s it. We’ve obtained our guidelines to separate prospects into teams (leaves). Now, we are able to iterate by means of teams and see which teams of shoppers we need to contact. Despite the fact that our mannequin is comparatively small, it’s daunting to repeat all circumstances from the picture. Fortunately, we are able to parse the tree structure and get all of the teams from the mannequin.

The Resolution Tree classifier has an attribute tree_ that may enable us to get entry to low-level attributes of the tree, similar to node_count .

n_nodes = mannequin.tree_.node_count
print(n_nodes)
# 13

The tree_ variable additionally shops the complete tree construction as parallel arrays, the place the i_th factor of every array shops the details about the node i. For the foundation i equals to 0.

Listed below are the arrays now we have to signify the tree construction: 

• children_left and children_right — IDs of left and proper nodes, respectively; if the node is a leaf, then -1.
• characteristic — characteristic used to separate the node i .
• threshold — threshold worth used for the binary cut up of the node i .
• n_node_samples — variety of coaching samples that reached the node i .
• values — shares of samples from every class.

Let’s save all these arrays. 

children_left = mannequin.tree_.children_left
# [ 1,  2,  3,  4,  5,  6, -1, -1, -1, -1, -1, -1, -1]
children_right = mannequin.tree_.children_right
# [12, 11, 10,  9,  8,  7, -1, -1, -1, -1, -1, -1, -1]
options = mannequin.tree_.characteristic
# [30, 34,  0,  3,  6,  6, -2, -2, -2, -2, -2, -2, -2]
thresholds = mannequin.tree_.threshold
# [ 0.5,  0.5, 59.5,  0.5,  6.5,  2.5, -2. , -2. , -2. , -2. , -2. , -2. , -2. ]
num_nodes = mannequin.tree_.n_node_samples
# [45211, 43700, 30692, 29328, 14165,  4165,  2053,  2112, 10000, 
#  15163,  1364, 13008,  1511] 
values = mannequin.tree_.worth
# [[[0.8830152 , 0.1169848 ]],
# [[0.90135011, 0.09864989]],
# [[0.87671054, 0.12328946]],
# [[0.88550191, 0.11449809]],
# [[0.8530886 , 0.1469114 ]],
# [[0.76686675, 0.23313325]],
# [[0.87043351, 0.12956649]],
# [[0.66619318, 0.33380682]],
# [[0.889     , 0.111     ]],
# [[0.91578184, 0.08421816]],
# [[0.68768328, 0.31231672]],
# [[0.95948647, 0.04051353]],
# [[0.35274653, 0.64725347]]]

It is going to be extra handy for us to work with a hierarchical view of the tree construction, so let’s iterate by means of all nodes and, for every node, save the mother or father node ID and whether or not it was a proper or left department. 

hierarchy = {}

for node_id in vary(n_nodes):
  if children_left[node_id] != -1: 
    hierarchy[children_left[node_id]] = {
      'mother or father': node_id, 
      'situation': 'left'
    }
  
  if children_right[node_id] != -1:
      hierarchy[children_right[node_id]] = {
       'mother or father': node_id, 
       'situation': 'proper'
  }

print(hierarchy)
# {1: {'mother or father': 0, 'situation': 'left'},
# 12: {'mother or father': 0, 'situation': 'proper'},
# 2: {'mother or father': 1, 'situation': 'left'},
# 11: {'mother or father': 1, 'situation': 'proper'},
# 3: {'mother or father': 2, 'situation': 'left'},
# 10: {'mother or father': 2, 'situation': 'proper'},
# 4: {'mother or father': 3, 'situation': 'left'},
# 9: {'mother or father': 3, 'situation': 'proper'},
# 5: {'mother or father': 4, 'situation': 'left'},
# 8: {'mother or father': 4, 'situation': 'proper'},
# 6: {'mother or father': 5, 'situation': 'left'},
# 7: {'mother or father': 5, 'situation': 'proper'}}

The following step is to filter out the leaf nodes since they’re terminal and essentially the most attention-grabbing for us as they outline the shopper segments. 

leaves = []
for node_id in vary(n_nodes):
    if (children_left[node_id] == -1) and (children_right[node_id] == -1):
        leaves.append(node_id)
print(leaves)
# [6, 7, 8, 9, 10, 11, 12]
leaves_df = pd.DataFrame({'node_id': leaves})

The following step is to find out all of the circumstances utilized to every group since they’ll outline our buyer segments. The primary perform get_condition will give us the tuple of characteristic, situation kind and threshold for a node. 

def get_condition(node_id, situation, options, thresholds, feature_names):
    # print(node_id, situation)
    characteristic = feature_names[features[node_id]]
    threshold = thresholds[node_id]
    cond = '>' if situation == 'proper'  else '<='
    return (characteristic, cond, threshold)

print(get_condition(0, 'left', options, thresholds, feature_names)) 
# ('poutcome_success', '<=', 0.5)

print(get_condition(0, 'proper', options, thresholds, feature_names))
# ('poutcome_success', '>', 0.5)

The following perform will enable us to recursively go from the leaf node to the foundation and get all of the binary splits. 

def get_decision_path_rec(node_id, decision_path, hierarchy):
  if node_id == 0:
    yield decision_path 
  else:
    parent_id = hierarchy[node_id]['parent']
    situation = hierarchy[node_id]['condition']
    for res in get_decision_path_rec(parent_id, decision_path + [(parent_id, condition)], hierarchy):
        yield res

decision_path = listing(get_decision_path_rec(12, [], hierarchy))[0]
print(decision_path) 
# [(0, 'right')]

fmt_decision_path = listing(map(
  lambda x: get_condition(x[0], x[1], options, thresholds, feature_names), 
  decision_path))
print(fmt_decision_path)
# [('poutcome_success', '>', 0.5)]

Let’s save the logic of executing the recursion and formatting right into a wrapper perform.

def get_decision_path(node_id, options, thresholds, hierarchy, feature_names):
  decision_path = listing(get_decision_path_rec(node_id, [], hierarchy))[0]
  return listing(map(lambda x: get_condition(x[0], x[1], options, thresholds, 
    feature_names), decision_path))

We’ve realized how one can get every node’s binary cut up circumstances. The one remaining logic is to mix the circumstances. 

def get_decision_path_string(node_id, options, thresholds, hierarchy, 
  feature_names):
  conditions_df = pd.DataFrame(get_decision_path(node_id, options, thresholds, hierarchy, feature_names))
  conditions_df.columns = ['feature', 'condition', 'threshold']

  left_conditions_df = conditions_df[conditions_df.condition == '<=']
  right_conditions_df = conditions_df[conditions_df.condition == '>']

  # deduplication 
  left_conditions_df = left_conditions_df.groupby(['feature', 'condition'], as_index = False).min()
  right_conditions_df = right_conditions_df.groupby(['feature', 'condition'], as_index = False).max()
  
  # concatination
  fin_conditions_df = pd.concat([left_conditions_df, right_conditions_df])
      .sort_values(['feature', 'condition'], ascending = False)
  
  # formatting 
  fin_conditions_df['cond_string'] = listing(map(
      lambda x, y, z: '(%s %s %.2f)' % (x, y, z),
      fin_conditions_df.characteristic,
      fin_conditions_df.situation,
      fin_conditions_df.threshold
  ))
  return ' and '.be part of(fin_conditions_df.cond_string.values)

print(get_decision_path_string(12, options, thresholds, hierarchy, 
  feature_names))
# (poutcome_success > 0.50)

Now, we are able to calculate the circumstances for every group. 

leaves_df['condition'] = leaves_df['node_id'].map(
  lambda x: get_decision_path_string(x, options, thresholds, hierarchy, 
  feature_names)
)

The final step is so as to add their measurement and conversion to the teams.

leaves_df['total'] = leaves_df.node_id.map(lambda x: num_nodes[x])
leaves_df['conversion'] = leaves_df['node_id'].map(lambda x: values[x][0][1])*100
leaves_df['converted_users'] = (leaves_df.conversion * leaves_df.complete)
  .map(lambda x: int(spherical(x/100)))
leaves_df['share_of_converted'] = 100*leaves_df['converted_users']/leaves_df['converted_users'].sum()
leaves_df['share_of_total'] = 100*leaves_df['total']/leaves_df['total'].sum()

Now, we are able to use these guidelines to make choices. We will type teams by conversion (chance of profitable contact) and choose the shoppers with the very best chance. 

leaves_df.sort_values('conversion', ascending = False)
  .drop('node_id', axis = 1).set_index('situation')

Think about now we have assets to contact solely round 10% of our consumer base, we are able to give attention to the primary three teams. Even with such a restricted capability, we’d count on to get virtually 40% conversion — it’s a extremely good outcome, and we’ve achieved it with only a bunch of easy heuristics.  

In actual life, it’s additionally value testing the mannequin (or heuristics) earlier than deploying it in manufacturing. I’d cut up the coaching dataset into coaching and validation components (by time to keep away from leakage) and see the heuristics efficiency on the validation set to have a greater view of the particular mannequin high quality.

Working with excessive cardinality classes

One other matter that’s value discussing on this context is class encoding, since now we have to encode the explicit variables for sklearn implementation. We’ve used an easy method with one-hot encoding, however in some instances, it doesn’t work.

Think about we even have a area within the information. I’ve synthetically generated English cities for every row. We’ve got 155 distinctive areas, so the variety of options has elevated to 190. 

mannequin = sklearn.tree.DecisionTreeClassifier(min_samples_leaf = 100, min_impurity_decrease=0.001)
mannequin.match(fin_df[feature_names], fin_df['y'])

So, the fundamental tree now has plenty of circumstances based mostly on areas and it’s not handy to work with them.

In such a case, it may not be significant to blow up the variety of options, and it’s time to consider encoding. There’s a complete article, “Categorically: Don’t explode — encode!”, that shares a bunch of various choices to deal with excessive cardinality categorical variables. I believe essentially the most possible ones in our case would be the following two choices:

• Depend or Frequency Encoder that reveals good efficiency in benchmarks. This encoding assumes that classes of comparable measurement would have comparable traits. 
• Goal Encoder, the place we are able to encode the class by the imply worth of the goal variable. It is going to enable us to prioritise segments with greater conversion and deprioritise segments with decrease. Ideally, it might be good to make use of historic information to get the averages for the encoding, however we are going to use the prevailing dataset. 

Nonetheless, it is going to be attention-grabbing to check completely different approaches, so let’s cut up our dataset into prepare and take a look at, saving 10% for validation. For simplicity, I’ve used one-hot encoding for all columns aside from area (because it has the very best cardinality).

from sklearn.model_selection import train_test_split
fin_df = pd.get_dummies(df, columns=['job', 'marital', 'education', 
  'poutcome', 'contact'], dtype = int, drop_first = False)
train_df, test_df = train_test_split(fin_df,test_size=0.1, random_state=42)
print(train_df.form[0], test_df.form[0])
# (40689, 4522)

For comfort, let’s mix all of the logic for parsing the tree into one perform.

def get_model_definition(mannequin, feature_names):
  n_nodes = mannequin.tree_.node_count
  children_left = mannequin.tree_.children_left
  children_right = mannequin.tree_.children_right
  options = mannequin.tree_.characteristic
  thresholds = mannequin.tree_.threshold
  num_nodes = mannequin.tree_.n_node_samples
  values = mannequin.tree_.worth

  hierarchy = {}

  for node_id in vary(n_nodes):
      if children_left[node_id] != -1: 
          hierarchy[children_left[node_id]] = {
            'mother or father': node_id, 
            'situation': 'left'
          }
    
      if children_right[node_id] != -1:
            hierarchy[children_right[node_id]] = {
             'mother or father': node_id, 
             'situation': 'proper'
            }

  leaves = []
  for node_id in vary(n_nodes):
      if (children_left[node_id] == -1) and (children_right[node_id] == -1):
          leaves.append(node_id)
  leaves_df = pd.DataFrame({'node_id': leaves})
  leaves_df['condition'] = leaves_df['node_id'].map(
    lambda x: get_decision_path_string(x, options, thresholds, hierarchy, feature_names)
  )

  leaves_df['total'] = leaves_df.node_id.map(lambda x: num_nodes[x])
  leaves_df['conversion'] = leaves_df['node_id'].map(lambda x: values[x][0][1])*100
  leaves_df['converted_users'] = (leaves_df.conversion * leaves_df.complete).map(lambda x: int(spherical(x/100)))
  leaves_df['share_of_converted'] = 100*leaves_df['converted_users']/leaves_df['converted_users'].sum()
  leaves_df['share_of_total'] = 100*leaves_df['total']/leaves_df['total'].sum()
  leaves_df = leaves_df.sort_values('conversion', ascending = False)
    .drop('node_id', axis = 1).set_index('situation')
  leaves_df['cum_share_of_total'] = leaves_df['share_of_total'].cumsum()
  leaves_df['cum_share_of_converted'] = leaves_df['share_of_converted'].cumsum()
  return leaves_df

Let’s create an encodings information body, calculating frequencies and conversions. 

region_encoding_df = train_df.groupby('area', as_index = False)
  .mixture({'id': 'rely', 'y': 'imply'}).rename(columns = 
    {'id': 'region_count', 'y': 'region_target'})

Then, merge it into our coaching and validation units. For the validation set, we may even fill NAs as averages.

train_df = train_df.merge(region_encoding_df, on = 'area')

test_df = test_df.merge(region_encoding_df, on = 'area', how = 'left')
test_df['region_target'] = test_df['region_target']
  .fillna(region_encoding_df.region_target.imply())
test_df['region_count'] = test_df['region_count']
  .fillna(region_encoding_df.region_count.imply())

Now, we are able to match the fashions and get their constructions.

count_feature_names = train_df.drop(
  ['y', 'id', 'region_target', 'region'], axis = 1).columns
target_feature_names = train_df.drop(
  ['y', 'id', 'region_count', 'region'], axis = 1).columns
print(len(count_feature_names), len(target_feature_names))
# (36, 36)

count_model = sklearn.tree.DecisionTreeClassifier(min_samples_leaf = 500, 
  min_impurity_decrease=0.001)
count_model.match(train_df[count_feature_names], train_df['y'])

target_model = sklearn.tree.DecisionTreeClassifier(min_samples_leaf = 500, 
  min_impurity_decrease=0.001)
target_model.match(train_df[target_feature_names], train_df['y'])

count_model_def_df = get_model_definition(count_model, count_feature_names)
target_model_def_df = get_model_definition(target_model, target_feature_names)

Let’s have a look at the constructions and choose the highest classes as much as 10–15% of our audience. We will additionally apply these circumstances to our validation units to check our method in observe. 

Let’s begin with Depend Encoder. 

count_selected_df = test_df[
    (test_df.poutcome_success > 0.50) | 
    ((test_df.poutcome_success <= 0.50) & (test_df.age > 60.50)) | 
    ((test_df.region_count > 3645.50) & (test_df.region_count <= 8151.50) & 
         (test_df.poutcome_success <= 0.50) & (test_df.contact_cellular > 0.50) & (test_df.age <= 60.50))
]

print(count_selected_df.form[0], count_selected_df.y.sum())
# (508, 227)

We will additionally see what areas have been chosen, and it’s solely Manchester.

Let’s proceed with the Goal encoding. 

target_selected_df = test_df[
    ((test_df.region_target > 0.21) & (test_df.poutcome_success > 0.50)) | 
    ((test_df.region_target > 0.21) & (test_df.poutcome_success <= 0.50) & (test_df.month <= 6.50) & (test_df.housing <= 0.50) & (test_df.contact_unknown <= 0.50)) | 
    ((test_df.region_target > 0.21) & (test_df.poutcome_success <= 0.50) & (test_df.month > 8.50) & (test_df.housing <= 0.50) 
         & (test_df.contact_unknown <= 0.50)) |
    ((test_df.region_target <= 0.21) & (test_df.poutcome_success > 0.50)) |
    ((test_df.region_target > 0.21) & (test_df.poutcome_success <= 0.50) & (test_df.month > 6.50) & (test_df.month <= 8.50) 
         & (test_df.housing <= 0.50) & (test_df.contact_unknown <= 0.50))
]

print(target_selected_df.form[0], target_selected_df.y.sum())
# (502, 248)

We see a barely decrease variety of chosen customers for communication however a considerably greater variety of conversions: 248 vs. 227 (+9.3%).

Let’s additionally have a look at the chosen classes. We see that the mannequin picked up all of the cities with excessive conversions (Manchester, Liverpool, Bristol, Leicester, and New Fortress), however there are additionally many small areas with excessive conversions solely because of likelihood.

region_encoding_df[region_encoding_df.region_target > 0.21]
  .sort_values('region_count', ascending = False)

In our case, it doesn’t affect a lot because the share of such small cities is low. Nonetheless, when you’ve got far more small classes, you would possibly see important drawbacks of overfitting. Goal Encoding could be tough at this level, so it’s value maintaining a tally of the output of your mannequin. 

Fortunately, there’s an method that may assist you to overcome this subject. Following the article “Encoding Categorical Variables: A Deep Dive into Target Encoding”, we are able to add smoothing. The thought is to mix the group’s conversion charge with the general common: the bigger the group, the extra weight its information carries, whereas smaller segments will lean extra in the direction of the worldwide common.

First, I’ve chosen the parameters that make sense for our distribution, taking a look at a bunch of choices. I selected to make use of the worldwide common for the teams beneath 100 folks. This half is a bit subjective, so use frequent sense and your information in regards to the enterprise area.

import numpy as np
import matplotlib.pyplot as plt

global_mean = train_df.y.imply()

ok = 100
f = 10
smooth_df = pd.DataFrame({'region_count':np.arange(1, 100001, 1) })
smooth_df['smoothing'] = (1 / (1 + np.exp(-(smooth_df.region_count - ok) / f)))

ax = plt.scatter(smooth_df.region_count, smooth_df.smoothing)
plt.xscale('log')
plt.ylim([-.1, 1.1])
plt.title('Smoothing')

Then, we are able to calculate, based mostly on the chosen parameters, the smoothing coefficients and blended averages.

region_encoding_df['smoothing'] = (1 / (1 + np.exp(-(region_encoding_df.region_count - ok) / f)))
region_encoding_df['region_target'] = region_encoding_df.smoothing * region_encoding_df.raw_region_target 
    + (1 - region_encoding_df.smoothing) * global_mean

Then, we are able to match one other mannequin with smoothed goal class encoding.

train_df = train_df.merge(region_encoding_df[['region', 'region_target']], 
  on = 'area')
test_df = test_df.merge(region_encoding_df[['region', 'region_target']], 
  on = 'area', how = 'left')
test_df['region_target'] = test_df['region_target']
  .fillna(region_encoding_df.region_target.imply())

target_v2_feature_names = train_df.drop(['y', 'id', 'region'], axis = 1)
  .columns

target_v2_model = sklearn.tree.DecisionTreeClassifier(min_samples_leaf = 500, 
  min_impurity_decrease=0.001)
target_v2_model.match(train_df[target_v2_feature_names], train_df['y'])
target_v2_model_def_df = get_model_definition(target_v2_model, 
  target_v2_feature_names)

target_v2_selected_df = test_df[
    ((test_df.region_target > 0.12) & (test_df.poutcome_success > 0.50)) | 
    ((test_df.region_target > 0.12) & (test_df.poutcome_success <= 0.50) & (test_df.month <= 6.50) & (test_df.housing <= 0.50) & (test_df.contact_unknown <= 0.50)) | 
    ((test_df.region_target > 0.12) & (test_df.poutcome_success <= 0.50) & (test_df.month > 8.50) & (test_df.housing <= 0.50) 
         & (test_df.contact_unknown <= 0.50)) | 
    ((test_df.region_target <= 0.12) & (test_df.poutcome_success > 0.50) ) | 
    ((test_df.region_target > 0.12) & (test_df.poutcome_success <= 0.50) & (test_df.month > 6.50) & (test_df.month <= 8.50) 
         & (test_df.housing <= 0.50) & (test_df.contact_unknown <= 0.50) )
]

target_v2_selected_df.form[0], target_v2_selected_df.y.sum()
# (500, 247)

We will see that we’ve eradicated the small cities and prevented overfitting in our mannequin whereas preserving roughly the identical efficiency, capturing 247 conversions.

region_encoding_df[region_encoding_df.region_target > 0.12]

You too can use TargetEncoder from sklearn, which smoothes and mixes the class and world means relying on the phase measurement. Nonetheless, it additionally provides random noise, which isn’t ultimate for our case of heuristics.

You’ll find the complete code on GitHub.

Abstract

On this article, we explored how one can extract easy “guidelines” from information and use them to tell enterprise choices. We generated heuristics utilizing a Resolution Tree Classifier and touched on the vital matter of categorical encoding since determination tree algorithms require categorical variables to be transformed.

We noticed that this rule-based method could be surprisingly efficient, serving to you attain enterprise choices shortly. Nonetheless, it’s value noting that this simplistic method has its drawbacks:

• We’re buying and selling off the mannequin’s energy and accuracy for its simplicity and interpretability, so should you’re optimising for accuracy, select one other method.
• Despite the fact that we’re utilizing a set of static heuristics, your information nonetheless can change, they usually would possibly turn into outdated, so it is advisable recheck your mannequin occasionally. 

---

Thank you a large number for studying this text. I hope it was insightful to you. When you have any follow-up questions or feedback, please depart them within the feedback part.

Reference

Dataset: Moro, S., Rita, P., & Cortez, P. (2014). Financial institution Advertising and marketing [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C5K306