Explainable Anomaly Detection with RuleFit: An Intuitive Guide

your anomaly detection outcomes to your stakeholders, the fast subsequent query is at all times “why?”.

In observe, merely flagging an anomaly is never sufficient. Understanding what went mistaken is crucial to figuring out one of the best subsequent motion.

But, most machine learning-based anomaly detection strategies cease at producing an anomaly rating. They’re black-box in nature, which makes it painful to make sense of their outputs-why does this pattern have a better anomaly rating than its neighbors?

To deal with this explainability problem, you might have seemingly already resorted to fashionable eXplainable AI (XAI) strategies. Maybe you’re calculating characteristic significance to establish which variables are driving the abnormality, or you’re working counterfactual evaluation to see how shut a case was to regular.

These are helpful, however what when you might do extra? What when you can derive a set of interpretable IF-THEN guidelines that characterize the recognized anomalies?

That is precisely what the RuleFit algorithm [1] guarantees.

On this publish, we’ll discover how the RuleFit algorithm works intuitively, how it may be utilized to clarify detected anomalies, and stroll by means of a concrete case research.

1. How Does It Work?

Earlier than diving into the technical particulars, let’s first make clear what we purpose to have after making use of the algorithm: We wish to have a set of IF-THEN guidelines that quantitatively characterize the irregular samples, in addition to the significance of these guidelines.

To get there, we have to reply two questions:

(1) How will we generate significant IF-THEN situations from the information?

(2) How will we calculate the rule significance rating to find out which of them truly matter?

The RuleFit algorithm addresses these questions by splitting the work into two complementary elements, the “Rule” and the “Match”.

1.1 The “Rule” in RuleFit

In RuleFit, a rule appears like this:

IF x1 < 10 AND x2 > 5 THEN 1 ELSE 0

Would this construction look a bit extra acquainted if we visualize it like this:

Sure, it’s a resolution tree! The rule right here is simply traversing one particular path by means of the tree, from the foundation node to the leaf node.

In RuleFit, the rule era course of closely depends on constructing resolution bushes, which predict the goal consequence given the enter options. As soon as the tree is constructed, any path from the foundation to a node in a tree will be transformed to a call rule, as we now have simply seen within the instance above.

To make sure the principles are numerous, RuleFit doesn’t simply match one resolution tree. As an alternative, it leverages tree ensemble algorithms (e.g., random forest, Gradient Boosting bushes, and so forth.) to generate many alternative resolution bushes.

Additionally, the depths of these bushes are, basically, totally different. This brings the advantages of producing guidelines with variable lengths, additional enhancing the range.

Right here, we must always notice that though the ensemble bushes are constructed with predicting the goal consequence in thoughts, the RuleFit algorithm does not likely care in regards to the finish prediction outcomes. It merely makes use of this tree-building train because the automobile to extract significant, quantitative guidelines.

Successfully, which means that we’ll discard the expected worth in every node and solely hold the situations that lead us to a node. These situations produce the principles we care about.

Okay, we are able to now wrap up the primary processing step within the RuleFit algorithm: the rule constructing. The result of this step is a pool of candidate guidelines that would probably clarify the particular knowledge habits.

However out of all these guidelines, which of them truly deserve our consideration?

Nicely, that is the place the second step of RuleFit is available in. We “match” to rank.

1.2 The “Match” in RuleFit

Basically, RuleFit uncovers a very powerful guidelines through characteristic choice.

First, RuleFit treats every rule as a brand new binary characteristic, that’s, if the rule is happy for a particular pattern, it will get a price of 1 for this binary characteristic; in any other case, its worth is 0.

Then, RuleFit performs sparse linear regression with Lasso through the use of all of the “uncooked” options from the unique dataset, in addition to the newly engineered binary options derived from the principles, to foretell the goal consequence. This manner, every characteristic (uncooked options + binary rule options) will get a coefficient.

One key attribute of Lasso is that its loss operate forces the coefficients of these unimportant options to be precisely zero. This successfully means these unimportant options are faraway from the mannequin.

Consequently, by merely inspecting which binary rule options survived the Lasso evaluation, we might instantly know which guidelines are essential by way of getting correct predictions of the goal consequence. As well as, by trying on the coefficient magnitudes related to the rule options, we might have the ability to rank the significance of the principles.

1.3 Recap

We have now simply lined the important principle behind the RuleFit algorithm. To summarize, we are able to view this strategy as a two-step answer for offering explainability:

(1) It first extracts the principles by coaching an ensemble of resolution bushes. That’s the “Rule” half.

(2) It then cleverly converts these guidelines into binary options and performs normal characteristic choice through the use of sparse linear regression (Lasso). That’s the “Match” half.

Lastly, the surviving guidelines with non-zero coefficients are essential ones which can be value our consideration.

At this level, you might have seen that “predicting goal consequence” pops up at each the “Rule” and “Match” steps. If we’re coping with a regression or classification downside, it’s simply comprehensible that the “goal consequence” is the numerical worth or the label we wish to predict, and the principles will be interpreted as patterns that drive the prediction.

However what about anomaly detection, which is essentially an unsupervised process? How can we apply RuleFit there?

2. Anomaly Rationalization with RuleFit

2.1 Software Sample

To start with, we have to remodel the unsupervised explainability downside right into a supervised one. Right here’s how.

As soon as we now have our anomaly detection outcomes (doesn’t matter which algorithm we used), we are able to create binary labels, i.e., 1 for an recognized anomaly and 0 for a traditional knowledge level, as our “goal consequence.” This manner, we now have precisely what RuleFit wants: the uncooked options, and the goal consequence to foretell.

Then, the RuleFit can work its magic to generate a pool of candidate guidelines and match a sparse linear regression mannequin to retain solely the essential guidelines. The coefficients of the ensuing mannequin would then point out how a lot every rule contributes to the log-odds of an occasion being categorised as an anomaly. To place it one other manner, they inform us which rule combos most strongly push a pattern towards being labeled as anomalous.

Word you can, in principle, additionally use the anomaly rating (produced by the first anomaly detection mannequin) because the “goal consequence”. This may change the appliance of RuleFit from a classification setting to a regression setting.

Each approaches are legitimate, however they reply barely totally different questions: With the binary label classification setting, the RuleFit uncovers “What makes one thing an anomaly?“; With the anomaly rating regression setting, the RuleFit uncovers “What drives the severity of an anomaly?“.

In observe, the principles generated by each approaches will most likely be very related. Nonetheless, utilizing a binary anomaly label because the goal for a RuleFit is extra generally used for explaining detected anomalies. It’s simple by way of interpretation and direct applicability to creating enterprise guidelines for flagging future anomalies.

2.2 Case Examine

Let’s stroll by means of a concrete instance to see how RuleFit works in motion. Right here, we’ll create an anomaly detection situation utilizing the Iris dataset [2] (licensed CC BY 4.0), the place every pattern consists of 4 options (sepal_length, sepal_width, petal_length, petal_width) and is labeled as one of many following three classes: Setosa, Versicolor, and Virginica.

Step 1: Information Setup

First, we’ll use all Setosa samples (50) and all Versicolor samples (50) because the “regular” samples. For the “irregular” samples, we’ll use a subset of Virginica samples (10).

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, confusion_matrix
np.random.seed(42)

# Load the Iris dataset
iris = load_iris()
X = pd.DataFrame(iris.knowledge, columns=iris.feature_names)
y_true = iris.goal

# Get regular samples (Setosa + Versicolor)
normal_mask = (y_true == 0) | (y_true == 1)
X_normal_all = X[normal_mask].copy()

# Get Virginica samples
virginica_mask = (y_true == 2)
X_virginica = X[virginica_mask].copy()

# Randomly choose 10
anomaly_indices = np.random.selection(len(X_virginica), measurement=10, exchange=False)
X_anomalies = X_virginica.iloc[anomaly_indices].copy()

To make the situation extra reasonable, we create a separate coaching set and take a look at set. The practice set incorporates pure “regular” samples, whereas the take a look at set consists of randomly sampled 20 “regular” samples and 10 “irregular” samples.

train_indices = np.random.selection(len(X_normal_all), measurement=80, exchange=False)
test_indices = np.setdiff1d(np.arange(len(X_normal_all)), train_indices)

X_train = X_normal_all.iloc[train_indices].copy()
X_normal_test = X_normal_all.iloc[test_indices].copy()

# Create take a look at set (20 regular + 10 anomalous)
X_test = pd.concat([X_normal_test, X_anomalies], ignore_index=True)
y_test_true = np.concatenate([
    np.zeros(len(X_normal_test)),   
    np.ones(len(X_anomalies))       
])

Step 2: Anomaly Detection

Subsequent, we carry out anomaly detection. Right here, we fake we don’t know the precise labels. On this case research, we apply Native Outlier Issue (LOF) because the anomaly detection algorithm, which locates anomalies by measuring how remoted an information level is in comparison with the density of its native neighbors. After all, you too can attempt different anomaly detection algorithms, corresponding to Gaussian Combination Fashions (GMM), Ok-Nearest Neighbors (KNN), and Autoencoders, amongst others. Nevertheless, remember that the intention right here is simply to get the detection outcomes, our major focus is the anomaly rationalization in step 3.

Particularly, we’ll use the pyOD library to coach the mannequin and make inferences:

# Set up the pyOD library
#!pip set up pyod

from pyod.fashions.lof import LOF

# Standardize options
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.remodel(X_test)

# Native Outlier Issue
lof = LOF(n_neighbors=3)
lof.match(X_train_scaled)

train_scores = lof.decision_function(X_train_scaled)
test_scores = lof.decision_function(X_test_scaled)
threshold = np.percentile(train_scores_lof, 99)
y_pred = (test_scores > threshold).astype(int)

Discover that we now have used the 99% quantile of the anomaly scores obtained on the coaching set as the edge. For particular person take a look at samples, if its anomaly rating is greater than the edge, this pattern shall be labeled as “anomaly”. In any other case, the pattern is taken into account “regular”.

At this stage, we are able to rapidly verify the detection efficiency with:

classification_report(y_test_true, y_pred, target_names=['Normal', 'Anomaly'])

Not tremendous nice outcomes. Out of 10 true anomalies, solely 5 of them are caught. Nevertheless, the excellent news is that LOF didn’t produce any false positives. You’ll be able to additional enhance the efficiency by tuning the LOF mannequin hyperparameters, adjusting the edge, and even contemplating ensemble studying methods. However bear in mind: our aim right here is to not get one of the best detection accuracy. As an alternative, we purpose to see if RuleFit can correctly generate guidelines to clarify the anomalies detected by the LOF mannequin.

Step 3: Anomaly Rationalization

Now we’re attending to the core subject. To use RuleFit, let’s first set up the library from imodels, which is a sklearn-compatible, Interpretable ML package deal for concise, clear, and correct predictive modeling:

pip set up imodels

On this case, we’ll take into account a binary label classification setting, the place the irregular samples (within the take a look at set) flagged by the LOF mannequin are labeled as 1, and different un-flagged regular samples (additionally within the take a look at set) are labeled as 0. Word that we’re labeling based mostly on LOF’s detection outcomes, not the precise floor reality, which we fake we don’t know.

To provoke the RuleFit mannequin:

from imodels import RuleFitClassifier

rf = RuleFitClassifier(                 
        max_rules = 30,           
        lin_standardise=True,           
        include_linear=True,           
        random_state = 42
)

We are able to then proceed with becoming the RuleFit mannequin:

rf.match(
    X_test, 
    y_pred, 
    feature_names=X_test.columns
)

In observe, it’s often a great observe to do a fast sanity verify to judge how effectively the RuleFit mannequin’s predictions align with the anomaly labels decided by the LOF algorithm:

from sklearn.metrics import accuracy_score, roc_auc_score

y_label = rf.predict(X_test)               
y_prob  = rf.predict_proba(X_test)[:, 1]   

print("accuracy:", accuracy_score(y_pred, y_label))
print("roc-auc:", roc_auc_score (y_pred, y_prob))

For our case, we see that each printouts are 1. This confirms that the RuleFit mannequin has efficiently discovered the patterns that LOF used to establish anomalies. On your personal issues, when you observe values a lot decrease than 1, you would wish to fine-tune your RuleFit hyperparameters.

Now let’s study the principles:

guidelines = rf._get_rules()
guidelines = guidelines[rules.coef != 0]                         
guidelines = guidelines[~rules.type.str.contains('linear')]      
guidelines['abs_coef'] = guidelines['coef'].abs()
guidelines = guidelines.sort_values('significance', ascending=False)

The RuleFit algorithm returns a complete of 24 guidelines. A snapshot is proven under:

Let’s first make clear the which means of the outcomes columns:

• The “rule” column and the “abs_coef” column are self-explanatory.
• The “sort” column has two distinctive values: “linear” and “rule”. The “linear” denotes the unique enter options, whereas “rule” denotes the “IF-THEN” situations generated from resolution bushes.
• The “coef” column represents the coefficients produced by the Lasso regression evaluation. A optimistic worth signifies that if the rule applies, the log-odds of being categorised because the irregular class will increase. A bigger magnitude signifies a stronger affect of that rule on the prediction.
• The “assist” column data the fraction of information samples the place the rule applies.
• The “significance” column is calculated as absolutely the worth of the coefficient multiplied by the usual deviation of the binary (0 or 1) values that the rule takes on. So why this calculation? As we now have simply mentioned, a bigger absolute coefficient means a stronger direct affect on the log-odds. That’s clear. For the usual deviation time period, it successfully measures the “discriminative energy” of the principles. For instance, if a rule is nearly at all times TRUE (very small normal deviation), it doesn’t cut up your knowledge successfully. The identical holds if the rule is nearly at all times FALSE. In different phrases, the rule can not clarify a lot of the variation within the goal variable. Subsequently, the significance rating combines each the power of the rule’s affect (coefficient magnitude) and the way effectively it discriminates between totally different samples (normal deviation).

For our particular case, we see just one high-impact rule (Rule #24):

If a flower’s petal is longer than 5.45 cm and wider than 2 cm, the percentages that LOF classifies it as “anomalous” enhance 85-fold. (Word that exp(4.448999) ~= 85)

Guidelines #26 and #27 are nested inside Rule #24. That is widespread in observe, as RuleFit usually produces “households” of comparable guidelines as a result of they arrive from neighbouring tree splits. Subsequently, the one rule that actually issues for characterizing the LOF-identified anomalies is Rule #24.

Additionally, we see that the assist for Rule #24 is 0.1667 (5/30). This successfully signifies that all 5 LOF-identified anomalies will be defined by this rule. We are able to see that extra clearly within the determine under:

There you’ve gotten it: the rule to explain the recognized anomalies!

3. Conclusion

On this weblog publish, we explored the RuleFit algorithm as a strong answer for explainable anomaly detection. We mentioned:

• The way it works: A two-step strategy the place resolution bushes are first fitted to derive significant guidelines, adopted by a sparse linear regression to rank the rule significance.
• Find out how to apply to anomaly rationalization: Use the detection outcomes because the pseudo labels and use them because the “goal consequence” for the RuleFit mannequin.

With RuleFit in your modeling toolkit, the following time stakeholders ask “Why is that this anomaly?”, you’ll have concrete IF-THEN guidelines that they’ll perceive and act upon.

Reference

[1] Jerome H. Friedman, Bogdan E. Popescu, Predictive studying through rule ensembles, arXiv, 2008.

[2] Fisher, R. A., Iris [Data set]. UCI Machine Learning Repository, 1936.