Why Regularization Isn’t Enough: A Better Way to Train Neural Networks with Two Objectives | by Mehdi Yazdani | May, 2025

When coaching neural networks, we regularly juggle two competing aims. for instance, maximizing predictive efficiency whereas additionally assembly a secondary objective like equity, interpretability, or vitality effectivity. The default method is normally to fold the secondary goal into the loss operate as a weighted regularization time period. This one-size-fits-all loss may be easy to implement, but it surely isn’t at all times ultimate. In actual fact, analysis has proven that simply including a regularization time period can overlook advanced interdependencies between aims and result in suboptimal trade-offs.

Enter bilevel optimization a technique that treats the issue as two linked sub-problems (a frontrunner and a follower) as a substitute of a single blended goal. On this publish, we’ll discover why the naive regularization method can fall quick for multi-objective issues, and the way a bilevel formulation with devoted mannequin elements for every objective can considerably enhance each readability and convergence in observe. We’ll use examples past equity (like interpretability vs. efficiency, or domain-specific constraints in bioinformatics and robotics) as an instance the purpose. We’ll additionally dive into some precise code snippets from the open-source FairBiNN venture, which makes use of a bilevel technique for equity vs. accuracy, and talk about sensible issues from the unique paper together with its limitations in scalability, continuity assumptions, and challenges with attention-based fashions.

TL;DR: When you’ve been tuning weighting parameters to steadiness conflicting aims in your neural community, there’s a extra principled different. Bilevel optimization provides every goal its personal “area” (layers, parameters, even optimizer), yielding cleaner design and sometimes higher efficiency on the first activity all whereas assembly secondary targets to a Pareto-optimal diploma. Let’s see how and why this works.

Multi-objective studying — say you need excessive accuracy and low bias — is normally arrange as a single loss:

the place L secondary​ is a penalty time period (e.g., a equity or simplicity metric) and λ is a tunable weight. This Lagrangian method treats the issue as one huge optimization, mixing aims with a knob to tune. In concept, by adjusting λ you possibly can hint out a Pareto curve of options balancing the 2 targets. In observe, nonetheless, this method has a number of pitfalls:

• Selecting the Commerce-off is Difficult: The end result is extremely delicate to the burden λ. A slight change in λ can swing the answer from one excessive to the opposite. There isn’t any intuitive method to decide a “appropriate” worth with out intensive trial and error to discover a acceptable trade-off. This hyperparameter search is actually guide exploration of the Pareto frontier.
• Conflicting Gradients: With a mixed loss, the identical set of mannequin parameters is liable for each aims. The gradients from the first and secondary phrases would possibly level in reverse instructions. For instance, to enhance equity a mannequin would possibly want to regulate weights in a approach that hurts accuracy, and vice versa. The optimizer updates develop into a tug-of-war on the identical weights. This will result in unstable or inefficient coaching, because the mannequin oscillates attempting to fulfill each standards directly.
• Compromised Efficiency: As a result of the community’s weights should fulfill each aims concurrently, the first activity might be unduly compromised. You usually find yourself dialing again the mannequin’s capability to suit the information with a view to scale back the penalty. Certainly, we word {that a} regularization-based method could “overlook the advanced interdependencies” between the 2 targets. In plain phrases, a single weighted loss can gloss over how bettering one metric actually impacts the opposite. It’s a blunt instrument typically enhancements within the secondary goal come at an outsized expense of the first goal, or vice versa.
• Lack of Theoretical Ensures: The weighted-sum methodology will discover a answer, however there’s no assure it finds a Pareto-optimal one besides in particular convex circumstances. If the issue is non-convex (as neural community coaching normally is), the answer you converge to may be dominated by one other answer (i.e. one other mannequin might be strictly higher in a single goal with out being worse within the different). In actual fact, we confirmed a bilevel formulation can guarantee Pareto-optimal options underneath sure assumptions, with an higher certain on loss that’s no worse (and doubtlessly higher) than the Lagrangian method.

In abstract, including a penalty time period is commonly a blunt and opaque repair. Sure, it bakes the secondary goal into the coaching course of, but it surely additionally entangles the aims in a single black-box mannequin. You lose readability on how every goal is being dealt with, and also you may be paying extra in main efficiency than essential to fulfill the secondary objective.

Instance Pitfall: Think about a well being diagnostic mannequin that have to be correct and honest throughout demographics. A normal method would possibly add a equity penalty (say, the distinction in false optimistic charges between teams) to the loss. If this penalty’s weight (λ) is just too excessive, the mannequin would possibly practically equalize group outcomes however at the price of tanking general accuracy. Too low, and also you get excessive accuracy with unacceptable bias. Even with cautious tuning, the single-model method would possibly converge to a degree the place neither goal is de facto optimized: maybe the mannequin sacrifices accuracy greater than wanted with out totally closing the equity hole. The FairBiNN paper really proves that the bilevel methodology achieves an equal or decrease loss certain in comparison with the weighted method suggesting that the naive mixed loss can go away efficiency on the desk.

Bilevel optimization reframes the issue as a sport between two “gamers” usually referred to as the chief (upper-level) and follower (lower-level). As a substitute of mixing the aims, we assign every goal to a unique degree with devoted parameters (e.g., separate units of weights, and even separate sub-networks). Conceptually, it’s like having two fashions that work together: one solely focuses on the first activity, and the opposite solely focuses on the secondary activity, with an outlined order of optimization.

Within the case of two aims, the bilevel setup sometimes works as follows:

• Chief (Higher Stage): Optimizes the first loss (e.g., accuracy) with respect to its personal parameters, assuming that the follower will optimally reply for the secondary goal. The chief “leads” the sport by setting the circumstances (usually this simply means it is aware of the follower will do its job in addition to doable).
• Follower (Decrease Stage): Optimizes the secondary loss (e.g., equity or one other constraint) with respect to its personal parameters, in response to the chief’s selections. The follower treats the chief’s parameters as fastened (for that iteration) and tries to greatest fulfill the secondary goal.

This association aligns with a Stackelberg sport: the chief strikes first and the follower reacts. However in observe, we normally resolve it by alternating optimization: at every coaching iteration, we replace one set of parameters whereas holding the opposite fastened, after which vice versa. Over many iterations, this alternation converges to an equilibrium the place neither replace can enhance its goal a lot with out the opposite compensating. Ideally a Stackelberg equilibrium that can be Pareto-optimal for the joint drawback.

Crucially, every goal now has its personal “slot” within the mannequin. This will yield a number of sensible and theoretical benefits:

• Devoted Mannequin Capability: The first goal’s parameters are free to give attention to predictive efficiency, with out having to additionally account for equity/interpretability/and so on. In the meantime, the secondary goal has its personal devoted parameters to handle that objective. There’s much less inner competitors for representational capability. For instance, one can allocate a small subnetwork or a set of layers particularly to encode equity constraints, whereas the remainder of the community concentrates on accuracy.
• Separate Optimizers & Hyperparameters: Nothing says the 2 units of parameters have to be educated with the identical optimizer or studying price. In actual fact, FairBiNN makes use of totally different studying charges for the accuracy vs equity parameters (e.g. equity layers prepare with a smaller step measurement). You may even use totally totally different optimization algorithms if it is smart (SGD for one, Adam for the opposite, and so on.). This flexibility enables you to tailor the coaching dynamics to every goal’s wants. We spotlight that “the chief and follower can make the most of totally different community architectures, regularizers, optimizers, and so on. as greatest suited to every activity”, which is a strong freedom.
• No Extra Gradient Tug-of-Warfare: After we replace the first weights, we solely use the first loss gradient. The secondary goal doesn’t instantly pull on these weights (no less than not in the identical replace). Conversely, when updating the secondary’s weights, we solely have a look at the secondary loss. This decoupling means every goal could make progress by itself phrases, slightly than interfering in each gradient step. The result’s usually extra steady coaching. Because the FairBiNN paper places it, “the chief drawback stays a pure minimization of the first loss, with none regularization phrases that will gradual or hinder its progress”.
• Improved Commerce-off (Pareto Optimality): By explicitly modeling the interplay between the 2 aims in a leader-follower construction, bilevel optimization can discover higher balanced options than a naive weighted sum. Intuitively, the follower repeatedly fine-tunes the secondary goal for any given state of the first goal. The chief, anticipating this, can select a setting that provides one of the best main efficiency realizing the secondary might be taken care of as a lot as doable. Beneath sure mathematical circumstances (e.g. smoothness and optimum responses), one can show this yields Pareto-optimal options. In actual fact, a theoretical end result within the FairBiNN work reveals that if the bilevel method converges, it might obtain strictly higher primary-loss efficiency than the Lagrangian method in some circumstances. In different phrases, you would possibly get greater accuracy for a similar equity (or higher equity for a similar accuracy) in comparison with the standard penalty methodology.
• Readability and Interpretability of Roles: Architecturally, having separate modules for every goal makes the design extra interpretable to the engineers (if not essentially interpretable to end-users like mannequin explainability). You may level to a part of the community and say “this half handles the secondary goal.” This modularity improves transparency within the mannequin’s design. For instance, if in case you have a set of fairness-specific layers, you possibly can monitor their outputs or weights to know how the mannequin is adjusting to fulfill equity. If the trade-off wants adjusting, you would possibly tweak the scale or studying price of that subnetwork slightly than guessing a brand new loss weight. This separation of issues is analogous to good software program engineering observe every element has a single accountability. As one abstract of FairBiNN famous, “the bilevel framework enhances interpretability by clearly separating accuracy and equity aims”. Even past equity, this concept applies: a mannequin that balances accuracy and interpretability may need a devoted module to implement sparsity or monotonicity (making the mannequin extra interpretable), which is less complicated to purpose about than an opaque regularization time period.

To make this concrete, let’s have a look at how the Truthful Bilevel Neural Community (FairBiNN) implements these concepts for the equity (secondary) vs. accuracy (main) drawback. FairBiNN is a NeurIPS 2024 venture that demonstrated a bilevel coaching technique achieves higher equity/accuracy trade-offs than customary strategies. It’s an excellent case research in bilevel optimization utilized to neural nets.

FairBiNN’s mannequin is designed with two units of parameters: one set θa​ for accuracy-related layers, and one other set θf​ for fairness-related layers. These are built-in right into a single community structure, however logically you possibly can consider it as two sub-networks:

• The accuracy community (with weights θa​) produces the principle prediction (e.g., chance of the optimistic class).
• The equity community (with weights θf​) influences the mannequin in a approach that promotes equity (particularly group equity like demographic parity).

How are these mixed? FairBiNN inserts the fairness-focused layers at a sure level within the community. For instance, in an MLP for tabular information, you may need:

Enter → [Accuracy layers] → [Fairness layers] → [Accuracy layers] → Output

The --fairness_position parameter in FairBiNN controls the place the equity layers are inserted within the stack of layers. For example, --fairness_position 2 means after two layers of the accuracy subnetwork, the pipeline passes via the equity subnetwork, after which returns to the remaining accuracy layers. This kinds an “intervention level” the place the equity module can modulate the intermediate illustration to scale back bias, earlier than the ultimate prediction is made.

Let’s see a simplified code sketch (in PyTorch-like pseudocode) impressed by the FairBiNN implementation. This defines a mannequin with separate accuracy and equity elements:

import torch
import torch.nn as nn
class FairBiNNModel(nn.Module):
def __init__(self, input_dim, acc_layers, fairness_layers, fairness_position):
tremendous(FairBiNNModel, self).__init__()
# Accuracy subnetwork (earlier than equity)
acc_before_units = acc_layers[:fairness_position]      # e.g. first 2 layers
acc_after_units  = acc_layers[fairness_position:]      # remaining layers (together with output layer)
# Construct accuracy community (earlier than equity)
self.acc_before = nn.Sequential()
prev_dim = input_dim
for i, models in enumerate(acc_before_units):
self.acc_before.add_module(f"acc_layer{i+1}", nn.Linear(prev_dim, models))
self.acc_before.add_module(f"acc_act{i+1}", nn.ReLU())
prev_dim = models
# Construct equity community
self.fair_net = nn.Sequential()
for j, models in enumerate(fairness_layers):
self.fair_net.add_module(f"fair_layer{j+1}", nn.Linear(prev_dim, models))
if j < len(fairness_layers) - 1:
self.fair_net.add_module(f"fair_act{j+1}", nn.ReLU())
prev_dim = models
# Construct accuracy community (after equity)
self.acc_after = nn.Sequential()
for okay, models in enumerate(acc_after_units):
self.acc_after.add_module(f"acc_layer{fairness_position + okay + 1}", nn.Linear(prev_dim, models))
# If this isn't the ultimate output layer, add an activation
if okay < len(acc_after_units) - 1:
self.acc_after.add_module(f"acc_act{fairness_position + okay + 1}", nn.ReLU())
prev_dim = models
# Word: For binary classification, the ultimate output might be a single logit (no activation right here, use BCEWithLogitsLoss).
def ahead(self, x):
x = self.acc_before(x)      # go via preliminary accuracy layers
x = self.fair_net(x)        # go via equity layers (could rework illustration)
out = self.acc_after(x)     # go via remaining accuracy layers to get prediction
return out

On this construction, acc_before and acc_after collectively make up the accuracy-focused a part of the community (θa ​parameters), whereas fair_net accommodates the fairness-focused parameters (θf). The equity layers take the intermediate illustration and may push it in direction of a kind that yields honest outcomes. For example, these layers would possibly suppress info correlated with delicate attributes or in any other case alter the function distribution to reduce bias.

Why insert equity within the center? One purpose is that it provides the equity module a direct deal with on the mannequin’s realized illustration, slightly than simply post-processing outputs. By the point information flows via a few layers, the community has realized some options; inserting the equity subnetwork there means it might probably modify these options to take away biases (as a lot as doable) earlier than the ultimate prediction is made. The remaining accuracy layers then take this “de-biased” illustration and attempt to predict the label with out reintroducing bias.

Now, the coaching loop units up two optimizers one for θa and one for θf and alternates updates as described. Right here’s a schematic coaching loop illustrating the bilevel replace scheme:

mannequin = FairBiNNModel(input_dim=INPUT_DIM, 
acc_layers=[128, 128, 1],       # instance: 2 hidden layers of 128, then output layer
fairness_layers=[128, 128],    # instance: 2 hidden equity layers of 128 models every
fairness_position=2)
criterion = nn.BCEWithLogitsLoss()        # binary classification loss for accuracy
# Equity loss: we'll outline demographic parity distinction (particulars under)
# Separate parameter teams
acc_params = listing(mannequin.acc_before.parameters()) + listing(mannequin.acc_after.parameters())
fair_params = listing(mannequin.fair_net.parameters())
optimizer_acc = torch.optim.Adam(acc_params, lr=1e-3)
optimizer_fair = torch.optim.Adam(fair_params, lr=1e-5)  # word: smaller LR for equity
for epoch in vary(num_epochs):
for X_batch, y_batch, sensitive_attr in train_loader:
# Ahead go
logits = mannequin(X_batch)
# Compute main loss (e.g., accuracy loss)
acc_loss = criterion(logits, y_batch)
# Compute secondary loss (e.g., equity loss - demographic parity)
y_pred = torch.sigmoid(logits.detach())  # use indifferent logits for equity calc
# Demographic Parity: distinction in optimistic prediction charges between teams
group_mask = (sensitive_attr == 1)
pos_rate_priv  = y_pred[group_mask].imply()
pos_rate_unpriv = y_pred[~group_mask].imply()
fairness_loss = torch.abs(pos_rate_priv - pos_rate_unpriv)  # absolute distinction
# Replace accuracy (chief) parameters, preserve equity frozen
optimizer_acc.zero_grad()
acc_loss.backward(retain_graph=True)   # retain computation graph for equity backprop
optimizer_acc.step()
# Replace equity (follower) parameters, preserve accuracy frozen
optimizer_fair.zero_grad()
# Backprop equity loss via equity subnetwork solely
fairness_loss.backward()
optimizer_fair.step()

Just a few issues to notice on this coaching snippet:

• We separate acc_params and fair_params and provides every to its personal optimizer. Within the instance above, we selected Adam for each, however with totally different studying charges. This displays FairBiNN’s technique (they used 1e-3 vs 1e-5 for classifier vs equity layers on tabular information). The equity goal usually advantages from a smaller studying price to make sure steady convergence, because it’s optimizing a refined statistical property.
• We compute the accuracy loss (acc_loss) as traditional (binary cross-entropy on this case). The equity loss right here is illustrated because the demographic parity (DP) distinction – absolutely the distinction in optimistic prediction charges between the privileged and unprivileged teams. In observe, FairBiNN helps a number of equity metrics (like equalized odds as nicely) by plugging in several formulation for fairness_loss. The hot button is that this loss is differentiable with respect to the equity community’s parameters. We use logits.detach() to make sure the equity loss gradient doesn’t propagate again into the accuracy weights (solely into fair_net), holding with the concept that throughout equity replace, accuracy weights are handled as fastened.
• The order of updates proven is: replace accuracy weights first, then replace equity weights. This corresponds to treating accuracy because the chief (upper-level) and equity because the follower. Curiously, one would possibly assume equity (the constraint) ought to lead, however FairBiNN’s formulation units accuracy because the chief. In observe, it means we first take a step to enhance classification accuracy (with the present equity parameters held fastened), then we take a step to enhance equity (with the brand new accuracy parameters held fastened). This alternating process repeats. Every iteration, the equity participant is reacting to the newest state of the accuracy participant. In concept, if we might resolve the follower’s optimization preciselyfor every chief replace (e.g., discover the right equity parameters given present accuracy params), we’d be nearer to a real bilevel answer. In observe, doing one gradient step at a time in alternation is an efficient heuristic that step by step brings the system to equilibrium. (FairBiNN’s authors word that underneath sure circumstances, unrolling the follower optimization and computing a precise hypergradient for the chief can present ensures, however in implementation they use the less complicated alternating updates.)
• We name backward(retain_graph=True) on the accuracy loss as a result of we have to later backpropagate the equity loss via (a part of) the identical graph. The equity loss is determined by the mannequin’s predictions as nicely, which rely on each θaθa​ and θfθf​. By retaining the graph, we keep away from recomputing the ahead go for the equity backward go. (Alternatively, one might recompute logits after the accuracy step – the tip result’s related. FairBiNN’s code doubtless makes use of one ahead per batch and two backward passes, as proven above.)

Throughout coaching, you’ll see two gradients flowing: one into the accuracy layers (from acc_loss), and one into the equity layers (from fairness_loss). They’re stored separate. Over time, this could result in a mannequin the place θa​ has realized to foretell nicely provided that θf​ will regularly nudge the illustration in direction of equity, and θf has realized to mitigate bias given how θa​ likes to behave. Neither is having to instantly compromise its goal; as a substitute, they arrive at a balanced answer via this interaction.

Readability in observe: One rapid good thing about this setup is that it’s a lot clearer to diagnose and alter the conduct of every goal. If after coaching you discover the mannequin isn’t honest sufficient, you possibly can look at the equity community: maybe it’s underpowered (possibly too few layers or too low studying price) you can enhance its capability or coaching aggressiveness. Conversely, if accuracy dropped an excessive amount of, you would possibly notice the equity goal was overweighted (in bilevel phrases, possibly you gave it too many layers or a too-large studying price). These are high-level dials distinct from the first community. In a single community + reg time period method, all you had was the λ weight to tweak, and it wasn’t apparent why a sure λ failed (was the mannequin unable to signify a good answer, or did the optimizer get caught, or was it simply the improper trade-off?). Within the bilevel method, the division of labor is specific. This makes it extra sensible to undertake in actual engineering pipelines you possibly can assign groups to deal with the “equity module” or “security module” individually from the “efficiency module,” and so they can purpose about their element in isolation to some extent.

To provide a way of outcomes: FairBiNN, with this structure, was in a position to obtain Pareto-optimal fairness-accuracy trade-offs that dominated these from customary single-loss coaching of their experiments. In actual fact, underneath assumptions of smoothness and optimum follower response, they show any answer from their methodology won’t incur greater loss than the corresponding Lagrangian answer (and sometimes incurs much less on the first loss). Empirically, on datasets like UCI Grownup (revenue prediction) and Heritage Well being, the bilevel-trained mannequin had greater accuracy on the identical equity degree in comparison with fashions educated with a equity regularization time period. It basically bridged the accuracy-fairness hole extra successfully. And notably, this method didn’t include a heavy efficiency penalty in coaching time the authors reported “no tangible distinction within the common epoch time between the FairBiNN (bilevel) and Lagrangian strategies” when working on the identical information. In different phrases, splitting into two optimizers and networks doesn’t double your coaching time; due to fashionable librarie coaching per epoch was about as quick because the single-objective case.

Whereas FairBiNN showcases bilevel optimization within the context of equity vs. accuracy, the precept is broadly relevant. At any time when you will have two aims that partially battle, particularly if one is a domain-specific constraint or an auxiliary objective, a bilevel design might be useful. Listed below are just a few examples throughout totally different domains:

• Interpretability vs. Efficiency: In lots of settings, we search fashions which might be extremely correct but additionally interpretable (for instance, a medical diagnostic device that medical doctors can belief and perceive). Interpretability usually means constraints like sparsity (utilizing fewer options), monotonicity (respecting recognized directional relationships), or simplicity of the mannequin’s construction. As a substitute of baking these into one loss (which may be a fancy concoction of L1 penalties, monotonicity regularizers, and so on.), we might break up the mannequin into two components.
  Instance: The chief community focuses on accuracy, whereas a follower community might handle a masks or gating mechanism on enter options to implement sparsity. One implementation might be a small subnetwork that outputs function weights (or selects options) aiming to maximise an interpretability rating (like excessive sparsity or adherence to recognized guidelines), whereas the principle community takes the pruned options to foretell the end result. Throughout coaching, the principle predictor is optimized for accuracy given the present function choice, after which the feature-selection community is optimized to enhance interpretability (e.g., improve sparsity or drop insignificant options) given the predictor’s conduct. This mirrors how one would possibly do function choice by way of bilevel optimization (the place function masks indicators are realized as steady parameters in a lower-level drawback). The benefit is the predictor isn’t instantly penalized for complexity; It simply has to work with no matter options the interpretable half permits. In the meantime, the interpretability module finds the only function subset that the predictor can nonetheless do nicely on. Over time, they converge to a steadiness of accuracy vs simplicity. This method was hinted at in some meta-learning literature (treating function choice as an interior optimization). Virtually, it means we get a mannequin that’s simpler to clarify (as a result of the follower pruned it) with out an enormous hit to accuracy, as a result of the follower solely prunes as a lot because the chief can tolerate. If we had accomplished a single L1-regularized loss, we’d should tune the burden of L1 and would possibly both kill accuracy or not get sufficient sparsity! With bilevel, the sparsity degree adjusts dynamically to take care of accuracy.
• Robotics: Vitality or Security vs. Activity Efficiency: Contemplate a robotic that should carry out a activity shortly (efficiency goal) but additionally safely and effectively (secondary goal, e.g., decrease vitality utilization or keep away from dangerous maneuvers). These aims usually battle: the quickest trajectory may be aggressive on motors and fewer secure. A bilevel method might contain a main controller community that tries to reduce time or monitoring error (chief), and a secondary controller or modifier that adjusts the robotic’s actions to preserve vitality or keep inside security limits (follower). For example, the follower might be a community that provides a small corrective bias to the motion outputs or that adjusts the management features, with the objective of minimizing a measured vitality consumption or jerkiness. Throughout coaching (which might be in simulation), you’d alternate: prepare the principle controller on the duty efficiency given the present security/vitality corrections, then prepare the security/vitality module to reduce these prices given the controller’s conduct. Over time, the controller learns to perform the duty in a approach that the security module can simply tweak to remain secure, and the security module learns the minimal intervention wanted to satisfy constraints. The end result may be a trajectory that could be a bit slower than the unconstrained optimum however makes use of far much less vitality and also you achieved that with out having to fiddle with a single weighted reward that mixes time and vitality (a typical ache level in reinforcement studying reward design). As a substitute, every half had a transparent objective. In actual fact, this concept is akin to “shielding” in reinforcement studying, the place a secondary coverage ensures security constraints, however bilevel coaching would be taught the defend along side the first coverage.
• Bioinformatics: Area Constraints vs. Prediction Accuracy: In bioinformatics or computational biology, you would possibly predict outcomes (protein operate, gene expression, and so on.) but additionally need the mannequin to respect area data. For instance, you prepare a neural web to foretell illness danger from genetic information (main goal), whereas making certain the mannequin’s conduct aligns with recognized organic pathways or constraints (secondary goal). A concrete situation: possibly we wish the mannequin’s selections to rely on teams of genes that make sense collectively (pathways), not arbitrary combos, to help scientific interpretability and belief. We might implement a follower community that penalizes the mannequin if it makes use of gene groupings which might be nonsensical, or that encourages it to make the most of sure recognized biomarker genes. Bilevel coaching would let the principle predictor maximize predictive accuracy, after which a secondary “regulator” community might barely alter weights or inputs to implement the constraints (e.g., suppress indicators from gene interactions that shouldn’t matter biologically). Alternating updates would yield a mannequin that predicts nicely however, say, depends on biologically believable indicators. That is preferable to hard-coding these constraints or including a stiff penalty which may stop the mannequin from studying refined however legitimate indicators that deviate barely from recognized biology. Basically, the mannequin itself finds a compromise between data-driven studying and prior data, via the interaction of two units of parameters.

These examples are a bit speculative, however they spotlight a sample: every time you will have a secondary goal that might be dealt with by a specialised mechanism, take into account giving it its personal module and coaching it in a bilevel style. As a substitute of baking all the pieces into one monolithic mannequin, you get an structure with components corresponding to every concern.

Earlier than you rush to refactor all of your loss features into bilevel optimizations, it’s essential to know the constraints and necessities of this method. The FairBiNN paper — whereas very encouraging — is upfront about a number of caveats that apply to bilevel strategies:

• Continuity and Differentiability Assumptions: Bilevel optimization, particularly with gradient-based strategies, sometimes assumes the secondary goal in all fairness clean and differentiable with respect to the mannequin parameters. In FairBiNN’s concept, we assume issues like Lipschitz continuity of the neural community features and losses In plain phrases, the gradients shouldn’t be exploding or wildly erratic, and the follower’s optimum response ought to change easily because the chief’s parameters change. In case your secondary goal will not be differentiable (e.g., a tough constraint or a metric like accuracy which is piecewise-constant), you could must approximate it with a clean surrogate to make use of this method. FairBiNN particularly targeted on binary classification with a sigmoid output, avoiding the non-differentiability of the argmax in multi-class classification. In actual fact, we level out that the generally used softmax activation will not be Lipschitz steady, which “limits the direct utility of our methodology to multiclass classification issues”. This implies if in case you have many lessons, the present concept may not maintain and the coaching might be unstable until you discover a workaround (they counsel exploring different activations or normalization to implement Lipschitz continuity for multi-class settings). So, one caveat: bilevel works greatest when each aims are good clean features of the parameters. Discontinuous jumps or extremely non-convex aims would possibly nonetheless work heuristically, however the theoretical ensures evaporate.
• Consideration and Complicated Architectures: Fashionable deep studying fashions (like Transformers with consideration mechanisms) pose an additional problem. We name out that consideration layers are usually not Lipschitz steady both, which “presents a problem for extending our methodology to state-of-the-art architectures in NLP and different domains that closely depend on consideration.” we reference analysis making an attempt to make consideration Lipschitz (e.g., LipschitzNorm for self-attention (arxiv.org) ), however as of now, making use of bilevel equity to a Transformer can be non-trivial. The priority is that spotlight can amplify small modifications quite a bit, breaking the graceful interplay wanted for steady leader-follower updates. In case your utility makes use of architectures with elements like consideration or different non-Lipschitz operations, you would possibly have to be cautious. It doesn’t imply bilevel gained’t work, however the concept doesn’t instantly cowl it, and also you may need to empirically tune extra. We’d see future analysis addressing find out how to incorporate such elements (maybe by constraining or regularizing them to behave extra properly).
  Backside line: the present bilevel successes have been in comparatively simple networks (MLPs, easy CNNs, GCNs). Additional fancy architectures might require extra care.
• No Silver Bullet Ensures: Whereas the bilevel methodology can provably obtain Pareto-optimal options underneath the correct circumstances, that doesn’t robotically imply your mannequin is “completely honest” or “totally interpretable” on the finish. There’s a distinction between balancing aims optimally and satisfying an goal completely. FairBiNN’s concept gives ensures relative to one of the best trade-off (and relative to the Lagrangian methodology) it doesn’t assure absolute equity or zero bias. In our case, we nonetheless had residual bias, simply a lot much less for the accuracy we achieved in comparison with baselines. So, in case your secondary goal is a tough constraint (like “must not ever violate security situation X”), a gentle bilevel optimization may not be sufficient! you would possibly must implement it in a stricter approach or confirm the outcomes after coaching. Additionally, FairBiNN thus far dealt with one equity metric at a time (demographic parity in most experiments). In real-world situations, you would possibly care about a number of constraints (e.g., equity throughout a number of attributes, or equity and interpretability and accuracy a tri-objective drawback). Extending bilevel to deal with a number of followers or a extra advanced hierarchy is an open problem (it might develop into a multi-level or multi-follower sport). One thought might be to break down a number of metrics into one secondary goal (possibly as a weighted sum or some worst-case metric), however that reintroduces the weighting drawback internally. Alternatively, one might have a number of follower networks, every for a unique metric, and round-robin via them however concept and observe for that aren’t totally established.
• Hyperparameter Tuning and Initialization: Whereas we escape tuning λ in a direct sense, the bilevel method introduces different hyperparameters: the educational charges for every optimizer, the relative capability of the 2 subnetworks, possibly the variety of steps to coach follower vs chief, and so on. In FairBiNN’s case, we had to decide on the variety of equity layers and the place to insert them, in addition to the educational charges. These had been set based mostly on some instinct and a few held-out validation (e.g., we selected a really low LR for equity to make sure stability). Basically, you’ll nonetheless must tune these elements. Nevertheless, these are typically extra interpretable hyperparameters e.g., “how expressive is my equity module” is less complicated to purpose about than “what’s the correct weight for this ethereal equity time period.” In some sense, the architectural hyperparameters exchange the burden tuning. Additionally, initialization of the 2 components might matter; one heuristic might be pre-training the principle mannequin for a bit earlier than introducing the secondary goal (or vice versa), to provide a great start line. FairBiNN didn’t require a separate pre-training; we educated each from scratch concurrently. However which may not at all times be the case for different issues.

Regardless of these caveats, it’s price highlighting that the bilevel method is possible with as we speak’s instruments. The FairBiNN implementation was accomplished in PyTorch with customized coaching loops one thing most practitioners are comfy with and it’s out there on GitHub for reference (Github). The additional effort (writing a loop with two optimizers) is comparatively small contemplating the potential features in efficiency and readability. In case you have a essential utility with two competing metrics, the payoff might be important.

Optimizing neural networks with a number of aims will at all times contain trade-offs that’s inherent to the issue. However how we deal with these trade-offs is underneath our management. The standard knowledge of “simply throw it into the loss operate with a weight” usually leaves us wrestling with that weight and questioning if we might have accomplished higher. As we’ve mentioned, bilevel optimization affords a extra structured and principled method to deal with two-objective issues. By giving every goal its personal devoted parameters, layers, and optimization course of, we enable every objective to be pursued to the fullest extent doable with out being in perpetual battle with the opposite.

The instance of FairBiNN demonstrates that this method isn’t simply tutorial fancy it delivered state-of-the-art leads to equity/accuracy trade-offs, proving mathematically that it might probably match or beat the previous regularization method by way of the loss achieved. Extra importantly for practitioners, it did so with a reasonably simple implementation and affordable coaching value. The mannequin structure turned a dialog between two components: one making certain equity, the opposite making certain accuracy. This type of architectural transparency is refreshing in a area the place we regularly simply alter scalar knobs and hope for one of the best.

For these in ML analysis and engineering, the take-home message is: subsequent time you face a competing goal; be it mannequin interpretability, equity, security, latency, or area constraints take into account formulating it as a second participant in a bilevel setup. Design a module (nonetheless easy or advanced) dedicated to that concern, and prepare it in tandem along with your major mannequin utilizing an alternating optimization. You would possibly discover which you can obtain a greater steadiness and have a clearer understanding of your system. It encourages a extra modular design: slightly than entangling all the pieces into one opaque mannequin, you delineate which a part of the community handles what.

Virtually, adopting bilevel optimization requires cautious consideration to the assumptions and a few tuning of coaching procedures. It’s not a magic wand in case your secondary objective is essentially at odds with the first, there’s a restrict to how completely happy an equilibrium you possibly can attain. However even then, this method will make clear the character of the trade-off. In one of the best case, it finds win-win options that the single-objective methodology missed. Within the worst case, you no less than have a modular framework to iterate on.

As machine studying fashions are more and more deployed in high-stakes settings, balancing aims (accuracy with equity, efficiency with security, and so on.) turns into essential. The engineering group is realizing that these issues may be higher solved with smarter optimization frameworks slightly than simply heuristics. Bilevel optimization is one such framework that deserves a spot within the sensible toolbox. It aligns with a systems-level view of ML mannequin design: typically, to unravel a fancy drawback, it’s essential to break it into components and let every half do what it’s greatest at, underneath a transparent protocol of interplay.

In closing, the following time you end up lamenting “if solely I might get excessive accuracy and fulfill X with out tanking Y,”keep in mind which you can strive giving every need its personal knob. Bilevel coaching would possibly simply provide the elegant compromise you want an “optimizer for every goal,” working collectively in concord. As a substitute of combating a battle of gradients inside one weight area, you orchestrate a dialogue between two units of parameters. And because the FairBiNN outcomes point out, that dialogue can result in outcomes the place everyone wins, or no less than nobody unnecessarily loses.

Completely satisfied optimizing, on each your aims!

When you discover this method invaluable and plan to include it into your analysis or implementation, please take into account citing our unique FairBiNN paper:

@inproceedings{NEURIPS2024_bef7a072,
creator = {Yazdani-Jahromi, Mehdi and Yalabadi, Ali Khodabandeh and Rajabi, AmirArsalan and Tayebi, Aida and Garibay, Ivan and Garibay, Ozlem Ozmen},
booktitle = {Advances in Neural Data Processing Methods},
editor = {A. Globerson and L. Mackey and D. Belgrave and A. Fan and U. Paquet and J. Tomczak and C. Zhang},
pages = {105780--105818},
writer = {Curran Associates, Inc.},
title = {Truthful Bilevel Neural Community (FairBiNN): On Balancing equity and accuracy by way of Stackelberg Equilibrium},
url = {https://proceedings.neurips.cc/paper_files/paper/2024/file/bef7a072148e646fcb62641cc351e599-Paper-Convention.pdf},
quantity = {37},
yr = {2024}
}

References:

• Mehdi Yazdani-Jahromi et al., “Truthful Bilevel Neural Community (FairBiNN): On Balancing Equity and Accuracy by way of Stackelberg Equilibrium,” NeurIPS 2024.arxiv.org
• FairBiNN Open-Supply Implementation (GitHub)github.com: code examples and documentation for the bilevel equity method.
• Moonlight AI Analysis Assessment on FairBiNN — summarizes the methodology and key insights themoonlight.io, together with the alternating optimization process and assumptions (like Lipschitz continuity).