Categorical Cross-Entropy Loss and Accuracy | by Mrscott | May, 2025

To this point, the mannequin spits out a likelihood distribution so our loss perform (will also be known as Price perform) must replicate that, therefore the explicit cross-entropy perform, aka Log Loss. It finds the distinction, or loss, between the precise ‘y’ and predicted distribution, ‘y-hat’ .

The standard type for categorical cross-entropy loss:

the place:

• C = variety of lessons (e.g., 3 if in case you have pink, blue, inexperienced)
• yi​ = 1 if class i is the true class, 0 in any other case (from the one-hot goal vector)
• pi(y-hat)​ = predicted likelihood for sophistication i (after softmax).

If our softmax output is [0.7, 0.1, 0.2], the one-hot encoding for this could be [1, 0, 0]. We now have 0.7 because the true class 1, and the opposite two outputs can be 0 for one-hot encoding. Lets plug some numbers into the system:

• (1*log(0.7) + 0 * log(0.1) + 0 * log (0.2)) = −(−0.3567) = 0.3567

With all of the craziness occurring all over the world proper now it’s good to know some issues haven’t modified appreciated multiplying by 0 nonetheless equals 0, so we will merely the system to:

L=−log(0.7) = 0.3567

The log used is the pure log or base e. The upper a mannequin’s confidence in its prediction the decrease the loss, which is smart for the reason that loss is the distinction between precise vs predicated values. Should you’re 100% assured that any quantity * 0 = 0 your loss can be 0.0. Your confidence about having the following successful lotto ticket is slightly low (accurately) in order that distinction can be a really massive quantity.

#Instance
print(math.log(1.0)) # 100% assured
print(math.log(0.5)) # 50% assured
print(math.log(0.000001)) # Extraordinarily low confidence

0.0
-0.6931471805599453
-13.815510557964274

This curvature ought to most likely be a bit extra excessive with a extra ‘hockey-stick’ look to the curvature however hey I’m making an attempt. The plot above exhibits how the cross-entropy loss 𝐿(𝑝) = −ln(𝑝) behaves because the mannequin’s predicted confidence 𝑝 (for the true class) varies from 0 to 1:

– As 𝑝→1: the loss drops towards 0, that means excessive confidence within the right class yields nearly no penalty.

– As 𝑝→0: the loss shoots towards +∞, closely penalizing predictions that assign near-zero likelihood to the true class. It “amplifies” the penalty on confidently improper predictions, pushing the optimizer to right them aggressively.

– Speedy lower: many of the loss change occurs for 𝑝 within the low vary (0–0.5). Gaining a bit confidence from very low 𝑝 yields a big discount in loss.

**This** curvature is what drives gradient updates.

Recall that that is solely the primary go by the community with randomly initialized weights, so this primary calculation could possibly be off by a large margin. You compute the softmax and get one thing like [0.7,0.1,0.2], then compute the loss and again‑propagate to replace the weights. On the subsequent ahead go, with these up to date weights, you’ll get a new output distribution — possibly [0.2,0.1,0.7] or one thing else solely. Over many such passes (epochs), gradient descent nudges the weights in order that finally the community’s outputs align extra intently with the true one‑sizzling targets. However we’re not stepping into back-propagation simply but.

Since I discussed multiplying by 0, dividing by 0, or in our case log(0) additionally must be talked about. Regardless it’s nonetheless undefined, regardless of what some elementary college instructor and principal mentioned (sure, a instructor claimed dividing by 0 = 0). The mannequin may output 0, so we have to cope with that contingency. log(p) and p = 0, you get -∞. Additionally we don’t need 1 as an output both, so we’ll clip each ends to make the numbers shut however not equal to 0 and 1.

y_pred_clipped = np.clip(y_pred, 1e-7, 1 - 1e-7)