Universal Approximation Theorem states {that a} neural community with a single hidden layer and a nonlinear activation operate can approximate any steady operate.
Sensible points apart, such that the variety of neurons on this hidden layer would develop enormously massive, we don’t want different community architectures. A easy feed-forward neural community may do the trick.
It’s difficult to estimate what number of community architectures have been developed.
If you open the favored AI mannequin platform Hugging Face immediately, you will discover a couple of million pretrained fashions. Relying on the duty, you’ll use completely different architectures, for instance transformers for pure language processing and convolutional networks for picture classification.
So, why can we want so many Neural Network architectures?
On this publish, I need provide a solution to this query from a physics perspective. It’s the construction within the information that evokes novel neural community architectures.
Symmetry and invariance
Physicists love symmetry. The elemental legal guidelines of physics make use of symmetries, comparable to the truth that the movement of a particle might be described by the identical equations, no matter the place it finds itself in time and house.
Symmetry all the time implies invariance with respect to some transformation. These ice crystals are an instance of translational invariance. The smaller buildings look the identical, no matter the place they seem within the bigger context.
Exploiting symmetries: convolutional neural community
In case you already know {that a} sure symmetry persists in your information, you may exploit this truth to simplify your neural community structure.
Let’s clarify this with the instance of picture classification. The panel reveals three scenes together with a goldfish. It will possibly present up in any location inside the picture, however the picture ought to all the time be labeled as goldfish.
A feed-forward neural community may actually obtain this, given adequate coaching information.
This community structure requires a flattened enter picture. Weights are then assigned between every enter layer neuron (representing one pixel within the picture) and every hidden layer neuron. Additionally, weights are assigned between every neuron within the hidden and the output layer.
Together with this structure, the panel reveals a “flattened” model of the three goldfish pictures from above. Do they nonetheless look alike to you?
By flattening the picture, we’ve incurred two issues:
- Photographs that comprise an analogous object don’t look alike as soon as they’re flattened,
- For prime-resolution pictures, we might want to prepare a number of weights connecting the enter layer and the hidden layer.
Convolutional networks, however, work with kernels. Kernel sizes usually vary between 3 and seven pixels, and the kernel parameters are learnable in coaching.
The kernel is utilized like a raster to the picture. A convolutional layer may have a couple of kernel, permitting every kernel to give attention to completely different features of the picture.
For instance, one kernel may decide up on horizontal strains within the picture, whereas one other may decide up on convex curves.
Convolutional neural networks protect the order of pixels and are nice to be taught localized buildings. The convolutional layers might be nested to create deep layers. Together with pooling layers, high-level options might be realized.
The ensuing networks are significantly smaller than in the event you would use a fully-connected neural community. A convolutional layer solely requires kernel_size x kernel_size x n_kernel trainable parameters.
It can save you reminiscence and computational finances, all by exploiting the truth that your object could also be situated wherever inside your picture!
Extra superior deep studying architectures that exploit symmetries are Graph Neural Networks and physics-informed neural networks.
Abstract
Convolutional neural networks work nice with pictures as a result of they protect the native data in your picture. As an alternative of flattening all of the pixels, rendering the picture meaningless, kernels with learnable parameters decide up on native options.
