Convolutional Architectures: Physics and Math

Convolutional Neural Networks (CNNs) leverage biological inspiration and mathematical principles like local connectivity, parameter sharing, and pooling to efficiently process spatial data, overcoming the limitations of traditional Multi-Layer Perceptrons (MLPs). Understanding the interplay between translation equivariance and invariance is key to designing effective CNN architectures.

Core Principles

• Local Receptive Fields: Neurons connect to small, regional neighborhoods, exploiting spatial locality.
• Parameter Sharing: A single filter (feature detector) uses the same weights across the entire input, curing the curse of dimensionality.
• Pooling/Subsampling: Controls overfitting and achieves spatial invariance by summarizing localized regions.
• Translation Equivariance: Shifts in input result in corresponding shifts in output (f(Ttx) = Ttf(x)). Useful for feature detection.
• Translation Invariance: Shifts in input are absorbed, leading to the same output (f(Ttx) ≈ f(x)). Useful for classification.
• The Invariance Trap: Overuse of pooling can destroy essential spatial coordinate information needed for tasks like landmark detection.

Action Steps

• Identify spatial locality in the input data.
• Design filters (kernels) with appropriate sizes for feature detection.
• Apply parameter sharing to reduce model complexity and prevent overfitting.
• Use pooling layers judiciously to achieve spatial invariance where needed.
• Balance equivariant and invariant layers based on the specific task requirements.
• Calculate output tensor shapes and parameter counts for CNN layers.

Formulas

• $y[i, j] = \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} x[i+m, j+n] \cdot w[m, n]$
• $f(T_t x) = T_t f(x)$ (Translation Equivariance)
• $f(T_t x) \approx f(x)$ (Translation Invariance)
• $W_{out} = \frac{W - K + 2P}{S} + 1$ (Output Width Formula)
• $f_{x,y}(S) = \max_{a,b=0}^{1} S_{2X+a, 2Y+b}$ (Max Pooling Operation)
• $\lim_{n\to\infty} ||\Phi_n(T_t f) - \Phi_n(f)|| = 0$ (Mallat's Theory of Scattering Networks)

Key Terms

• Convolutional Kernel: A small filter used to detect features by sliding across the input.
• Spatial Locality: The principle that nearby pixels in an image are more related than distant ones.
• Parameter Sharing: Using the same weights for a filter across the entire input to reduce parameters.
• Translation Equivariance: Output shifts proportionally to input shifts.
• Translation Invariance: Output remains unchanged despite input shifts.
• Pooling: A downsampling operation (e.g., Max Pooling) that reduces spatial dimensions and introduces invariance.
• Curse of Dimensionality: The problem of having too many parameters in a model, leading to overfitting and computational issues.

Pro Tips

• The backward pass of a convolutional layer is also a convolution, using a rotated kernel.
• PyTorch's `Conv2d.backward()` handles fractional striding and transposed convolution math automatically.
• Align the network's physics (equivariance/invariance properties) with the task's geometry.

Pitfalls to Avoid

• Flattening spatial data for MLPs leads to loss of neighborhood structure and parameter explosion.
• Over-reliance on pooling layers can destroy precise coordinate information critical for localization tasks.
• Assuming invariance is always desirable without considering the task's need for precise spatial information.

Myth vs Reality

• MLPs are suitable for processing raw image data.: MLPs fail on spatial grids due to destruction of topology and massive parameter explosion; CNNs are designed to handle this.
• Translation invariance is always the goal for classification.: While useful for classification, excessive invariance (from deep pooling) is detrimental for tasks requiring precise spatial localization.

Real World Examples

• Image Classification (e.g., identifying a cat): Translation invariance is beneficial; the network should recognize the cat regardless of its position in the image.
• Facial Landmark Detection (e.g., pupil location): Translation equivariance is crucial; the exact coordinates of landmarks must be preserved, making excessive pooling detrimental.

Timeline

• 1968: Hubel & Wiesel publish work on the visual cortex, inspiring local receptive fields.
• N/A: Development of Convolutional Neural Networks (CNNs) incorporating these principles.
• N/A: Mallat's Theory of Scattering Networks formalizes the role of pooling in translation invariance.

People

• Hubel & Wiesel: Neuroscientists whose work on visual cortices inspired CNN architectures.
• Mallat: Developed the Theory of Scattering Networks, explaining the mathematical properties of pooling.