Convolutional Architectures: Physics & Math

Convolutional Neural Networks (CNNs) leverage principles like local receptive fields, parameter sharing, and pooling to efficiently process spatial data, overcoming limitations of traditional MLPs. Understanding translation equivariance and invariance is key to designing effective CNN architectures.

Core Principles

• Local Receptive Fields: Neurons connect to small, regional neighborhoods, exploiting spatial locality.
• Parameter Sharing: A single filter (feature detector) uses the same weights across the entire input, reducing parameters.
• Pooling/Subsampling: Controls overfitting and achieves spatial invariance by summarizing localized regions.
• Translation Equivariance: Input shifts result in corresponding output shifts (e.g., convolution with stride 1).
• Translation Invariance: Input shifts are absorbed, leading to position-independent representations (e.g., pooling).
• Curse of Dimensionality: Addressed by parameter sharing, drastically reducing the number of weights needed.
• Spatial Topology: Preserving or transforming spatial relationships is crucial for different tasks.

Action Steps

• Define input dimensions and layer parameters (kernel size, stride, padding).
• Calculate output tensor shape after convolutional and pooling layers.
• Compute the total number of trainable parameters, including weights and biases.
• Choose between equivariant or invariant layers based on the task's need for spatial coordinate information.
• Align network physics (layer operations) with task geometry (e.g., landmark detection vs. classification).

Formulas

• $y[i, j] = \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} x[i+m, j+n] \cdot w[m,n]$
• $f(T_t x) = T_t f(x)$ (Equivariance)
• $f(T_t x) \approx f(x)$ (Invariance)
• $W_{out} = \frac{W - K + 2P}{S} + 1$

Key Terms

• Kernel Filter: A small matrix of weights used in convolution to detect features.
• Stride: The step size the kernel moves across the input.
• Padding: Adding zeros around the input borders to control output size and handle edges.
• Receptive Field: The specific region of the input that a neuron's output is influenced by.
• Translation Equivariance: The property where a shift in input causes an identical shift in the output.
• Translation Invariance: The property where a shift in input does not change the output.
• Pooling: A downsampling operation (e.g., Max Pooling) that reduces spatial dimensions and introduces invariance.
• Parameter Sharing: Using the same set of weights (filter) across different spatial locations in the input.

Real World Examples

• Image Classification: Uses translation invariance (via pooling) to recognize objects regardless of their position.
• Object Detection / Segmentation: Requires a balance of equivariance (to locate features) and invariance (to handle variations).
• Facial Landmark Detection: Needs translation covariance/equivariance to preserve exact coordinate information for precise feature localization.

Timeline

• 1968: Hubel & Wiesel's research on animal visual cortices, introducing the concept of receptive fields.
• Unknown: Development of Convolutional Neural Networks (CNNs) inspired by biological vision.
• Unknown: Introduction of pooling layers to achieve translation invariance.
• Unknown: Mallat's Theory of Scattering Networks formalizing the role of pooling in translation invariance.
• Ongoing: Refinement of CNN architectures and understanding the trade-offs between equivariance and invariance for specific tasks.

People

• Hubel & Wiesel: Pioneers in neuroscience, their work on visual cortex inspired receptive field concepts in CNNs.
• Mallat: Developed the Theory of Scattering Networks, explaining how pooling controls translation invariance.