Neural Networks Explained: Activation Functions, Perceptrons & Backpropagation

Introduction to Neural Networks: Learning Rules, Activation Functions & Backpropagation Variations

Artificial Neural Networks (ANNs) are at the heart of modern AI. This post explores:

1. Learning rules
2. Activation functions
3. Single-layer perceptrons
4. Backpropagation networks
5. Architecture & learning process
6. Variations of standard backpropagation

1. What Are Neural Networks?

Neural networks are computational models inspired by biological brains—collections of interconnected "neurons" that process information 1. Each neuron receives inputs, applies weights plus a bias, then passes the sum through an activation function to produce an output 2.

2. Learning Rules: How Networks Learn

Learning rules dictate how weights and biases are updated during training. Common paradigms:

• Hebbian learning: “Cells that fire together wire together” 3.
• Perceptron learning rule: Adjusts weights by comparing target vs actual outputs, converging if data is linearly separable 4.
• Oja’s rule: A stable variant of Hebbian, preventing weight explosion 5.

3. Activation Functions: Introducing Nonlinearity

Activation functions enable networks to learn complex, non-linear patterns 6. Here are key types:

3.1 Step & Linear

• Binary Step: Maps input to ±1 based on threshold—used in classic perceptrons 7.
• Linear: Identity function; renders multi-layer nets ineffective unless used for final regression layers 8.

3.2 Sigmoid & Tanh

These S-shaped functions are differentiable, enabling gradient-based training. Sigmoid outputs 0–1, ideal for binary classification; Tanh outputs –1 to 1, zero-centered hence often preferred 9. However, both suffer from vanishing gradients 10.

3.3 ReLU and Variants

ReLU = max(0, x); popular due to simplicity and reduced vanishing gradient 11. But it can “die” when neurons output zero continuously. Variants:

• Leaky ReLU: Small slope for x < 0.
• PReLU: Learned slope parameter 12.
• ELU: Smooth negative region—faster learning and better generalization 13.

3.4 Softmax & Advanced Functions

Softmax converts outputs to class probabilities (sum = 1) for multiclass classification 14. Other advanced functions:

• Swish: Smooth, non-monotonic—good in deep nets.
• GELU & SELU: Used in modern models (like BERT), with self-normalizing benefits 15.

4. The Single-Layer Perceptron

Introduced by Rosenblatt (1958), it’s the simplest neural model: inputs → weighted sum + bias → step activation → output class 16. It solves only linearly separable tasks. The classic learning rule is:

w_new = w_old + η (t – o) xi

If data isn’t linearly separable (e.g., XOR), it fails—a motivation for multi-layer networks.

5. Backpropagation Networks & Architecture

Multi-layer perceptrons (MLPs) combine input, hidden, and output layers. Weights are trained via backpropagation (chain rule + gradient descent) 17. Architecture includes:

• Input layer (features)
• One or more hidden layers with non-linear activations
• Output layer suited to task: regression or classification

Training proceeds in epochs: forward pass → compute error → backward pass → weight updates 18.

6. Backpropagation Learning: Step-by-Step

The algorithm:

1. Initialize weights/biases (often small random values).
2. Forward pass: compute outputs layer by layer.
3. Error: difference between predicted & actual.
4. Backward pass: calculate gradients via chain rule.
5. Update weights: w ← w – η * gradient.
6. Repeat: over multiple epochs until convergence 19.

Forward pass sums inputs and applies activation, backward pass propagates errors and updates parameters 20.

7. Variations of Standard Backpropagation

To improve vanilla backpropagation:

• Momentum: Accelerates convergence and prevents oscillation.
• Adaptive methods: RMSprop, Adam—adjust learning rates per parameter.
• Batch vs. Stochastic: Full-batch stable but slow; stochastic and mini-batch add noise for better minima.
• Regularization: Dropout, weight decay, early stopping—to reduce overfitting.
• Levenberg–Marquardt: For small networks—fast but memory-intensive.
• Second-order methods: Utilize Hessian (expensive but precise).

8. Practical Tips & Summary

For beginners:

• Start with MLP + ReLU hidden layers + softmax (classification) or linear (regression).
• Use Adam optimizer and mini-batches (~32–256 samples).
• Monitor validation loss and apply regularization.
• Tune learning rate, architecture depth, batch size.

In summary, neural networks combine:

• Learning rules to adjust weights
• Activation functions that introduce non-linearity
• Backpropagation to train multi-layer architectures
• Advanced optimizers & variations to enhance performance

For deeper insights—especially regarding ethical AI and bias—check out our related post: Ethical AI: Bias, Fairness & Guidelines.

Conclusion

This foundational understanding equips you to build and train neural networks confidently. Stay tuned to SRF Developer for more deep dives into advanced topics like CNNs, RNNs, and Transformer models.