What are Neural Networks?

By Jaeden Schafer · 9 min read

What are Neural Networks?

A neural network is a machine learning model inspired by how biological brains work (loosely). It’s a system of interconnected artificial neurons that work together to process information and learn patterns.

The “neural” part is somewhat poetic. These aren’t actually neurons—they’re mathematical functions. But the overall structure is inspired by brains: layers of simple processing units connected together, each connection having a weight that strengthens or weakens the signal.

The power of neural networks comes from depth. A single layer of neurons is limited. But stack many layers, where output of one layer feeds into the next, and you can model incredibly complex patterns. This is why deep learning (neural networks with many layers) has been revolutionary.

Here’s the practical reality: neural networks are a powerful but often overkill solution. For most problems, simpler machine learning models work as well and are easier to interpret. But for complex problems with massive amounts of data—image recognition, language understanding, game playing—neural networks often excel.

The Biological Inspiration (and Limitations)

It’s useful to understand the metaphor, but also where it breaks down:

The Inspiration: Biological brains have neurons that fire or don’t fire based on input from other neurons. Connected synapses get stronger with repeated activation (Hebbian learning: “neurons that fire together, wire together”). Information processing emerges from billions of simple units working in parallel.

The Reality of Artificial Neural Networks: Artificial neurons are nothing like biological ones. They perform a weighted sum of inputs plus a bias term, pass through an activation function, done. No temporal dynamics, no biochemistry, no consciousness. The math is elegant but reductive.

Biological Learning vs. Backpropagation: Brains don’t use backpropagation (a mathematical algorithm for adjusting weights). We don’t understand exactly how brains learn. The metaphor is helpful for intuition but misleading if you take it too literally.

Scale: A human brain has about 86 billion neurons. Large language models like GPT-4 have roughly 170 billion parameters (roughly analogous to connection weights). We’re approaching brain-scale in terms of parameters, but organization and dynamics are completely different.

The metaphor helps beginners understand the basic idea (lots of simple units connected together can solve hard problems). Just don’t confuse the metaphor with the reality.

Neural Network Architecture

Basic Structure

A simple feedforward neural network has three types of layers:

Input Layer — One neuron per input feature. For image recognition, one neuron per pixel. For text processing, one per word embedding dimension. These don’t compute anything—they just hold input values.

Hidden Layers — Where the actual computation happens. Each neuron in a hidden layer receives input from all neurons in the previous layer, computes a weighted sum, applies a nonlinear activation function, and outputs the result. Why nonlinear? Because linear transformations of linear transformations are just linear, and you can’t learn complex patterns with only linear operations.

Output Layer — Produces the final prediction. For classification, usually one neuron per class with softmax activation (outputs sum to 1, giving probabilities). For regression, a single neuron outputting a continuous value.

Weights and Biases

Each connection between neurons has a weight. During training, these weights are adjusted. The more important a connection for making correct predictions, the larger its weight (roughly speaking). Each neuron also has a bias term—a constant added before the activation function. Biases help shift the activation function’s decision boundary.

A network with 784 input neurons (for 28×28 images), two hidden layers with 128 neurons each, and 10 output neurons has roughly 784×128 + 128×128 + 128×10 = 118,272 parameters to learn. A large language model has billions.

Activation Functions

Nonlinear functions that add expressiveness to the network. Common ones:

• ReLU (Rectified Linear Unit): Returns max(0, x). Simple, fast, works surprisingly well.
• Sigmoid: Maps input to (0, 1). Used historically, less common now.
• Tanh: Maps to (-1, 1). Similar to sigmoid, slightly different range.
• Softmax: For output layers in classification, converts values to probabilities.

How Neural Networks Learn

The Training Loop

1. Forward pass: Feed training data through the network, get predictions.
2. Compute loss: Compare predictions to actual values. How wrong were we?
3. Backward pass: Compute gradients of the loss with respect to each weight (using backpropagation).
4. Update weights: Adjust weights in the direction that reduces loss (using gradient descent or a variant).
5. Repeat thousands or millions of times.

Backpropagation

The algorithm that makes learning practical. It efficiently computes how much each weight contributed to the error, starting from the output layer and working backward (hence “backpropagation”). Without it, training would be exponentially slower.

The math: chains of partial derivatives applying the chain rule. For a network with millions of parameters, this is intensive computation but computers handle it well.

Gradient Descent and Variants

After computing gradients, you update weights: new_weight = old_weight – learning_rate × gradient. The learning rate controls step size. Too high, and you overshoot. Too low, and training is slow.

Simple gradient descent processes all training data per update (slow, stable). Stochastic gradient descent processes one example at a time (fast, noisy). Mini-batch gradient descent is a compromise. Modern variants like Adam adjust the learning rate per parameter, converging faster.

Overfitting and Regularization

Neural networks have tremendous capacity. They can memorize training data, learning every quirk and noise. This looks great on the training set but fails on new data.

Solutions:

• Dropout: Randomly disable neurons during training. Forces redundancy, prevents co-adaptation.
• L1/L2 Regularization: Penalize large weights, prefer simpler models.
• Early stopping: Stop training when performance on validation data starts decreasing.
• Data augmentation: Artificially increase training data with transformations (crop, rotate, flip images).

Types of Neural Networks

Feedforward Neural Networks (Fully Connected)

The simplest type. Each neuron connects to all neurons in the next layer. Works for tabular data and simple problems. Scales poorly to image or sequence data due to number of parameters.

Convolutional Neural Networks (CNNs)

Designed for images. Uses convolutional filters that scan across the image, detecting local patterns (edges, textures, shapes). Hierarchical: early layers detect simple patterns, deeper layers combine them into complex ones. Dramatically more efficient than fully connected networks for images.

Examples: Image classification, object detection, face recognition. These power most computer vision applications.

Recurrent Neural Networks (RNNs)

For sequences (text, time series). Have internal state/memory that persists across sequence steps. Each step takes current input plus hidden state from previous step, produces output and new hidden state.

Problem: vanishing gradient. In long sequences, gradients from early steps disappear, so networks struggle to learn long-range dependencies.

Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU)

Improvements over basic RNNs with gating mechanisms that control information flow. Can maintain long-range dependencies better than vanilla RNNs. Still used but being replaced by transformers.

Transformers

The current state-of-the-art for sequence problems, especially language. Key innovation: attention mechanism. Instead of sequential processing, transformers process entire sequences in parallel, letting each token attend to all other tokens. Dramatically more efficient, parallelizable, and effective at capturing long-range dependencies.

All modern large language models (GPT, Claude, Llama) use transformer architecture. This is what powers the chatbots you interact with.

Modern Applications and Models

Large Language Models (LLMs): Transformers trained on massive text datasets. GPT-4 (OpenAI), Claude (Anthropic), Llama (Meta). These can generate text, answer questions, summarize, translate, code, and more. Used for chatbots, content generation, coding assistance, search.

Vision Models: CNNs or vision transformers trained on millions of images. Used for image classification (is this a cat?), object detection (find all cats), segmentation (label each pixel as cat or background), and more.

Multimodal Models: Process multiple modalities (text, image, audio). GPT-4 with vision can analyze images. Models like DALL-E generate images from text. Increasingly important as problems involve mixed data types.

Recommendation Systems: Neural networks that learn user preferences from interaction history and predict what users will like. Powers Netflix, Spotify, YouTube, Amazon recommendations.

Reinforcement Learning: Neural networks trained through interaction and rewards. AlphaGo beat Go champion using deep reinforcement learning. Used for game AI, robotics, optimization problems.

Training Challenges and Solutions

Vanishing/Exploding Gradients: In deep networks, gradients can become very small (vanish) or very large (explode) as they backpropagate. Solution: careful initialization, batch normalization, residual connections, activation functions like ReLU.

Computational Cost: Training large networks on large datasets is expensive. GPT-4 training reportedly cost millions of dollars in compute. Solutions: use smaller models, transfer learning (fine-tune pre-trained models instead of training from scratch), distributed training across GPUs/TPUs.

Data Requirements: Deep learning typically needs lots of data. With transfer learning and few-shot learning, you can do more with less, but there’s still a floor. Solutions: data augmentation, synthetic data, domain adaptation techniques.

Interpretability: Why did the network make a particular prediction? Hard to say. Solutions: attention visualization, saliency maps, probing classifiers, but no perfect solution. This is ongoing research.

Robustness: Tiny perturbations in input can cause completely different predictions. A pixel-level change can flip an image classifier. Solutions: adversarial training, certified robustness, robustness testing, but still an open problem.

Frequently Asked Questions