← Back to Modules Directory

Module 12 - Basic Artificial Neural Net (ANN)

Code Example: NumPy Implementation

Here is the full implementation referenced throughout this module — self.weights1, self.bias1, forward, backward, and sigmoid_derivative.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    # x here is already sigmoid(x), i.e. an activation, not a raw input
    return x * (1 - x)

class NeuralNetwork:
    def __init__(self, input_size, hidden_size1, hidden_size2, output_size, learning_rate=0.1):
        self.learning_rate = learning_rate

        # Weights: random initialization breaks symmetry
        self.weights1 = np.random.randn(input_size, hidden_size1)
        self.weights2 = np.random.randn(hidden_size1, hidden_size2)
        self.weights3 = np.random.randn(hidden_size2, output_size)

        # Biases: zero initialization is common
        self.bias1 = np.zeros((1, hidden_size1))
        self.bias2 = np.zeros((1, hidden_size2))
        self.bias3 = np.zeros((1, output_size))

    def forward(self, X):
        self.hidden_layer1 = sigmoid(np.dot(X, self.weights1) + self.bias1)
        self.hidden_layer2 = sigmoid(np.dot(self.hidden_layer1, self.weights2) + self.bias2)
        self.output_layer = sigmoid(np.dot(self.hidden_layer2, self.weights3) + self.bias3)
        return self.output_layer

    def backward(self, X, y, output):
        # Error at the output layer
        output_error = y - output
        d_output = output_error * sigmoid_derivative(output)

        # Error propagated back to the second hidden layer
        hidden_error2 = d_output.dot(self.weights3.T)
        d_hidden2 = hidden_error2 * sigmoid_derivative(self.hidden_layer2)

        # Error propagated back to the first hidden layer
        hidden_error1 = d_hidden2.dot(self.weights2.T)
        d_hidden1 = hidden_error1 * sigmoid_derivative(self.hidden_layer1)

        # Update weights and biases using the gradients and learning rate
        self.weights3 += self.learning_rate * self.hidden_layer2.T.dot(d_output)
        self.bias3 += self.learning_rate * np.sum(d_output, axis=0, keepdims=True)

        self.weights2 += self.learning_rate * self.hidden_layer1.T.dot(d_hidden2)
        self.bias2 += self.learning_rate * np.sum(d_hidden2, axis=0, keepdims=True)

        self.weights1 += self.learning_rate * X.T.dot(d_hidden1)
        self.bias1 += self.learning_rate * np.sum(d_hidden1, axis=0, keepdims=True)

    def train(self, X, y, epochs=10000):
        for epoch in range(epochs):
            output = self.forward(X)
            self.backward(X, y, output)
            if epoch % 1000 == 0:
                loss = np.mean(np.square(y - output))
                print(f"Epoch {epoch}, Loss: {loss:.4f}")


# X: 2 samples, 4 features each (matches the module's example shapes)
X = np.array([[0, 0, 1, 1],
              [1, 1, 0, 0]])

# y: binary class labels for the 2 samples
y = np.array([[0],
              [1]])

net = NeuralNetwork(input_size=4, hidden_size1=5, hidden_size2=4, output_size=1, learning_rate=0.1)
net.train(X, y, epochs=10000)

# Predict on a new, unseen sample
X_new = np.array([[0.3, 0.4, 0.5, 0.6]])
prediction = net.forward(X_new)
print(f"Prediction: {prediction}")

Initializing the Weights

In this code, self.weights1, self.weights2, and self.weights3 represent the weight matrices for the connections between different layers of the neural network. These weights play a crucial role in determining the network's behavior and learning process.

Here's a breakdown of each weight matrix:

self.weights1

self.weights2

self.weights3

Why are weights important?

Initialization:

In the code, the weights are initialized with random values using np.random.randn. This random initialization is crucial for breaking symmetry and allowing the network to learn effectively.

Initialize the Biases

In this code, self.bias1, self.bias2, and self.bias3 represent the bias vectors for each layer of the neural network. These biases, along with the weights, are essential parameters that the network learns during training.

Here's a breakdown of each bias vector:

self.bias1

self.bias2

self.bias3

Why are biases important?

Initialization:

In the code, the biases are initialized with zeros using np.zeros. While zero initialization is common for biases, other initialization strategies can also be used depending on the specific network architecture and activation functions.

In summary:

Biases are important parameters in neural networks that provide flexibility, enhance representation capabilities, and improve learning efficiency. They act as offsets for the activation functions, allowing neurons to activate even with zero or negative input sums.

The Learning Rate

The learning rate, often denoted as α (alpha), is a crucial hyperparameter in the training of neural networks. It controls how much the model's parameters (weights and biases) are adjusted during each iteration of the optimization algorithm (e.g., gradient descent).

Think of it like this:

How it Works

During training, the neural network calculates the gradients of the loss function with respect to the weights and biases. These gradients indicate the direction of the steepest ascent of the loss function. To minimize the loss, we want to move the parameters in the opposite direction (steepest descent).

The learning rate scales these gradients, determining the step size taken in that direction. The update rule for a parameter (e.g., a weight) typically looks like this:

weight = weight - learning_rate * gradient

Choosing the Right Learning Rate

The choice of learning rate is critical for successful training:

Finding an appropriate learning rate often involves experimentation and tuning. Common techniques include:

In the Code

In the provided code, the learning_rate is passed as an argument to the NeuralNetwork class and used in the backward method to scale the gradients before updating the weights and biases. This controls how much the parameters are adjusted in each iteration of the training loop.

Key Takeaways

Forward Propagation

Forward propagation is the process of passing the input data through the neural network to get the output. It's like a chain reaction, where the output of one layer becomes the input to the next layer.

  1. Input (X): The input data is passed as an argument to the forward method. This data represents the features of the input samples.
  2. First Hidden Layer
  3. Second Hidden Layer
  4. Output Layer
  5. Return Output: The forward method returns the self.output_layer, which represents the network's prediction for the given input.

In essence, forward propagation involves:

This process continues until the final output is generated. The output represents the network's prediction based on the input data and the learned parameters (weights and biases).

The Sigmoid Function

The sigmoid function is a mathematical function that has a characteristic "S"-shaped curve. It's often used as an activation function in neural networks due to its properties that make it well-suited for introducing non-linearity and representing probabilities.

Mathematical Definition

The sigmoid function is defined as:

sigmoid(x) = 1 / (1 + exp(-x))

where:

Properties

How it Works in Neural Networks

In neural networks, the sigmoid function is typically used as an activation function. It takes the weighted sum of inputs and biases for a neuron and produces an output between 0 and 1. This output represents the activation level of the neuron.

Advantages

Limitations

Alternatives

Due to its limitations, other activation functions like ReLU (Rectified Linear Unit) and tanh (hyperbolic tangent) are often preferred in modern deep learning architectures.

In Summary

The sigmoid function is a non-linear activation function commonly used in neural networks. It squashes the input to a range between 0 and 1, making it suitable for representing probabilities and introducing non-linearity. However, it has limitations like vanishing gradients and not being zero-centered, which have led to the adoption of alternative activation functions in many cases.

Backward Propagation

Backpropagation is the heart of how neural networks learn. It's the algorithm that calculates how much each weight and bias in the network contributes to the overall error, and then updates those parameters accordingly to improve the network's accuracy.

  1. Calculate the Error
  2. Propagate the Error Backwards
  3. Calculate Gradients for the Output Layer
  4. Calculate Gradients for the Hidden Layers
  5. Update Weights and Biases

Key Ideas in Backpropagation

In essence, backpropagation involves:

This process allows the neural network to learn from the data and improve its accuracy over time.

"Propagating the error backwards" is the core idea behind backpropagation. It refers to the process of moving the error signal from the output layer back through the hidden layers to the input layer. This is done to figure out how much each weight and bias in the network contributed to the overall error.

  1. Start at the Output: The process begins by calculating the error at the output layer. This error represents the difference between the network's prediction and the actual target value.
  2. Chain Reaction: The error is then propagated back through the network, layer by layer, like a chain reaction. At each layer, we calculate how much of the error from the previous layer is attributed to the current layer's activations.
  3. Using the Chain Rule: The chain rule from calculus is used to calculate these error attributions. It essentially tells us how much a small change in a neuron's activation affects the overall error.
  4. Calculating Gradients: By propagating the error backward and applying the chain rule, we calculate the gradients of the error with respect to each weight and bias in the network. These gradients indicate the direction and magnitude of the influence of each parameter on the error.
  5. Updating Parameters: Finally, the gradients are used to update the weights and biases in a way that reduces the error. This is typically done using an optimization algorithm like gradient descent.

Analogy

Imagine a team working on a project where the final outcome has an error. To improve, they need to figure out where things went wrong:

In the Code

In the backward method, the lines like hidden_error2 = d_output.dot(self.weights3.T) and hidden_error1 = d_hidden2.dot(self.weights2.T) are performing this error propagation. They calculate how much of the error from the subsequent layer is attributed to the activations of the current layer.

Key Takeaway

Propagating the error backwards is essential for training neural networks. It allows the network to identify how each parameter contributes to the error and adjust those parameters accordingly to improve its performance. This process is what enables the network to learn from the data and make more accurate predictions.

Sigmoid Derivative

The sigmoid derivative plays a crucial role in the backpropagation algorithm used to train neural networks. It tells us how much the output of the sigmoid function changes with respect to a small change in its input. This information is essential for calculating the gradients of the error with respect to the weights and biases in the network.

Mathematical Definition

The derivative of the sigmoid function is:

sigmoid_derivative(x) = sigmoid(x) * (1 - sigmoid(x))

where sigmoid(x) is the output of the sigmoid function for input x.

Why is it important in backpropagation?

During backpropagation, we need to calculate how much the error at the output layer is affected by the activations of the neurons in the previous layers. This involves calculating the gradients of the error with respect to the activations.

Since the sigmoid function is used as the activation function in this neural network, its derivative tells us how much a small change in the activation of a neuron will affect the output of that neuron. This information is then used to calculate how much that change in activation will affect the overall error.

Chain Rule

The sigmoid derivative is used in conjunction with the chain rule from calculus to calculate the gradients of the error with respect to the weights and biases. The chain rule allows us to break down the calculation of the gradient into smaller, manageable steps.

In the Code

In the backward method, the lines like:

d_output = output_error * sigmoid_derivative(output)

d_hidden2 = hidden_error2 * sigmoid_derivative(self.hidden_layer2)

d_hidden1 = hidden_error1 * sigmoid_derivative(self.hidden_layer1)

are using the sigmoid_derivative function to calculate the gradients of the error with respect to the activations of each layer.

Key Takeaways

The sigmoid derivative indirectly relates to the gradient descent update forumula, which represents the gradient descent update rule. Here's how:

The Role of the Sigmoid Derivative

Connecting to the gradient descent update forumula

In Summary

While the sigmoid derivative is not explicitly present in the gradient descent update rule, it's an essential part of the process that calculates the gradient used in that update rule. The sigmoid derivative is used in the backpropagation algorithm to determine how much each weight and bias contributes to the overall error, and this information is then used to update the parameters and improve the accuracy of the neural network.

Key Takeaway

The sigmoid derivative plays a crucial behind-the-scenes role in enabling the gradient descent update rule to work effectively in neural networks that use the sigmoid activation function.

θ_j := θ_j - α * (∂/∂θ) J(θ₀, θ₁)

To explicitly show the sigmoid derivative in the gradient descent update rule, we need to expand the gradient term (∂/∂θ) J(θ₀, θ₁) using the chain rule.

Let's assume θ_j represents a weight connecting a neuron in layer l-1 to a neuron in layer l. The update rule with the sigmoid derivative would look like this:

θ_j := θ_j - α * Σ_i [ (y_i - ŷ_i) * σ'(z_i^l) * a_j^{l-1} ]

where:

Explanation

Simplified for a Single Example

If we consider a single training example and omit the summation, the update rule becomes:

θ_j := θ_j - α * (y - ŷ) * σ'(z^l) * a_j^{l-1}

Key Takeaway

This expanded form of the gradient descent update rule explicitly shows the role of the sigmoid derivative in calculating the gradient. It highlights how the error is propagated backward through the network, considering the influence of the activation function at each layer. This detailed representation provides a clearer understanding of how the sigmoid derivative contributes to the learning process in neural networks.

Build the Neural Net

X (Input Data)

y (Target Output)

Goal: Training the Neural Network

The purpose of X and y is to train the neural network to learn a mapping between the input features and the target output. By feeding the network with X and y during training, you're essentially teaching it to recognize patterns and relationships in the data so that it can make accurate predictions on new, unseen data.

Prediction

After training, the neural network can be used to predict the output for new input data. Given a new set of features (X_new), the network will process the input through its layers and produce a prediction (ŷ) that represents the probability of belonging to class 1.

Example Scenario

Let's say you're building a neural network to predict whether a customer will click on an ad (1 for click, 0 for no click).

By training the network on X and y, you aim to create a model that can predict the likelihood of a new customer clicking on an ad based on their features.

In summary:

X and y are the training data used to teach the neural network to make predictions. X provides the input features, and y provides the corresponding target outputs. The goal is to learn a mapping between the features and the labels so that the network can accurately predict the output for new, unseen data.

Train the Neural Net

Make Predictions

Given the provided X and y arrays, the goal is to train a neural network that can predict the probability of an input belonging to class 0 or class 1.

Training Process

During training, the neural network will learn to map the input features (X) to the corresponding target labels (y). Since this is a binary classification problem, the network will output a probability between 0 and 1 for each input sample.

Prediction

After training, if you feed the network a new input sample (with four features), it will predict the probability of that sample belonging to class 1.

Example

Let's say after training, you provide the network with a new input:

X_new = np.array([[0.3, 0.4, 0.5, 0.6]])

The network might output a probability like 0.8. This indicates that the network predicts an 80% chance that this new input belongs to class 1.

Key Takeaway

The goal is to train a neural network that can effectively learn the relationship between the input features and the binary output labels, enabling it to make accurate predictions on new, unseen data.

Summary

  1. Initialization (__init__)
  2. Forward Propagation (forward)
  3. Backpropagation (backward)
  4. Training
  5. Prediction

This code demonstrates a basic implementation of a neural network with two hidden layers using only NumPy. It showcases the fundamental concepts of forward propagation, backpropagation, and gradient descent for training a neural network. You can modify this code to experiment with different network architectures, activation functions, and datasets.

Runtime Processing

Here's a breakdown of the differences between CPUs, T4 GPUs, and TPU v2-8s:

CPU (Central Processing Unit)

T4 GPU (Graphics Processing Unit)

TPU v2-8 (Tensor Processing Unit)

Key Differences

Feature

CPU

T4 GPU

TPU v2-8

Primary Purpose

General-purpose processing

Graphics and parallel processing

Machine learning and AI

Architecture

Sequential processing, few cores

Parallel processing, many cores

Matrix multiplication focus, high throughput

Strengths

Versatile, handles various tasks

High performance for parallel workloads

Extremely fast for deep learning

Weaknesses

Limited parallel processing power

Can be power-hungry

Specialized for TensorFlow

When to Use Each

In Summary

CPUs, T4 GPUs, and TPU v2-8s are all processors designed for different purposes and with varying strengths. CPUs are general-purpose workhorses, GPUs are parallel powerhouses, and TPUs are specialized for AI. Choosing the right processor depends on the specific workload and requirements.

In PyTorch, a torch.Tensor is a multi-dimensional array that is the fundamental building block for all operations and models. You can think of it as the PyTorch equivalent of a NumPy array, but with some key advantages for deep learning.

Here's a breakdown of what makes torch.Tensor special:

1. GPU Support

2. Automatic Differentiation

3. Optimized Operations

4. Neural Network Building Blocks

5. Dynamic Computation Graph

Creating Tensors

You can create PyTorch tensors in various ways:

Key Takeaways

Code Reference: PyTorch Implementation

The same architecture as the NumPy version above, expressed with torch.nn, criterion, and optimizer exactly as described.

import torch
import torch.nn as nn
import torch.optim as optim

class TorchNet(nn.Module):
    def __init__(self, input_size, hidden_size1, hidden_size2, output_size):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size1)
        self.fc2 = nn.Linear(hidden_size1, hidden_size2)
        self.fc3 = nn.Linear(hidden_size2, output_size)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.sigmoid(self.fc1(x))
        x = self.sigmoid(self.fc2(x))
        x = self.sigmoid(self.fc3(x))
        return x

net = TorchNet(input_size=4, hidden_size1=5, hidden_size2=4, output_size=1)

criterion = nn.MSELoss()                          # Mean Squared Error loss
optimizer = optim.SGD(net.parameters(), lr=0.1)   # Stochastic Gradient Descent

X = torch.tensor([[0., 0., 1., 1.],
                   [1., 1., 0., 0.]])
y = torch.tensor([[0.], [1.]])

for epoch in range(10000):
    optimizer.zero_grad()           # clear gradients from the previous step
    output = net(X)                 # 1. Forward Pass
    loss = criterion(output, y)     # 2. Loss Calculation
    loss.backward()                 # 3. Backward Pass (autograd computes gradients)
    optimizer.step()                # 4. Parameter Update

    if epoch % 1000 == 0:
        print(f"Epoch {epoch}, Loss: {loss.item():.4f}")

# Predict on a new sample
X_new = torch.tensor([[0.3, 0.4, 0.5, 0.6]])
prediction = net(X_new)
print(f"Prediction: {prediction.item():.4f}")

criterion = nn.MSELoss() # Mean Squared Error loss

optimizer = optim.SGD(net.parameters(), lr=0.1) # Stochastic Gradient Descent

These two lines of code are essential for training your neural network in PyTorch. They define the loss function and the optimizer that will be used to update the network's parameters during training.

criterion = nn.MSELoss()

optimizer = optim.SGD(net.parameters(), lr=0.1)

In Summary

These two components work together during the training loop:

  1. Forward Pass: The input data is passed through the network to generate predictions.
  2. Loss Calculation: The criterion (loss function) is used to calculate the error between the predictions and the true labels.
  3. Backward Pass: The optimizer calculates the gradients of the loss with respect to the parameters.
  4. Parameter Update: The optimizer updates the network's parameters based on the calculated gradients and the learning rate.

This iterative process continues for a specified number of epochs, gradually improving the network's accuracy by minimizing the loss function.