NeuralNetworksFromScratchwithRegularization.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "b6078e50-8791-4818-891a-edaa3045a44e",
      "metadata": {
        "scrolled": true,
        "id": "b6078e50-8791-4818-891a-edaa3045a44e"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from tensorflow.keras.datasets import mnist"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "5bee2ac1-7383-4cc9-9fcd-007c67373c5f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5bee2ac1-7383-4cc9-9fcd-007c67373c5f",
        "outputId": "6accc4ed-5227-403b-aeb9-e3d0d9a5676f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "\u001b[1m11490434/11490434\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 0us/step\n"
          ]
        }
      ],
      "source": [
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "1c114fbe-a575-42a7-97ab-9f83a10e1f76",
      "metadata": {
        "id": "1c114fbe-a575-42a7-97ab-9f83a10e1f76"
      },
      "outputs": [],
      "source": [
        "assert x_train.shape == (60000, 28, 28)\n",
        "assert x_test.shape == (10000, 28, 28)\n",
        "assert y_train.shape == (60000,)\n",
        "assert y_test.shape == (10000,)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9355a3bd-99c2-41cf-b75d-75d3628b0a52",
      "metadata": {
        "id": "9355a3bd-99c2-41cf-b75d-75d3628b0a52"
      },
      "source": [
        "## Load the MNIST Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "712741fc-359a-47b9-878d-84292c52c469",
      "metadata": {
        "id": "712741fc-359a-47b9-878d-84292c52c469"
      },
      "source": [
        "`Flattening`: The reshape(x_train.shape[0], -1) method transforms the training data (x_train) from its original multi-dimensional shape (e.g., for images, this could be width x height x channels) into a two-dimensional array. Each sample is converted into a single row, where the first dimension is the number of samples, and the second dimension contains all the pixel values in a flattened format. This is essential for feeding the data into models that expect a flat input, such as fully connected layers in neural networks.\n",
        "\n",
        "`Normalization`: The division by 255.0 scales the pixel values to a range between 0 and 1, as pixel values in images typically range from 0 to 255. Normalization helps improve model training by ensuring that the input features have similar scales, which enhances convergence speed and overall model performance.\n",
        "\n",
        "`Combined Effect`: By flattening and normalizing the data simultaneously, the code prepares the input data for effective processing by machine learning models, ensuring that they can learn efficiently from the data while reducing the risk of issues caused by varying input scales."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "af3d578c-66f5-4be6-a3b7-b5d024a882e5",
      "metadata": {
        "id": "af3d578c-66f5-4be6-a3b7-b5d024a882e5"
      },
      "outputs": [],
      "source": [
        "x_train = x_train.reshape(x_train.shape[0], -1) / 255.0\n",
        "x_test = x_test.reshape(x_test.shape[0], -1) / 255.0"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6e6c2485-b7d0-451f-94a8-09c421990157",
      "metadata": {
        "id": "6e6c2485-b7d0-451f-94a8-09c421990157"
      },
      "source": [
        "## Time to one-hot encode the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "b4a769a6-84ae-4311-8022-2f4086b2444f",
      "metadata": {
        "id": "b4a769a6-84ae-4311-8022-2f4086b2444f"
      },
      "outputs": [],
      "source": [
        "def one_hot_encode(y, num_classes):\n",
        "    return np.eye(num_classes)[y]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "90a3d17d-bfa2-440c-b802-071b8c9eca74",
      "metadata": {
        "id": "90a3d17d-bfa2-440c-b802-071b8c9eca74"
      },
      "source": [
        "One-hot encoding is applied only to the target variable (y_train and y_test) to convert categorical class labels into a binary format suitable for classification tasks. This transformation allows the model to output probabilities for each class and avoids implying any ordinal relationships. The input features (x_train and x_test) are typically left in their original format to preserve their numerical or categorical significance for effective model training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "52680bb7-54f1-4437-9014-ad69814c0eee",
      "metadata": {
        "id": "52680bb7-54f1-4437-9014-ad69814c0eee"
      },
      "outputs": [],
      "source": [
        "y_train_encoded = one_hot_encode(y_train, 10)\n",
        "y_test_encoded = one_hot_encode(y_test, 10)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1c2a899d-f109-4809-83aa-b6a9cef5b1f9",
      "metadata": {
        "id": "1c2a899d-f109-4809-83aa-b6a9cef5b1f9"
      },
      "source": [
        "# Let's define a Neural Network Now:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "89de8137-141b-415a-80eb-09d74aa8fffe",
      "metadata": {
        "id": "89de8137-141b-415a-80eb-09d74aa8fffe"
      },
      "outputs": [],
      "source": [
        "import numpy as np  # Import NumPy for numerical operations\n",
        "\n",
        "class NeuralNetwork:\n",
        "    def __init__(self, input_size, hidden_size, output_size, learning_rate=0.01):\n",
        "        # Initialize weights and biases for input to hidden layer\n",
        "        self.W1 = np.random.randn(input_size, hidden_size) * 0.01  # Small random weights\n",
        "        self.b1 = np.zeros((1, hidden_size))  # Biases initialized to zero\n",
        "\n",
        "        # Initialize weights and biases for hidden to output layer\n",
        "        self.W2 = np.random.randn(hidden_size, output_size) * 0.01  # Small random weights\n",
        "        self.b2 = np.zeros((1, output_size))  # Biases initialized to zero\n",
        "\n",
        "        self.learning_rate = learning_rate  # Set learning rate\n",
        "\n",
        "    def softmax(self, x):\n",
        "        # Compute the softmax of the input for multi-class classification\n",
        "        exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))  # For numerical stability\n",
        "        return exp_x / np.sum(exp_x, axis=1, keepdims=True)  # Normalize to get probabilities\n",
        "\n",
        "    def forward(self, X):\n",
        "        # Perform forward propagation\n",
        "        self.z1 = np.dot(X, self.W1) + self.b1  # Linear combination for hidden layer\n",
        "        self.a1 = np.tanh(self.z1)  # Activation function for hidden layer (tanh)\n",
        "        self.z2 = np.dot(self.a1, self.W2) + self.b2  # Linear combination for output layer\n",
        "        self.a2 = self.softmax(self.z2)  # Softmax activation for output layer\n",
        "        return self.a2  # Return the output probabilities\n",
        "\n",
        "    def backward(self, X, y, output):\n",
        "        # Perform backward propagation to update weights and biases\n",
        "        m = X.shape[0]  # Number of samples\n",
        "\n",
        "        # Calculate the error at the output layer\n",
        "        output_error = output - y  # Derivative of loss w.r.t. output\n",
        "\n",
        "        # Calculate the error at the hidden layer\n",
        "        hidden_error = np.dot(output_error, self.W2.T) * (1 - np.tanh(self.z1) ** 2)  # Derivative of tanh\n",
        "\n",
        "        # Compute gradients\n",
        "        dW2 = np.dot(self.a1.T, output_error) / m  # Gradient for W2\n",
        "        db2 = np.sum(output_error, axis=0, keepdims=True) / m  # Gradient for b2\n",
        "        dW1 = np.dot(X.T, hidden_error) / m  # Gradient for W1\n",
        "        db1 = np.sum(hidden_error, axis=0, keepdims=True) / m  # Gradient for b1\n",
        "\n",
        "        # Update weights and biases using gradient descent\n",
        "        self.W1 -= self.learning_rate * dW1  # Update W1\n",
        "        self.b1 -= self.learning_rate * db1  # Update b1\n",
        "        self.W2 -= self.learning_rate * dW2  # Update W2\n",
        "        self.b2 -= self.learning_rate * db2  # Update b2\n",
        "\n",
        "    def train(self, X, y, epochs):\n",
        "        # Train the neural network for a specified number of epochs\n",
        "        for epoch in range(epochs):\n",
        "            output = self.forward(X)  # Forward pass\n",
        "            self.backward(X, y, output)  # Backward pass\n",
        "\n",
        "            # Print loss every 100 epochs\n",
        "            if epoch % 100 == 0:\n",
        "                loss = -np.mean(np.sum(y * np.log(output + 1e-10), axis=1))  # Cross-entropy loss\n",
        "                print(f\"Epoch {epoch}, Loss: {loss:.4f}\")\n",
        "\n",
        "    def predict(self, X):\n",
        "        # Make predictions for input data\n",
        "        output = self.forward(X)  # Forward pass to get output\n",
        "        return np.argmax(output, axis=1)  # Return the class with the highest probability\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "d55b1aa5-d8ca-41d5-9714-858a37683f3c",
      "metadata": {
        "id": "d55b1aa5-d8ca-41d5-9714-858a37683f3c"
      },
      "outputs": [],
      "source": [
        "# Hyperparameters\n",
        "input_size = 784  # 28x28 pixels\n",
        "hidden_size = 128  # Number of neurons in the hidden layer\n",
        "output_size = 10   # 10 classes for digits 0-9\n",
        "learning_rate = 0.01\n",
        "epochs = 100"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "e149f221-b151-47ec-ae51-42af6012e6da",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e149f221-b151-47ec-ae51-42af6012e6da",
        "outputId": "b10d3d09-823b-48ea-fe92-99bfdbc01684"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 0, Loss: 2.3016\n"
          ]
        }
      ],
      "source": [
        "nn = NeuralNetwork(input_size, hidden_size, output_size, learning_rate)\n",
        "nn.train(x_train, y_train_encoded, epochs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "e27d6278-aa88-4c27-84ad-cae423c1e5b2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e27d6278-aa88-4c27-84ad-cae423c1e5b2",
        "outputId": "90c39543-960a-4545-b641-f46987b14a7c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Accuracy: 0.6103\n"
          ]
        }
      ],
      "source": [
        "# Evaluate the model\n",
        "predictions = nn.predict(x_test)\n",
        "accuracy = np.mean(predictions == y_test)\n",
        "print(f\"Test Accuracy: {accuracy:.4f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "dc559ff4-f301-4e68-aeb9-743492193342",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 494
        },
        "id": "dc559ff4-f301-4e68-aeb9-743492193342",
        "outputId": "1a409b2d-ff07-4b9e-c621-dc81578aefee"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x500 with 10 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA94AAAHdCAYAAADrZqGsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABL/klEQVR4nO3dZ3hVZfb38d+BhBIQxBCalFAVRUQpitLUgEqTpiLogA6IIgi2KEUpBpkRhj8OCsjIiBQZBEUZRwFhKIowMyBFREaNEEApoQ4JICHZzwsv8hiz72N2cu6cku/nuvKCdZ+19zohKycr+2TfPsdxHAEAAAAAACuKBbsAAAAAAAAiGYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeOeBz+fL08fatWuDXWoua9eu9VvzhAkTgl0iwlA498SxY8c0adIktWnTRnFxcbr00kt14403atGiRcEuDWEsnHtCkhYtWqT7779f9evXl8/nU7t27YJdEsJcuPeEJC1btkzXX3+9SpUqpZo1a2rMmDG6cOFCsMtCmIqEnrgoOTlZpUqVks/n0+bNm4NdTtiICnYB4WDevHk5/j137lx98sknueINGzYszLLypGHDhrnqlH5+TitXrlSHDh2CUBXCXTj3xMaNGzVq1Ch17NhRo0ePVlRUlN5991317t1bu3bt0rhx44JdIsJQOPeEJM2YMUNbtmxR8+bNdezYsWCXgwgQ7j3x8ccfq1u3bmrXrp2mTZumL7/8UklJSTpy5IhmzJgR7PIQhsK9J37piSeeUFRUlH766adglxJWfI7jOMEuItwMGTJEr732mn7rU3fmzBnFxMQUUlXeXLyq8c033wS7FESAcOqJPXv2qFixYqpVq1Z2zHEcJSQkaMOGDTp27JjKlCkTxAoRCcKpJyRp//79uvzyy1WsWDE1atRIFStWDIurLggf4dYTV199taKjo7V582ZFRf18nWr06NF66aWXtGvXLl155ZVBrhDhLtx64qIVK1aoa9euSkxMVFJSkv7zn/+oWbNmwS4rLPBW8wBp166dGjVqpC1btqhNmzaKiYnRyJEjJf381pKxY8fmyomPj1f//v1zxE6ePKnhw4erRo0aKlmypOrVq6c//vGPysrKyvG4gwcPavfu3crIyPBc67///W9999136tu3r+dcIK9CtSdq166dY+i+WE+3bt30008/6fvvv/f+ZIE8CNWekKQaNWqoWDF+JEDhCtWe2LVrl3bt2qWHH344e+iWpMGDB8txHC1ZsiR/Txj4DaHaExdlZGRo2LBhGjZsmOrWrZuv51iU8VbzADp27JjuvPNO9e7dW/fff78qV67sKf/MmTNq27atfvjhBw0aNEg1a9bU559/rhEjRujgwYOaOnVq9mNHjBiht956S3v27FF8fLyn8yxYsECSGLxhXbj0hCQdOnRIklSxYkXPuUBehVNPAIUhFHti69atkpTrKl61atVUvXr17HXAhlDsiYumTp2qEydOaPTo0Xrvvfc8PjMweAfQoUOHNHPmTA0aNChf+VOmTFFycrK2bt2q+vXrS5IGDRqkatWqadKkSXrqqadUo0aNAtWYmZmpRYsWqUWLFqpXr16BjgX8lnDoCUk6fvy43njjDbVu3VpVq1Yt8PEAk3DpCaCwhGJPHDx4UJJcXw+qVq2qH3/8MV+1AnkRij1xsa4XX3xRkydPVrly5fJVW1HH+8oCqGTJknrwwQfznb948WK1bt1aFSpU0NGjR7M/EhISlJmZqfXr12c/ds6cOXIcx/NVjNWrV+vw4cNc7UahCIeeyMrKUt++fXXy5ElNmzYt37UCeREOPQEUplDsibNnz2bX9mulSpXKXgdsCMWekKRnn31WderU0YABA/JdW1HHFe8Auvzyy1WiRIl853/77bfasWOH4uLiXNePHDmS72NftGDBAhUvXlz33ntvgY8F/JZw6ImhQ4dq+fLlmjt3rq699toCHw/wJxx6AihModgTpUuXliTXOzafO3cuex2wIRR7YtOmTZo3b55Wr17N/UAKgME7gLx+I87MzMzx76ysLLVv316JiYmuj2/QoEG+a5N+/g3u0qVLlZCQ4PnvRYD8CPWeGDdunKZPn64//OEPeuCBBwp0LCAvQr0ngMIWij1x8S3mBw8ezPWW3IMHD6pFixaejwnkVSj2RGJiolq3bq3atWtr7969kqSjR49K+rkn9u3bp5o1a3o+blHD4F0IKlSooJMnT+aInT9/PvtviC6qW7eu0tLSlJCQYKWOZcuW6fTp07zNHEEXCj3x2muvaezYsRo+fLieffbZgB8f8CIUegIIJcHsiSZNmkiSNm/enGPI/vHHH3XgwAE9/PDDATsXkFfB7Il9+/YpJSVFtWvXzrXWtWtXlS9fPldtyI33ChSCunXr5vh7CkmaNWtWrt9Q3XPPPdq4caNWrFiR6xgnT57UhQsXsv+dn+3E3n77bcXExKh79+4enwEQWMHuiUWLFunxxx9X3759NWXKlHw+CyBwgt0TQKgJZk9cffXVuvLKK3Odb8aMGfL5fOrVq1d+nhJQIMHsiVmzZmnp0qU5PoYOHSpJmjx5cvaOSfCPK96FYMCAAXrkkUfUs2dPtW/fXtu3b9eKFStybVv0zDPPaNmyZercubP69++vpk2bKj09XV9++aWWLFmivXv3Zud4vf3/8ePH9fHHH6tnz54qW7asjacJ5Fkwe+Lf//63fve73yk2Nla33XZbrheLm266SXXq1An4cwb8CfbrxPr167N/oEtNTVV6erqSkpIkSW3atFGbNm0C/6QBP4LdE5MmTVLXrl3VoUMH9e7dWzt37tSrr76qAQMGqGHDhraeNmAUzJ7o0KFDrtjFK9xt27bNtfUe3DF4F4KBAwdqz549mj17tpYvX67WrVvrk08+0W233ZbjcTExMVq3bp1eeuklLV68WHPnzlW5cuXUoEEDjRs3TuXLl893DYsXL1ZGRob69OlT0KcDFFgwe2LXrl06f/68UlNT9dBDD+Vaf/PNNxm8UeiC/Trxz3/+U+PGjcsRe/755yVJY8aMYfBGoQt2T3Tu3Fnvvfeexo0bp6FDhyouLk4jR47UCy+8EIinB3gW7J5Awfkcx3GCXQQAAAAAAJGKv/EGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsYvAOE/Hx8erfv3+wywBCBj0B5ERPADnRE0BO9ERwMXjnwZw5c+Tz+bI/SpUqpQYNGmjIkCE6fPhwsMv7TWPHjs1R/68/NmzYEOwSEWbCvSd2796txMRENWnSRJdccomqVq2qTp06afPmzcEuDWEq3HtCkiZMmKCuXbuqcuXK8vl8Gjt2bLBLQhiLhJ7IysrSyy+/rNq1a6tUqVJq3LixFi5cGOyyEKYioSe+++479erVSxUqVFBMTIxatWqlNWvWBLussBEV7ALCyfjx41W7dm2dO3dOn332mWbMmKGPPvpIO3fuVExMTLDLM+rRo4fq1auXKz5y5EilpaWpefPmQagKkSBce+KNN97Q7Nmz1bNnTw0ePFinTp3S66+/rhtvvFHLly9XQkJCsEtEmArXnpCk0aNHq0qVKrruuuu0YsWKYJeDCBHOPTFq1Cj94Q9/0MCBA9W8eXN98MEH6tOnj3w+n3r37h3s8hCmwrUn9u/fr5YtW6p48eJ65plnVKZMGb355pvq0KGDVq9erTZt2gS7xNDn4De9+eabjiTnP//5T474k08+6Uhy3n77bWNuWlpaQGqoVauW069fv4Acy3EcZ9++fY7P53MGDhwYsGOi6Aj3nti8ebNz+vTpHLGjR486cXFxzs033xyA6lDUhHtPOI7j7Nmzx3Ecx0lNTXUkOWPGjAlIXSiawr0nDhw44ERHRzuPPfZYdiwrK8tp3bq1U716defChQsBqRFFR7j3xODBg52oqChn9+7d2bH09HSnRo0azvXXXx+Q+iIdbzUvgFtvvVWStGfPHklS//79VbZsWSUnJ6tjx4665JJL1LdvX0k/v11p6tSpuvrqq1WqVClVrlxZgwYN0okTJ3Ic03EcJSUlqXr16oqJidEtt9yir776yvX8ycnJSk5OzlftCxculOM42fUBgRAuPdG0aVOVLVs2Ryw2NlatW7fW119/7fl5Aybh0hPSz3/7B9gWLj3xwQcfKCMjQ4MHD86O+Xw+Pfroozpw4IA2btyYr+cP/Fq49MSnn36q6667TldccUV2LCYmRl27dtUXX3yhb7/9Nl/PvyjhreYFcPGLNDY2Njt24cIF3X777WrVqpUmT56c/ZaRQYMGac6cOXrwwQf1+OOPa8+ePXr11Ve1detWbdiwQdHR0ZKkF154QUlJSerYsaM6duyoL774Qh06dND58+dznf+2226TJO3du9dz7QsWLFCNGjV4WwgCKpx7QpIOHTqkihUr5isXcBPuPQEEWrj0xNatW1WmTBk1bNgwR7xFixbZ661atcrfJwH4hXDpiZ9++kkVKlTIFb9Y25YtW1S/fn3vn4CiJJiX28PFxbeGrFq1yklNTXX279/v/O1vf3NiY2Od0qVLOwcOHHAcx3H69evnSHKee+65HPmffvqpI8lZsGBBjvjy5ctzxI8cOeKUKFHC6dSpk5OVlZX9uJEjRzqScr01pFatWk6tWrU8P5+dO3c6kpzExETPuYDjRF5POI7jrF+/3vH5fM7zzz+fr3wUbZHUE7zVHIEQ7j3RqVMnp06dOrni6enprvUCvyXce6JLly7OpZde6vzvf//LEW/ZsqUjyZk8eXJePxVFFm819yAhIUFxcXGqUaOGevfurbJly2rp0qW6/PLLczzu0UcfzfHvxYsXq3z58mrfvr2OHj2a/XHx7a4X7wa4atUqnT9/XkOHDpXP58vOHz58uGs9e/fuzffVbkm8zRwFFik9ceTIEfXp00e1a9dWYmKi53zgokjpCSBQwrUnzp49q5IlS+aKlypVKnsdyI9w7YlHH31UJ0+e1L333qutW7fqm2++0fDhw7N3hKEnfhtvNffgtddeU4MGDRQVFaXKlSvriiuuULFiOX93ERUVperVq+eIffvttzp16pQqVarketwjR45IklJSUiQp19s04uLiXN/akR+O4+jtt99Wo0aN1Lhx44AcE0VXJPREenq6OnfurNOnT+uzzz7L9bffgBeR0BNAIIVrT5QuXVo//fRTrvi5c+ey14H8CNeeuPPOOzVt2jQ999xzuv766yVJ9erV04QJE5SYmMjPT3nA4O1BixYt1KxZM7+PKVmyZK7mycrKUqVKlbKvNP9aXFxcwGr8LRs2bFBKSoomTpxYaOdE5Ar3njh//rx69OihHTt2aMWKFWrUqFGhnBeRK9x7Agi0cO2JqlWras2aNXIcJ8dVw4MHD0qSqlWrZvX8iFzh2hOSNGTIED344IPasWOHSpQooSZNmmj27NmSpAYNGlg/f7hj8C4EdevW1apVq3TzzTf7/Q1prVq1JP38G606depkx1NTU3PdrTC/FixYIJ/Ppz59+gTkeEB+hEJPZGVl6Xe/+51Wr16td955R23bti3Q8YCCCIWeAEJJsHuiSZMmeuONN/T111/rqquuyo7/61//yl4HClOwe+KiMmXKqGXLltn/XrVqlUqXLq2bb765wMeOdPyNdyG45557lJmZqRdffDHX2oULF3Ty5ElJP//NR3R0tKZNmybHcbIfM3XqVNfjet1OLCMjQ4sXL1arVq1Us2ZNT88BCKRQ6ImhQ4dq0aJFmj59unr06OH5OQCBFAo9AYSSYPfEXXfdpejoaE2fPj075jiOZs6cqcsvv1w33XSTtycEFFCwe8LN559/rvfee0+///3vVb58+XwdoyjhinchaNu2rQYNGqSJEydq27Zt6tChg6Kjo/Xtt99q8eLFeuWVV9SrVy/FxcXp6aef1sSJE9W5c2d17NhRW7du1ccff+y6xZHXbWJWrFihY8eOcVM1BF2we2Lq1KmaPn26WrZsqZiYGM2fPz/Hevfu3VWmTJmAPV/gtwS7JyRp3rx5SklJ0ZkzZyRJ69evV1JSkiTpgQceyL6KAhSGYPdE9erVNXz4cE2aNEkZGRlq3ry53n//fX366adasGCBihcvbuNpA0bB7omUlBTdc8896tq1q6pUqaKvvvpKM2fOVOPGjfXSSy/ZeMoRh8G7kMycOVNNmzbV66+/rpEjRyoqKkrx8fG6//77c7w1IykpSaVKldLMmTO1Zs0a3XDDDVq5cqU6depU4BoWLFig6Oho3X333QU+FlBQweyJbdu2SZI2btyojRs35lrfs2cPgzcKXbBfJ2bPnq1169Zl/3vNmjXZd8lt1aoVgzcKXbB74g9/+IMqVKig119/XXPmzFH9+vU1f/58/lwPQRPMnihXrpyqVq2qV199VcePH9fll1+uxx9/XKNGjdIll1wSiKcX8XzOL9+DAAAAAAAAAoq/8QYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwKKovD7Q5/PZrAMIioJsY09PIBLRE0Bu+e0LegKRiNcJILe89AVXvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwRsAAAAAAIuigl0AgPD19NNPu8ZLly5tzGncuLFrvFevXp7PP2PGDOPaxo0bXePz5s3zfB4AAACgILjiDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEU+x3GcPD3Q57NdC1Do8vjl76qo9MSiRYuMa/m5E3lhSU5Odo0nJCQYc/bt22ernLBBT0S2Bg0auMZ3795tzBk2bJhrfNq0aQGpKRzkty/oifwpU6aMa3zSpEnGnEGDBrnGt2zZYsy5++67XeMpKSl+qgOvE0BueekLrngDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWRQW7AAChwbRtWKC3DDNtW7RixQpjTp06dVzjXbp0MebUrVvXNd63b19jzsSJE41rQCS47rrrXONZWVnGnAMHDtgqB3BVtWpV1/jAgQONOaav4aZNmxpzOnfu7Bp/7bXX/FQHFMz1119vXHvvvfdc4/Hx8ZaqsatDhw7Gta+//to1vn//flvlBB1XvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIu5oDRUizZs2Ma927d/d8vK+++so13rVrV2PO0aNHXeNpaWnGnBIlSrjGN23aZMy59tprXeOxsbHGHCDSNWnSxDWenp5uzFm6dKmlalCUxcXFGdfeeuutQqwEKFy33367ca1kyZKFWIl9/nafeeihh1zjvXv3tlVO0HHFGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsCgithPr1auXa3zgwIHGnB9//NE1fu7cOWPOggULXOOHDh0y5nz33XfGNaCwVa1a1bjm8/lc46YtwyTzlhgHDx70VthveOqpp1zjV111ledj/eMf/yhoOUBIa9SokXFtyJAhrvF58+bZKgdF3OOPP+4a79atmzGnRYsWlqrJqU2bNq7xYsXM16W2b9/uGl+/fn1AakLkiIpyH7M6duxYyJUEz5YtW4xrTz75pGu8TJkyxhx/W1+GA654AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeAMAAAAAYFFE3NX85Zdfdo3Hx8cH9DyDBg1yjZ8+fdqY4++O0OHowIEDrnHT/4Ekbd682VY58Ojvf/+7ca1evXqucX9f38ePHy9wTXnRu3dv13h0dHShnB8IJ1deeaVxzXS32EWLFtkqB0Xc//3f/7nGs7KyCrmS3Hr06OEpLkkpKSmu8XvvvdeY4+/Ozohct9xyi2u8ZcuWxhx/P0+HowoVKhjXTDvTxMTEGHO4qzkAAAAAADBi8AYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsYvAGAAAAAMCiiNhObODAga7xxo0bG3O+/vpr13jDhg2NOddff71rvF27dsacG2+80TW+f/9+Y06NGjWMa15duHDBuJaamuoar1q1qufz7Nu3z7jGdmLhwbRFSmF55plnjGsNGjTwfLx//etfnuJApEhMTDSumfqc79MoiI8++si4VqxYcK/xHDt2zLiWlpbmGq9Vq5Yxp3bt2q7xf//738ac4sWLG9cQ3ho1amRcW7hwoWs8OTnZmPPSSy8VuKZQctdddwW7hJDCFW8AAAAAACxi8AYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsioi7mq9evdpT3J/ly5d7zqlQoYJxrUmTJq7xLVu2GHOaN2/uuQaTc+fOGde++eYb17jpju+SdNlll7nG/d2hEfilzp07u8bHjx9vzClRooRr/MiRI8acESNGuMbPnDnjpzogPMTHxxvXmjVrZlwzfd9PT08vaEkoAtq2besav+KKK4w5WVlZnuL5NXPmTNf4ypUrjTmnTp1yjd96663GnFGjRnkrTNKjjz7qGp8xY4bnYyG0jB492rhWpkwZ1/gdd9xhzDHdaT/UmeYD0/cMKfDfA8IBV7wBAAAAALCIwRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLImI7sWA7ceKEcW3NmjWej5efbdDyo2fPnq5xf9ujffnll67xRYsWBaQmRD7TVkemLcP88fd1t27dOs/HA8KFvy1a/ElNTQ1wJYg0/raq+9vf/uYar1ixYkBrSElJcY2/++67xpxx48a5xvOzhaTp/JL08MMPu8bj4uKMOS+//LJrvFSpUsacV1991TWekZFhzIE9vXr1co137NjRmPPdd9+5xjdv3hyQmkKJaZs9f1uGrV271jV+8uTJAFQUmrjiDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEXc1TzCVapUybg2ffp013ixYubfx4wfP941fvz4cW+FIaK9//77xrUOHTp4Pt7cuXNd46NHj/Z8LCASXHPNNfnKM91dGbgoKsr8o2Eg717ub+eJ3r17u8aPHj0asPP74++u5hMnTnSNT5kyxZgTExPjGvfXj8uWLXONJycnG3Ngz9133+0aN/3fSuafs8OVvx0P+vbt6xrPzMw05iQlJbnGI/nO/VzxBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALGI7sQj32GOPGdfi4uJc4ydOnDDm/Pe//y1wTYgcVatWdY3fdNNNxpySJUu6xv1tE2PaciItLc1PdUD4u/HGG13jDz74oDFn69atxrVPPvmkwDUBXmzevNk1/tBDDxlzCmvbsPwwbfNl2k5Jkpo3b26rHARQ+fLljWum78X+zJgxoyDlhJyHH37YuGbaavDrr7825qxZs6bANYUbrngDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEXc1jxA333yza/y5557zfKxu3boZ13bu3On5eIhc7777rms8NjbW87Hmz59vXEtOTvZ8PCASJCQkuMYvu+wyY87y5cuNa+fOnStwTSi6ihXzfr3mhhtusFBJ8Ph8Pte4v89Nfj5vY8eOdY0/8MADno+FvDHtuiJJl19+uWt84cKFtsoJOXXr1vWcw9yQE1e8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAi9hOLEJ07NjRNR4dHW3MWb16tWt848aNAakJkaFr167Gteuvv97z8dauXesaHzNmjOdjAZHu2muvdY07jmPMWbJkia1yUAQ88sgjxrWsrKxCrCQ0denSxTV+3XXXGXNMnzd/n0/TdmKw5/Tp08a1bdu2ucYbN25szDFt+3j8+HFPdRW2SpUqucZ79erl+VifffZZQcuJKFzxBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALGI7sTBSunRp49odd9zhGj9//rwxx7R9U0ZGhrfCEBFiY2Nd4yNHjjTm+NuuzsS0JUdaWprnYwGRoEqVKsa11q1bu8b/+9//GnOWLl1a4JpQdJm2y4pEcXFxrvGrrrrKmOPvNdGr1NRU4xo/ixW+s2fPGteSk5Nd4z179jTm/OMf/3CNT5kyxVth+dSoUSPjWp06dYxr8fHxrnF/21iasAVhTlzxBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCLuah5GnnnmGePadddd5xpfvny5Mefzzz8vcE2IHE899ZRrvHnz5p6P9f777xvXTHfTB4qq/v37G9cqVarkGv/4448tVQMUHaNGjXKNP/bYYwE9z969e13j/fr1M+bs27cvoDWgYEw/u/h8PmNOp06dXOMLFy4MSE2/5ejRo8Y1f3cor1ixYsBqmDNnTsCOFQm44g0AAAAAgEUM3gAAAAAAWMTgDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjEdmIhyLT9wPPPP2/M+d///ucaHz9+fEBqQuR78sknA3asIUOGGNfS0tICdh4gEtSqVctzzokTJyxUAkSejz76yLh2xRVXFEoNu3btco1/9tlnhXJ+FNzu3btd4/fcc48xp0mTJq7xevXqBaKk37RkyZJ85b311luu8b59+3o+1tmzZ/NVQ6TiijcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWcVfzIImNjTWu/fnPf3aNFy9e3Jhjumvnpk2bvBUGBMBll11mXMvIyCiUGk6dOuX5/NHR0a7x8uXLez7/pZdealwL5B3kMzMzjWvPPvusa/zMmTMBOz8KrnPnzp5z/v73v1uoBJB8Pp9xrVgx79dr7rzzTs85s2bNco1Xq1bN87H81ZyVleX5ePnRpUuXQjkPQsu2bds8xUPF999/H7BjNWrUyLi2c+fOgJ0nXHDFGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsIjtxCwzbQG2fPlyY07t2rVd48nJycac559/3lthgEU7duwIdglavHixa/zgwYPGnMqVK7vG77333oDUVNgOHTrkGp8wYUIhVwJJatWqlWu8SpUqhVwJYDZjxgzj2ssvv+z5eB9++KFrPD9beQV6+69AHm/mzJkBOxYQTKYtBf1tNWhSFLcM84cr3gAAAAAAWMTgDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjEXc0tq1u3rmu8adOmno/15JNPGtf83fEcyIuPPvrINX7XXXcVciWBcffddxfKeS5cuOAaz8/dcpctW2Zc27x5s+fjffrpp55zYE/37t1d46bdLyRp69atrvH169cHpCbg19577z3j2jPPPOMaj4uLs1WOVampqa7xr7/+2pjz8MMPu8b97ZgBhBPHcTzFkXdc8QYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxiO7EAqFWrlnFt5cqVno9n2q7jww8/9HwsIK969OjhGk9MTDTmREdHB+z8V199tXHt3nvvDdh5/vrXvxrX9u7d6/l47777rmt89+7dno+F8BcTE2Nc69ixo+fjLVmyxDWemZnp+VhAXqSkpBjXevfu7Rrv1q2bMWfYsGEFLcmaCRMmuMZfe+21Qq4ECB2lSpXynHP27FkLlUQerngDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgkc9xHCdPD/T5bNcStkx3xZSkESNGeD5eixYtXOObN2/2fCz4l8cvf1f0BCIRPVEw/u70v27dOtf4kSNHjDl9+vRxjZ85c8ZbYSiQ/PYFPSHdcccdrvGHH37YmNOlSxfX+LJly4w5s2bNco37+z/YtWuXa3zfvn3GHPA6EekOHTrkGo+KMm+G9eKLL7rGX3nllYDUFA7y0hdc8QYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxiOzEPWrVq5Rr/6KOPjDlly5b1fB62Eys8bIkB5ERPALmxnRjw//E6Edn+/ve/u8anTJlizFmzZo2tcsIG24kBAAAAABBkDN4AAAAAAFjE4A0AAAAAgEUM3gAAAAAAWMTgDQAAAACARVHBLiCctG7d2jWenzuXJycnG9fS0tI8Hw8AAAAACqJLly7BLiFiccUbAAAAAACLGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiO3ELNu+fbtr/LbbbjPmHD9+3FY5AAAAAIBCxhVvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALPI5juPk6YE+n+1agEKXxy9/V/QEIhE9AeSW376gJxCJeJ0AcstLX3DFGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsCjP24kBAAAAAADvuOINAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEUM3gAAAAAAWMTgDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN6/wefz5elj7dq1wS7V1aJFi3T//ferfv368vl8ateuXbBLQgQI9774peTkZJUqVUo+n0+bN28OdjkIU+HeE2lpaRo+fLiqV6+ukiVLqmHDhpoxY0awy0IYC/eeiI+Pd633kUceCXZpCFPh3hO8ThRcVLALCHXz5s3L8e+5c+fqk08+yRVv2LBhYZaVZzNmzNCWLVvUvHlzHTt2LNjlIEKEe1/80hNPPKGoqCj99NNPwS4FYSyceyIzM1O33367Nm/erMcee0z169fXihUrNHjwYJ04cUIjR44MdokIQ+HcExc1adJETz31VI5YgwYNglQNwl049wSvEwHiwJPHHnvMycunLT09vRCq+W379u1zMjMzHcdxnKuvvtpp27ZtcAtCRAq3vrho+fLlTokSJZzRo0c7kpz//Oc/wS4JESKceuKdd95xJDmzZ8/OEe/Zs6dTqlQp5/Dhw0GqDJEknHrCcRynVq1aTqdOnYJdBiJYOPUErxOBwVvNA6Bdu3Zq1KiRtmzZojZt2igmJib7Nz8+n09jx47NlRMfH6/+/fvniJ08eVLDhw9XjRo1VLJkSdWrV09//OMflZWVleNxBw8e1O7du5WRkfGbtdWoUUPFivHfjMIXyn0hSRkZGRo2bJiGDRumunXr5us5Al6Eak98+umnkqTevXvniPfu3Vvnzp3TBx984PGZAnkTqj3xS+fPn1d6errn5wbkR6j2BK8TgcFEFiDHjh3TnXfeqSZNmmjq1Km65ZZbPOWfOXNGbdu21fz58/W73/1Of/7zn3XzzTdrxIgRevLJJ3M8dsSIEWrYsKF++OGHQD4FIOBCuS+mTp2qEydOaPTo0Z5qAgoiFHvip59+UvHixVWiRIkc8ZiYGEnSli1bPNUIeBGKPXHRP//5T8XExKhs2bKKj4/XK6+84qk2ID9CsSd4nQgM/sY7QA4dOqSZM2dq0KBB+cqfMmWKkpOTtXXrVtWvX1+SNGjQIFWrVk2TJk3SU089pRo1agSyZMC6UO2LQ4cO6cUXX9TkyZNVrly5fNUG5Eco9sQVV1yhzMxMbdq0Sa1atcqOX7zCwS95YVMo9oQkNW7cWK1atdIVV1yhY8eOac6cORo+fLh+/PFH/fGPf8xXrUBehGJP8DoRGFzxDpCSJUvqwQcfzHf+4sWL1bp1a1WoUEFHjx7N/khISFBmZqbWr1+f/dg5c+bIcRzFx8cHoHLAnlDti2effVZ16tTRgAED8l0bkB+h2BN9+vRR+fLl9dBDD+mTTz7R3r17NWvWLE2fPl2SdPbs2XzXC/yWUOwJSVq2bJkSExN111136aGHHtK6det0++23a8qUKTpw4EC+6wV+Syj2BK8TgcEV7wC5/PLLc739wotvv/1WO3bsUFxcnOv6kSNH8n1sIFhCsS82bdqkefPmafXq1dz/AIUuFHuiSpUqWrZsmR544AF16NBBklSuXDlNmzZN/fr1U9myZfNdL/BbQrEn3Ph8Pj3xxBNasWKF1q5dq/vvvz8gxwV+LRR7gteJwGDwDpDSpUt7enxmZmaOf2dlZal9+/ZKTEx0fTzbVyAchWJfJCYmqnXr1qpdu7b27t0rSTp69Kikn28ysm/fPtWsWdPzcYG8CMWekKQ2bdro+++/15dffqn09HRde+21+vHHHwt0TCAvQrUn3Fx8e+7x48cDdkzg10K1J3idKDgGb8sqVKigkydP5oidP39eBw8ezBGrW7eu0tLSlJCQUIjVAcERzL7Yt2+fUlJSVLt27VxrXbt2Vfny5XPVBtgWCq8VxYsXV5MmTbL/vWrVKknidQlBEQo98Wvff/+9JBmvJAI2hUJP8DpRMLzP0rK6devm+FsKSZo1a1au307dc8892rhxo1asWJHrGCdPntSFCxey/52f7TCAUBLMvpg1a5aWLl2a42Po0KGSpMmTJ2vBggX5fVpAvoXaa0Vqaqr++Mc/qnHjxvxAhaAIZk8cP34813kyMjL0hz/8QSVKlPB8l2kgEHidCH9c8bZswIABeuSRR9SzZ0+1b99e27dv14oVK1SxYsUcj3vmmWe0bNkyde7cWf3791fTpk2Vnp6uL7/8UkuWLNHevXuzc0aMGKG33npLe/bs+c2bIaxfvz67SVNTU5Wenq6kpCRJP79lpE2bNoF/0sBvCGZfXPzbpF+6+Bvktm3bqlmzZgF7nkBeBfu1om3btmrZsqXq1aunQ4cOadasWUpLS9OHH37IvRAQFMHsiWXLlikpKUm9evVS7dq1dfz4cb399tvauXOnXnrpJVWpUsXmUwdc8ToR/hi8LRs4cKD27Nmj2bNna/ny5WrdurU++eQT3XbbbTkeFxMTo3Xr1umll17S4sWLNXfuXJUrV04NGjTQuHHjVL58+Xyd/5///KfGjRuXI/b8889LksaMGcPgjaAIdl8AoSbYPdG0aVMtXrxYP/zwg8qVK6f27dvrxRdfVJ06dQLx9ADPgtkT11xzja666irNnz9fqampKlGihJo0aaJ33nlHd999d6CeIuAJrxPhz+c4jhPsIgAAAAAAiFS8LwAAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIwTtMxMfHq3///sEuAwgZ9ASQEz0B5ERPADnRE8HF4J0Hc+bMkc/ny/4oVaqUGjRooCFDhujw4cPBLi9PJkyYoK5du6py5cry+XwaO3ZssEtCGIuEnjh48KAefvhh1a5dW6VLl1bdunX15JNP6tixY8EuDWEoEnpCkpKTk9WnTx9VqlRJpUuXVv369TVq1Khgl4UwFO49sXv3biUmJqpJkya65JJLVLVqVXXq1EmbN28OdmkIU+HeE2PHjs1R/68/NmzYEOwSQ15UsAsIJ+PHj1ft2rV17tw5ffbZZ5oxY4Y++ugj7dy5UzExMcEuz6/Ro0erSpUquu6667RixYpgl4MIEa49kZaWppYtWyo9PV2DBw9WjRo1tH37dr366qtas2aNtmzZomLF+L0kvAvXnpCkbdu2qV27drr88sv11FNPKTY2Vvv27dP+/fuDXRrCWLj2xBtvvKHZs2erZ8+eGjx4sE6dOqXXX39dN954o5YvX66EhIRgl4gwFa490aNHD9WrVy9XfOTIkUpLS1Pz5s2DUFV4YfD24M4771SzZs0kSQMGDFBsbKymTJmiDz74QPfdd59rTnp6usqUKVOYZbras2eP4uPjdfToUcXFxQW7HESIcO2JZcuWKSUlRR9++KE6deqUHb/ssss0fvx4bd++Xdddd10QK0S4CteeyMrK0gMPPKArr7xSa9asUenSpYNaDyJHuPbEfffdp7Fjx6ps2bLZsYceekgNGzbU2LFjGbyRb+HaE40bN1bjxo1zxPbv368DBw5owIABKlGiRJAqCx9c0imAW2+9VdLPQ60k9e/fX2XLllVycrI6duyoSy65RH379pX08w81U6dO1dVXX61SpUqpcuXKGjRokE6cOJHjmI7jKCkpSdWrV1dMTIxuueUWffXVV67nT05OVnJycp5qjY+Pz+ezBPIuXHrif//7nySpcuXKOeJVq1aVJIYOBEy49MTKlSu1c+dOjRkzRqVLl9aZM2eUmZlZkKcOuAqXnmjatGmOoVuSYmNj1bp1a3399deenzdgEi494WbhwoVyHCe7PvjHFe8CuPhFGhsbmx27cOGCbr/9drVq1UqTJ0/OfsvIoEGDNGfOHD344IN6/PHHtWfPHr366qvaunWrNmzYoOjoaEnSCy+8oKSkJHXs2FEdO3bUF198oQ4dOuj8+fO5zn/bbbdJkvbu3Wv5mQJ5Ey490aZNGxUrVkzDhg3Tn/70J1WvXl07duzQhAkT1K1bN1155ZWB+HQAYdMTq1atkiSVLFlSzZo105YtW1SiRAl1795d06dP12WXXVbgzwUghU9PmBw6dEgVK1bMVy7gJpx7YsGCBapRo4batGnjObdIcvCb3nzzTUeSs2rVKic1NdXZv3+/87e//c2JjY11Spcu7Rw4cMBxHMfp16+fI8l57rnncuR/+umnjiRnwYIFOeLLly/PET9y5IhTokQJp1OnTk5WVlb240aOHOlIcvr165cjv1atWk6tWrU8PZfU1FRHkjNmzBhPecAvRUJPvPHGG86ll17qSMr+6Nevn5ORkeHxswGEf0907drVkeTExsY6ffv2dZYsWeI8//zzTlRUlHPTTTflOBeQF+HeE27Wr1/v+Hw+5/nnn89XPoq2SOuJnTt3OpKcxMREz7lFFYN3HlxslF9/1KpVy1m+fHn24y42SkpKSo78xx9/3Clfvrxz5MgRJzU1NcdH2bJlnQEDBjiO4zhvv/22IynHMR3n5wZya5T8YPBGIERCT3z88cdOhw4dnKlTpzpLly51nnzySScqKsp56qmn8n1MFF3h3hO33nqrI8m54447csQnTpzoSHI++eSTfB0XRVe498SvHT582KlevbpTp04d5/Tp0wE5JoqWSOuJESNGOJKc7du3B+R4RQFvNffgtddeU4MGDRQVFaXKlSvriiuuyHXn46ioKFWvXj1H7Ntvv9WpU6dUqVIl1+MeOXJEkpSSkiJJql+/fo71uLg4VahQIVBPAwiYcO2JDRs2qHPnztq0aVP2DU66deumcuXKady4cXrooYd01VVX5fv4KLrCtScu3tfg1zf26dOnj0aMGKHPP/+cm0khX8K1J34pPT1dnTt31unTp/XZZ5/l+ttvwItI6AnHcfT222+rUaNGuW64BjMGbw9atGiR/UO6ScmSJXM1T1ZWlipVqqQFCxa45nCXcYSrcO2J119/XZUrV85Ve9euXTV27Fh9/vnnDN7Il3DtiWrVqknKfcPBiz/g/frGPUBehWtPXHT+/Hn16NFDO3bs0IoVK9SoUaNCOS8iV7j3hPTzBYyUlBRNnDix0M4ZCRi8C0HdunW1atUq3XzzzX7vllyrVi1JP/9Gq06dOtnx1NRUfuhBRAl2Txw+fNj1js0ZGRmSfr6pCVCYgt0TTZs21V/+8hf98MMPOeI//vijJH5BjMIX7J6Qfh50fve732n16tV655131LZt2wIdDyiIUOiJixYsWCCfz6c+ffoE5HhFBduJFYJ77rlHmZmZevHFF3OtXbhwQSdPnpQkJSQkKDo6WtOmTZPjONmPmTp1qutxC3L7fyCYgt0TDRo00OHDh7V27doc8YULF0oSe3ij0AW7J+666y6VLFlSb775prKysrLjb7zxhiSpffv2Hp4NUHDB7glJGjp0qBYtWqTp06erR48enp8DEEih0BPSzxcpFi9erFatWqlmzZqenkNRxxXvQtC2bVsNGjRIEydO1LZt29ShQwdFR0fr22+/1eLFi/XKK6+oV69eiouL09NPP62JEyeqc+fO6tixo7Zu3aqPP/7YdesKL7f/nzdvnlJSUnTmzBlJ0vr165WUlCRJeuCBB7J/OwYUhmD3xJAhQ/Tmm2+qS5cuGjp0qGrVqqV169Zp4cKFat++vW644QYbTxswCnZPVKlSRaNGjdILL7ygO+64Q926ddP27dv1l7/8Rffdd5+aN29u42kDRsHuialTp2r69Olq2bKlYmJiNH/+/Bzr3bt3V5kyZQL2fIHfEuyeuGjFihU6duwYe3fnA4N3IZk5c6aaNm2q119/XSNHjlRUVJTi4+N1//336+abb85+XFJSkkqVKqWZM2dqzZo1uuGGG7Ry5Up16tSpQOefPXu21q1bl/3vNWvWaM2aNZKkVq1aMXij0AWzJ6644gpt2bJFo0eP1vz583Xo0CFVq1ZNTz/9tMaNGxeIpwd4FuzXidGjR6tChQqaNm2ahg8fnmMYB4IhmD2xbds2SdLGjRu1cePGXOt79uxh8EahC/brhPTz28yjo6N19913F/hYRY3P+eV7EAAAAAAAQEDxN94AAAAAAFjE4A0AAAAAgEUM3gAAAAAAWMTgDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFgUldcH+nw+m3UAQVGQbezpCUQiegLILb99QU8gEvE6AeSWl77gijcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWMXgDAAAAAGBRVLALAAAAAEJFhQoVjGs1a9YM2HlSUlKMa0888YRrfOfOncacb775xjW+fft2b4UBsIIr3gAAAAAAWMTgDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEVsJxYhunTp4hpftmyZMWfIkCGu8ZkzZxpzMjMzvRWGsFepUiXj2jvvvOMa//zzz405s2bNco3v3bvXU12hrnz58sa1Nm3auMaXL19uzMnIyChwTQBQ1HTq1Mm41rVrV9d4u3btjDn16tUraEnZTNt/SVKtWrVc4yVLlvR8nuLFi3vOARB4XPEGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALGLwBgAAAADAIu5qDgBAIShXrpxrfOLEicacRo0aucYTEhKMOdwBH+Gubt26rvHHHnvMmDNw4EDXeOnSpY05Pp/PW2EB1qBBg6CeH0DhYvAOI7Gxsca16dOnez7eq6++6hr/61//asw5e/as5/MgPFSoUME1/tVXXxlzTFtmHT582JhTVLYN27JlizEnLi7ONd60aVNjznfffeetMAAAAIQM3moOAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEUM3gAAAAAAWMTgDQAAAACARdzVPIy0adPGuFa9enXPx1u4cKFr/Ny5c56PhfBQsWJF49qiRYtc45dddpkxx3Q3/aFDh3orLIyNHj3aNV67dm1jzqBBg1zj3Lk8/PXt29e4NmHCBNd4jRo1PJ/HtDWZJB07dszz8YBQYvqZZtiwYYVcSWDs3r3bNe5v1xAgr+rVq+ca9/czX/fu3Y1r7dq1c41nZWUZc2bOnOka37BhgzGnKP7MwxVvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALOKu5gAAeGS66/LUqVONObGxsa5xx3E8n3/atGnGtSFDhhjXjh8/7vlcwEWmuyT7u9u46a7Gy5cvN+b89NNPrvFTp04Zc9LT013jZcqUMeasXLnSNb5z505jzr/+9S/X+NatW405Z8+edY2bakbR1ahRI9e4v+/rPXr0cI37u6t5oN1www2u8QsXLhhz/vvf/7rGP/vsM2OO6XvN+fPn/VQXOhi8Q1DJkiVd46NGjQroeebNm+caz88PgQgP119/vXHNtH2EP+PHjy9ANeHj6quvNq499dRTrvGlS5cac0xbtwEAACAy8VZzAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALOKu5iHommuucY03bdrU87H83cb/448/9nw8hIdKlSq5xnv27On5WL///e+Na6mpqZ6PF8pMdy9ftWqV52P5u6v56dOnPR8PoeXpp592jV922WWFcv57773XuHbHHXcY1yZMmOAa97c9Wbhs04LAyM/2W9dee60xp3v37p5r2LRpk2vc384ce/fudY3XrFnTmHPgwAHXeFZWlrk4II8aN27sGn/ssceMOabv7eXKlfN8/h9++MG49umnnxrX9uzZ4xpPTEw05mzZssU13qJFC2OO6fWyY8eOxpzt27e7xmfOnGnMCSVc8QYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAi7moOAICLWrVqGdcefPBBz8fbsWOHa/zw4cPGnISEBM/nKV++vHHNdDf2BQsWGHMOHTrkuQaEvhIlSrjG3377bWOO6e7lL730kjEnP7tCmJjuXO7Pvn37AnZ+4Ndef/1145rpjv4VK1b0fJ7Vq1cb17788kvX+MiRI405586d81zDTTfdZFx79NFHXeN//etfjTlNmjRxjft7TXzttddc4++++64xJ5R24GHwDkH52fLJxLT1ByLbn/70J9f4/fffb8wxbQWxePHigNQUDlq3bu0ar1y5sjFnzpw5rvH58+cHoiQAAABEAN5qDgAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEXc1TwEtWnTxnPO+fPnXeOjRo0qaDkIQ47juMazsrKMOT/++KNr3PS1FepKly7tGve3vcbgwYNd46bPpyQ99NBD3gpD2DBtdSJJl1xyiWv8008/Nea0bdvWNV6qVCljzn333eca9/d1XLduXeNalSpVXOMffPCBMefOO+90jR8/ftyYg9BQtmxZ49qIESNc4507dzbmHD161DU+efJkY86ZM2eMa0Ao8fe9ODEx0TU+YMAAY47P53ON+9veasaMGa7xSZMmGXPS09ONa4EUGxtrXCtevLhrfOzYscac5cuXu8b9beUZ7rjiDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEXc1RwAABclS5Y0rpnudP9///d/ns9z7tw549qbb77pGr/77ruNOXXq1PFcg787T4frzgaQunXrZlx77rnnXOP79u0z5rRu3do1furUKU91AaGoXbt2xrVnnnnGNW66c7kk/fDDD67xnj17GnP+/e9/G9cCyXQXckmqUaOGa3zu3LnGnI8++sg1XqFCBW+Fyf/ndN68ea7xkydPej5PMDB4B8lNN92UrzUT01YC27Zt83wsFE2dOnVyja9cudKYY/pGZ9oOI9BM2zNJ5hfQG2+80fN5lixZ4jkHAAAAuIi3mgMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARdzUPkubNmwf0eIV1F2mEh1deecU1fssttxhzqlWr5hpv06aNMce05UPXrl39VBc4/racMG335M/333/vGh85cqTnYyH83XfffZ5zTLsDSNL7779fgGpyatasWcCOJUmbNm0yrqWlpQX0XCg8+dklZevWrca1AwcOFKQcIKT522IrMzPT8/EuXLjgGr/hhhuMOb169XKNX3nllZ7Pf/bsWeNaw4YNPa8dPXrUmFO5cuW8F/YbDh8+bFxLSkpyjWdkZATs/DZxxRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLuKs5AAAuFi5caFwz3bnf344VprvSXnPNNcac7t27u8YrVKhgzDl58qRxzZQ3cOBAY868efNc47t27TLmIDSY7pDszx133GFcGzNmjGv8gw8+MOZs27bNcw1AMPzzn/80rq1Zs8Y1npCQYMypWbOma/zPf/6zMSc/O7KY7rju7y7t+ZGfO5dnZWUZ15YuXeoaf/zxx405Bw8e9FxDKGHwDpL8bAXj74cpthPDL23ZssU13rhxY2NOkyZNXOP+fgh75plnXOOpqanGnLfeesu45pVpIJCk7du3ez7e559/7hpPTk72fCwAAADgIt5qDgAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEU+J4/3rff5fLZriUitWrVyja9bt86YU6yY++9DUlJSjDnx8fGe6sLP8rNtw0X0RPDVqVPHuPbdd9+5xv1tbXP77be7xv3dpT3S0BP/32WXXWZcM319lS9f3phj+vzk53O+atUq49pjjz1mXPvwww9d4/Xr1zfm/OUvf3GNP/LII8acSJPfvgh2T/ir2982P175O9bMmTNd45s2bTLmmLZhMvWdJH311VfGNZOrr77aNb5x40ZjzoEDBzyfJ9LwOvH/XXrppca15557zjV+8803G3OOHTvmGt+3b58xp2TJkq7xa6+91pjTokUL41ogmfpfkkaOHOka97eLUyjLS19wxRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALAoKtgFRLrY2FjXuGnLMH8++eSTgpYDRJQXXnjBuGba1uHZZ5815hSlbcPw244fP25cu+eee1zjS5YsMeb422rMZNq0aa5xf1/H586dM6699957rnHTtjeSeZu9unXrGnOSk5ONayg8kydPNq49+eSTATuPv59pBg8e7CkeCvy9Fqxdu9Y13rt3b0vVIJT52/rK3/fVwjB37lzjWn62Ezt9+rRxzfT9ZM6cOcaczMxMzzWEO654AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeAMAAAAAYJHPMd3699cP9Pls1xKR5s2b5xq///77jTmmOyS2b9/emLN582ZPdeFnefzyd0VPFJ67777bNb5o0SJjjunum7fccosx54svvvBWWASiJwomISHBuNanTx/XuL+74pru3J+WluaprotKly7tGn/77beNOV27dnWNz58/35jTr18/b4WFuPz2RbB7onjx4sa16667zjXu72shKsp9M5waNWoYc/Kzi0soM30tjB071piTlJRkqZrg4HUitCQmJrrG/X3dmXrZn759+xrXFi5c6Pl4kSYvfRFZ3w0BAAAAAAgxDN4AAAAAAFjE4A0AAAAAgEUM3gAAAAAAWMTgDQAAAACARQzeAAAAAABYxHZiAVC9enXjWkpKimvc3/YaO3fudI1fc8013grDb2JLjPDw17/+1TXev39/Y45pawt/22GAniiqevfubVxbsGCBa/yHH34w5jRp0sQ1fvz4cU91hYpw3U6ssNx2223GtejoaNe4v+23mjdvXtCSCt2yZcuMa927dy/ESuzjdaLwDRgwwLg2ZcoU13jZsmXzda6vvvrKNd6sWTNjzk8//ZSvc0USthMDAAAAACDIGLwBAAAAALCIwRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAi6KCXUAkuOmmm4xr/u5ebvL+++8XoBog8tx5552u8fT0dGPOn/70J1vlABHnnXfeMa517drVNX7vvfcac4YMGeIaHz9+vLfCEBZWr17tOcd053vJfFfzCxcuGHPefPNN1/hf/vIXY87w4cNd43369DHmADa1aNHCNe7vZ5r83L08LS3NuPbII4+4xrlzecFxxRsAAAAAAIsYvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCI7cQCIDY21nPO0aNHjWuvvPJKQcoBwpJp+wpJqly5smv8yJEjxpwvvviiwDUBRUVWVpZx7eWXX3aN33XXXcacMWPGuMb/9re/GXO++eYb4xoiz8qVK41rEyZMcI1HRZl/bB04cKBrvF69esacdu3aGde8OnDgQMCOhaKrS5curvFLLrnE87H8bblq2iZSkjZs2OD5XMgbrngDAAAAAGARgzcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEXc1D4Dbb7/dc86+ffuMa6dOnSpIOUBY8ndXc8dxXOP/+Mc/PJ/H351BK1So4Br3169ApNu2bZtr/IUXXjDmTJo0yTX+0ksvGXMeeOAB1/jZs2fNxSFsff3118a1d955xzV+zz33eD7PLbfc4jknMzPTuGZ63Xnuuec8nwdFk7+fQxITEwN2ngULFhjX1q5dG7DzIO+44g0AAAAAgEUM3gAAAAAAWMTgDQAAAACARQzeAAAAAABYxOANAAAAAIBFDN4AAAAAAFjEdmIeREdHu8br1q3r+Vjnzp0zrmVkZHg+HlAU+dvypW/fvq7xJ554wpjz1Vdfucb79evnrTCgCJg7d65xbdCgQa7xHj16GHPGjx/vGt+xY4e3whAW/G0TN3z4cNd42bJljTnNmjVzjVeqVMmYs3fvXtf4vHnzjDljx441rgG/ZPp63bVrlzHHNGv4Y/oeaeojBA9XvAEAAAAAsIjBGwAAAAAAixi8AQAAAACwiMEbAAAAAACLGLwBAAAAALCIu5p7kJWV5RrfvHmzMadRo0au8e+++y4gNQFF2YABA4xrv//9713js2fPNua8+OKLBa4JKCpSU1ONawkJCa5x012kJenZZ591jZt2KEDkOnz4sGu8S5cuxpwHHnjANX7jjTcac8aNG+caP3LkiJ/qgLy59dZbXePVq1c35jiO4/k8pt1a/O2ghODgijcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAWMXgDAAAAAGCRz8njfet9Pp/tWsJWtWrVjGtJSUmu8S1bthhzXnvttQLXhLzJz7YNF9ETgdWqVSvj2vjx413j69evN+bMmDHDNX7ixAljzvnz541rRQU9AZtWrlxpXGvZsqVr/IYbbjDm7Nq1q8A15UV++4KeQCTidSJvtm/f7hq/5pprPB9r0qRJxjXTVowoXHnpC654AwAAAABgEYM3AAAAAAAWMXgDAAAAAGARgzcAAAAAABYxeAMAAAAAYBF3NUeRxp05gZzoCdhUrlw545rpDsDDhg0z5ixbtqzANeUFdzUH/j9eJ/Jm//79rvHq1asbc44cOeIab9KkiTHn4MGDnuqCHdzVHAAAAACAIGPwBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwKKoYBcAAACKhv/973/Gtdq1axdiJQBg15QpUzzFJenFF190jbNlWGTgijcAAAAAABYxeAMAAAAAYBGDNwAAAAAAFjF4AwAAAABgEYM3AAAAAAAW+RzHcfL0QJ/Pdi1Aocvjl78regKRiJ4AcstvX9ATiES8TgC55aUvuOINAAAAAIBFDN4AAAAAAFjE4A0AAAAAgEUM3gAAAAAAWMTgDQAAAACARQzeAAAAAABYlOftxAAAAAAAgHdc8QYAAAAAwCIGbwAAAAAALGLwBgAAAADAIgZvAAAAAAAsYvAGAAAAAMAiBm8AAAAAACxi8AYAAAAAwCIGbwAAAAAALGLwBgAAAADAov8HdH/G6mBnCTMAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Visualize some predictions\n",
        "def plot_predictions(X, y_true, y_pred, num_images=10):\n",
        "    plt.figure(figsize=(10, 5))\n",
        "    for i in range(num_images):\n",
        "        plt.subplot(2, 5, i + 1)\n",
        "        plt.imshow(X[i].reshape(28, 28), cmap='gray')\n",
        "        plt.title(f'True: {y_true[i]}\\nPred: {y_pred[i]}')\n",
        "        plt.axis('off')\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "\n",
        "# Visualize predictions on test set\n",
        "plot_predictions(x_test, y_test, predictions)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Now Lets try the same with Regularization!"
      ],
      "metadata": {
        "id": "ZWoBe8DhO_v7"
      },
      "id": "ZWoBe8DhO_v7"
    },
    {
      "cell_type": "code",
      "source": [
        "class NeuralNetwork:\n",
        "    def __init__(self, input_size, hidden_size, output_size, learning_rate=0.01, l1_lambda=0.0, l2_lambda=0.0):\n",
        "        self.W1 = np.random.randn(input_size, hidden_size) * 0.01\n",
        "        self.b1 = np.zeros((1, hidden_size))\n",
        "        self.W2 = np.random.randn(hidden_size, output_size) * 0.01\n",
        "        self.b2 = np.zeros((1, output_size))\n",
        "        self.learning_rate = learning_rate\n",
        "        self.l1_lambda = l1_lambda\n",
        "        self.l2_lambda = l2_lambda\n",
        "\n",
        "    def softmax(self, x):\n",
        "        exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))\n",
        "        return exp_x / np.sum(exp_x, axis=1, keepdims=True)\n",
        "\n",
        "    def forward(self, X):\n",
        "        self.z1 = np.dot(X, self.W1) + self.b1\n",
        "        self.a1 = np.tanh(self.z1)\n",
        "        self.z2 = np.dot(self.a1, self.W2) + self.b2\n",
        "        self.a2 = self.softmax(self.z2)\n",
        "        return self.a2\n",
        "\n",
        "    def backward(self, X, y, output):\n",
        "        m = X.shape[0]\n",
        "        output_error = output - y\n",
        "        hidden_error = np.dot(output_error, self.W2.T) * (1 - np.tanh(self.z1) ** 2)\n",
        "\n",
        "        dW2 = np.dot(self.a1.T, output_error) / m\n",
        "        db2 = np.sum(output_error, axis=0, keepdims=True) / m\n",
        "        dW1 = np.dot(X.T, hidden_error) / m\n",
        "        db1 = np.sum(hidden_error, axis=0, keepdims=True) / m\n",
        "\n",
        "        # L1 regularization\n",
        "        dW1 += self.l1_lambda * np.sign(self.W1)\n",
        "        dW2 += self.l1_lambda * np.sign(self.W2)\n",
        "\n",
        "        # L2 regularization\n",
        "        dW1 += self.l2_lambda * self.W1\n",
        "        dW2 += self.l2_lambda * self.W2\n",
        "\n",
        "        self.W1 -= self.learning_rate * dW1\n",
        "        self.b1 -= self.learning_rate * db1\n",
        "        self.W2 -= self.learning_rate * dW2\n",
        "        self.b2 -= self.learning_rate * db2\n",
        "\n",
        "    def train(self, X, y, epochs):\n",
        "        for epoch in range(epochs):\n",
        "            output = self.forward(X)\n",
        "            self.backward(X, y, output)\n",
        "            if epoch % 100 == 0:\n",
        "                loss = -np.mean(np.sum(y * np.log(output + 1e-10), axis=1))\n",
        "                print(f\"Epoch {epoch}, Loss: {loss:.4f}\")\n",
        "\n",
        "    def predict(self, X):\n",
        "        output = self.forward(X)\n",
        "        return np.argmax(output, axis=1)"
      ],
      "metadata": {
        "id": "fppzhKqqOdi_"
      },
      "id": "fppzhKqqOdi_",
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Hyperparameters\n",
        "input_size = 784\n",
        "hidden_size = 128\n",
        "output_size = 10\n",
        "learning_rate = 0.01\n",
        "epochs = 100"
      ],
      "metadata": {
        "id": "R2iswBuaOqE1"
      },
      "id": "R2iswBuaOqE1",
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train without regularization\n",
        "nn_no_reg = NeuralNetwork(input_size, hidden_size, output_size, learning_rate)\n",
        "nn_no_reg.train(x_train, y_train_encoded, epochs)\n",
        "predictions_no_reg = nn_no_reg.predict(x_test)\n",
        "accuracy_no_reg = np.mean(predictions_no_reg == y_test)\n",
        "print(f\"Test Accuracy (No Regularization): {accuracy_no_reg:.4f}\")"
      ],
      "metadata": {
        "id": "8es-WERIOr5w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "532203c2-143f-4b48-c8a6-edbe6a48ffce"
      },
      "id": "8es-WERIOr5w",
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 0, Loss: 2.3023\n",
            "Test Accuracy (No Regularization): 0.5613\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Train with L1 regularization\n",
        "nn_l1 = NeuralNetwork(input_size, hidden_size, output_size, learning_rate, l1_lambda=0.001)\n",
        "nn_l1.train(x_train, y_train_encoded, epochs)\n",
        "predictions_l1 = nn_l1.predict(x_test)\n",
        "accuracy_l1 = np.mean(predictions_l1 == y_test)\n",
        "print(f\"Test Accuracy (L1 Regularization): {accuracy_l1:.4f}\")"
      ],
      "metadata": {
        "id": "uiOjIWbbOvby",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "195fa477-f281-46fc-d1e3-dfe3a71223e0"
      },
      "id": "uiOjIWbbOvby",
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 0, Loss: 2.3026\n",
            "Test Accuracy (L1 Regularization): 0.6081\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Train with L2 regularization\n",
        "nn_l2 = NeuralNetwork(input_size, hidden_size, output_size, learning_rate, l2_lambda=0.001)\n",
        "nn_l2.train(x_train, y_train_encoded, epochs)\n",
        "predictions_l2 = nn_l2.predict(x_test)\n",
        "accuracy_l2 = np.mean(predictions_l2 == y_test)\n",
        "print(f\"Test Accuracy (L2 Regularization): {accuracy_l2:.4f}\")"
      ],
      "metadata": {
        "id": "ts_fVxT2O7gR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "206c37ee-4fae-4821-a46d-6449e742ca17"
      },
      "id": "ts_fVxT2O7gR",
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 0, Loss: 2.3016\n",
            "Test Accuracy (L2 Regularization): 0.5711\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Q9euA-6eJyCh"
      },
      "id": "Q9euA-6eJyCh",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.6"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}