appendix-D/01_main-chapter-code/appendix-D.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a5936bd-af17-4a7e-a4d2-e910411708ea",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b",
   "metadata": {},
   "source": [
    "## Appendix D: Adding Bells and Whistles to the Training Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f58c142-9434-49af-b33a-356b80a45b86",
   "metadata": {},
   "source": [
    "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n",
    "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n",
    "- The final section adds these techniques to the training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723",
   "metadata": {},
   "source": [
    "- We start by initializing a model reusing the code from chapter 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch version: 2.2.2\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "import torch\n",
    "\n",
    "print(\"torch version:\", version(\"torch\"))\n",
    "\n",
    "\n",
    "from previous_chapters import GPTModel\n",
    "\n",
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,   # Vocabulary size\n",
    "    \"context_length\": 256, # Shortened context length (orig: 1024)\n",
    "    \"emb_dim\": 768,        # Embedding dimension\n",
    "    \"n_heads\": 12,         # Number of attention heads\n",
    "    \"n_layers\": 12,        # Number of layers\n",
    "    \"drop_rate\": 0.1,      # Dropout rate\n",
    "    \"qkv_bias\": False      # Query-key-value bias\n",
    "}\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.eval();  # Disable dropout during inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51574e57-a098-412c-83e8-66dafa5a0b99",
   "metadata": {},
   "source": [
    "- Next, using the same code we used in chapter 5, we initialize the data loaders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "386ca110-2bb4-42f1-bd54-8836df80acaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "\n",
    "file_path = \"the-verdict.txt\"\n",
    "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n",
    "\n",
    "if not os.path.exists(file_path):\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        text_data = response.read().decode('utf-8')\n",
    "    with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
    "        file.write(text_data)\n",
    "else:\n",
    "    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "        text_data = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae96992b-536a-4684-a924-658b9ffb7e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import create_dataloader_v1\n",
    "\n",
    "# Train/validation ratio\n",
    "train_ratio = 0.90\n",
    "split_idx = int(train_ratio * len(text_data))\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = create_dataloader_v1(\n",
    "    text_data[:split_idx],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=True,\n",
    "    shuffle=True,\n",
    "    num_workers=0\n",
    ")\n",
    "\n",
    "val_loader = create_dataloader_v1(\n",
    "    text_data[split_idx:],\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=False,\n",
    "    shuffle=False,\n",
    "    num_workers=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939c08d8-257a-41c6-b842-019f7897ac74",
   "metadata": {},
   "source": [
    "## D.1 Learning rate warmup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133",
   "metadata": {},
   "source": [
    "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n",
    "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n",
    "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 15\n",
    "initial_lr = 0.0001\n",
    "peak_lr = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3",
   "metadata": {},
   "source": [
    "- Typically, the number of warmup steps is between 0.1% to 10% of the total number of steps\n",
    "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5f6d083f-1b25-4c23-b46d-ef7783446690",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27\n"
     ]
    }
   ],
   "source": [
    "total_steps = len(train_loader) * n_epochs\n",
    "warmup_steps = int(0.2 * total_steps) # 20% warmup\n",
    "print(warmup_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e075f80e-a398-4809-be1d-8019e1d31c90",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "global_step = -1\n",
    "track_lrs = []\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        if global_step < warmup_steps:\n",
    "            lr = initial_lr + global_step * lr_increment\n",
    "        else:\n",
    "            lr = peak_lr\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.defaults[\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights\n",
    "        # ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6LElEQVR4nO3dfViUVd4H8O8MLzOIMCgkA4oyGkUKokIQxq5rUmSokZbm8qiZq9VSafRk+QbVYw+FuevaWupuT7bX5su6W6Su0hKWZiIoIIrvJYqJAwgxgyhvM+f5g7i3WVEZHLhn4Pu5rrlw7vt3z/yO0vw65z5zjkIIIUBERER2Ryl3AkRERNQ2FmkiIiI7xSJNRERkp1ikiYiI7BSLNBERkZ1ikSYiIrJTLNJERER2ikWaiIjITjnLnYCjMpvNKCsrg4eHBxQKhdzpEBGRzIQQqK2thb+/P5RK2/SBWaQ7qKysDAEBAXKnQUREdubChQsYMGCATV6LRbqDPDw8ALT8Y3h6esqcDRERyc1oNCIgIECqD7bAIt1BrUPcnp6eLNJERCSx5S1QThwjIiKyUyzSREREdopFmoiIyE7JXqTXrFmDwMBAqNVqREVFIS8v76bxW7duRXBwMNRqNUJDQ7Fz506L859++ikeeugheHt7Q6FQ4PDhw9e9Rn19PZKSkuDt7Y3evXtjypQpKC8vt2WziIiIbpusRXrLli1ITk5GamoqCgoKEBYWhri4OFRUVLQZv3//fkyfPh1z5sxBYWEhEhISkJCQgOLiYimmrq4OMTExeOedd274vi+99BK2b9+OrVu3Ys+ePSgrK8PkyZNt3j4iIqLboRBCCLnePCoqCvfeey/++Mc/AmhZICQgIAAvvPACXnvttevip02bhrq6OuzYsUM6dt9992HEiBFYu3atRey5c+eg0+lQWFiIESNGSMcNBgPuuOMObNy4EY8//jgA4OTJk7jnnnuQk5OD++67r125G41GaDQaGAwGzu4mIqJOqQuyfQWrsbER+fn5WLRokXRMqVQiNjYWOTk5bV6Tk5OD5ORki2NxcXHIyMho9/vm5+ejqakJsbGx0rHg4GAMHDjwpkW6oaEBDQ0N0nOj0dju9yTrmc0Cb+08gZN6/j0TUedzVirx8dORcqdxHdmK9OXLl2EymeDr62tx3NfXFydPnmzzGr1e32a8Xq9v9/vq9Xq4urrCy8vLqtdJS0vDG2+80e73odtz4GwVPtxXIncaRNRDuDrJPkWrTVzMpJ0WLVpk0YtvXVmGOse2ojIAwAPB/fDoCH+ZsyGi7k5pp3swyFakfXx84OTkdN2s6vLycmi12jav0Wq1VsXf6DUaGxtRU1Nj0Zu+1euoVCqoVKp2vw91XGOzGbuKW0Y1fhOjw+g7fWTOiIhIHrL1711dXREeHo7s7GzpmNlsRnZ2NqKjo9u8Jjo62iIeALKysm4Y35bw8HC4uLhYvM6pU6dQWlpq1etQ5/nmTCUM15rQz0OFqMHecqdDRCQbWYe7k5OTMWvWLERERCAyMhKrVq1CXV0dZs+eDQCYOXMm+vfvj7S0NADA/PnzMWbMGKxcuRLx8fHYvHkzDh06hPXr10uvWV1djdLSUpSVtQyXnjp1CkBLD1qr1UKj0WDOnDlITk5G37594enpiRdeeAHR0dHtntlNnat1qDt+uB+clPY5BEVE1BVkLdLTpk1DZWUlUlJSoNfrMWLECGRmZkqTw0pLSy325Bw9ejQ2btyIpUuXYvHixQgKCkJGRgZCQkKkmG3btklFHgCefPJJAEBqaipef/11AMDvf/97KJVKTJkyBQ0NDYiLi8P777/fBS2mW7nWaELW8ZZbGhPDeC+aiHo2Wb8n7cj4PenOsb2oDC9sKkRAXzfsfWWsTXeTISLqTJ1RF+xzzjn1WNt/GuqeONyfBZqIejwWabIbhmtN+PpUJQBgEr92RUTEIk3244tjejSazAjq1xt3+3rInQ4RkexYpMlutA51TwrjUDcREcAiTXbi8pUG7P++CgBndRMRtWKRJruw8+glmMwCwwdoEOjjLnc6RER2gUWa7MLPh7qJiKgFizTJ7mLNNRw89yMUCmDCcBZpIqJWLNIkux0/9aLvDewLrUYtczZERPaDRZpkt41D3UREbWKRJll9X3kFx8qMcFYq8Eion9zpEBHZFRZpklXrhLGYIB/0dXeVORsiIvvCIk2yEUJIQ90TOWGMiOg6LNIkm+OXjDhbWQeVsxIPDfOVOx0iIrvDIk2yae1FPxDcDx5qF5mzISKyPyzSJAuzWWBH0SUAXAaUiOhGWKRJFoUXfsTFmmvorXLGA8H95E6HiMgusUiTLLYdbhnqfmioL9QuTjJnQ0Rkn1ikqcs1m8z451EOdRMR3QqLNHW5A2ercflKI/r0ckFMkI/c6RAR2S0Waepy24ouAgDGh/rBxYm/gkREN8JPSOpSDc0m7CrWA+Ba3UREt8IiTV1qz6lK1NY3w9dThXsD+8qdDhGRXWORpi61/UjLhLEJw/3hpFTInA0RkX1jkaYuc7WxGV8eLwfAoW4iovZgkaYuk3W8HNeaTBjk3QvDB2jkToeIyO6xSFOX2d66DOhwfygUHOomIroVFmnqEoarTdhzugIAMGkEh7qJiNqDRZq6ROaxS2gyCdzt64G7fD3kToeIyCGwSFOXaB3qZi+aiKj9ZC/Sa9asQWBgINRqNaKiopCXl3fT+K1btyI4OBhqtRqhoaHYuXOnxXkhBFJSUuDn5wc3NzfExsbizJkzFjGnT5/Go48+Ch8fH3h6eiImJgZfffWVzdtGLSpq67H/+8sAWu5HExFR+8hapLds2YLk5GSkpqaioKAAYWFhiIuLQ0VFRZvx+/fvx/Tp0zFnzhwUFhYiISEBCQkJKC4ulmLS09OxevVqrF27Frm5uXB3d0dcXBzq6+ulmAkTJqC5uRm7d+9Gfn4+wsLCMGHCBOj1+k5vc0+088glmAUwIsALA717yZ0OEZHjEDKKjIwUSUlJ0nOTyST8/f1FWlpam/FTp04V8fHxFseioqLEM888I4QQwmw2C61WK1asWCGdr6mpESqVSmzatEkIIURlZaUAIPbu3SvFGI1GAUBkZWW1O3eDwSAACIPB0O5reqrH1uwTg17dIf78zVm5UyEi6jSdURdk60k3NjYiPz8fsbGx0jGlUonY2Fjk5OS0eU1OTo5FPADExcVJ8SUlJdDr9RYxGo0GUVFRUoy3tzfuvvtu/OUvf0FdXR2am5uxbt069OvXD+Hh4bZuZo93ofoqCkproFAAE4b7yZ0OEZFDcZbrjS9fvgyTyQRfX1+L476+vjh58mSb1+j1+jbjW4epW3/eLEahUODLL79EQkICPDw8oFQq0a9fP2RmZqJPnz43zLehoQENDQ3Sc6PR2M6W9mw7floG9D6dN3w91TJnQ0TkWGSfONbVhBBISkpCv3798M033yAvLw8JCQmYOHEiLl26dMPr0tLSoNFopEdAQEAXZu24thWVAQAmchlQIiKryVakfXx84OTkhPLycovj5eXl0Gq1bV6j1WpvGt/682Yxu3fvxo4dO7B582bcf//9GDVqFN5//324ubnh448/vmG+ixYtgsFgkB4XLlywrsE90HcVtThxyQhnpQLjQ9r+NyUiohuTrUi7uroiPDwc2dnZ0jGz2Yzs7GxER0e3eU10dLRFPABkZWVJ8TqdDlqt1iLGaDQiNzdXirl69SqAlvvfP6dUKmE2m2+Yr0qlgqenp8WDbm7bT9+N/uVdd6CPu6vM2RAROR7Z7kkDQHJyMmbNmoWIiAhERkZi1apVqKurw+zZswEAM2fORP/+/ZGWlgYAmD9/PsaMGYOVK1ciPj4emzdvxqFDh7B+/XoALfebFyxYgOXLlyMoKAg6nQ7Lli2Dv78/EhISALQU+j59+mDWrFlISUmBm5sb/vSnP6GkpATx8fGy/D10R0IIbJeGujlhjIioI2Qt0tOmTUNlZSVSUlKg1+sxYsQIZGZmShO/SktLLXq8o0ePxsaNG7F06VIsXrwYQUFByMjIQEhIiBSzcOFC1NXVYd68eaipqUFMTAwyMzOhVrdMWvLx8UFmZiaWLFmCBx54AE1NTRg2bBg+//xzhIWFde1fQDd2rMyIkst1UDkr8eBQDnUTEXWEQggh5E7CERmNRmg0GhgMBg59t+F/d57A+r1nER/qhzWJo+ROh4io03VGXehxs7up85nNPx/q5qxuIqKOYpEmmzt0/kdcMtTDQ+WMX919h9zpEBE5LBZpsrnWXvRDw7RQuzjJnA0RkeNikSabajaZsfMot6UkIrIFFmmyqW+/r0JVXSP6urti9BBvudMhInJoLNJkU61D3Y+EauHixF8vIqLbwU9Rspn6JhO+KG7ZyGRSWH+ZsyEicnws0mQzX5+qRG1DM/w0akQMuvGOYkRE1D4s0mQz24+0DHVPGO4HpVIhczZERI6PRZpsoq6hGdknWnYf41A3EZFtdKhINzc348svv8S6detQW1sLACgrK8OVK1dsmhw5jqzj5ahvMkPn446Q/lwmlYjIFqzeYOP8+fN4+OGHUVpaioaGBjz44IPw8PDAO++8g4aGBqxdu7Yz8iQ7t611GdDhflAoONRNRGQLVvek58+fj4iICPz4449wc3OTjj/22GPX7fVMPUPN1UbsPV0JgAuYEBHZktU96W+++Qb79++Hq6urxfHAwEBcvHjRZomR49hVrEezWeAeP0/c2c9D7nSIiLoNq3vSZrMZJpPpuuM//PADPDz4Ad0TbTvcuuOVn8yZEBF1L1YX6YceegirVq2SnisUCly5cgWpqal45JFHbJkbOYAKYz0OlFQBACYO51A3EZEtWT3cvXLlSsTFxWHo0KGor6/Hr3/9a5w5cwY+Pj7YtGlTZ+RIdmzHkUsQAhg10AsBfXvJnQ4RUbdidZEeMGAAioqKsGXLFhQVFeHKlSuYM2cOEhMTLSaSUc8gzeoOYy+aiMjWrC7Se/fuxejRo5GYmIjExETpeHNzM/bu3Ytf/vKXNk2Q7NeF6qs4fKEGSgUQP5z3o4mIbM3qe9Jjx45FdXX1dccNBgPGjh1rk6TIMbT2oqOHeKOfh1rmbIiIuh+ri7QQos3FKqqqquDu7m6TpMgxbJcWMOFQNxFRZ2j3cPfkyZMBtMzmfuqpp6BSqaRzJpMJR44cwejRo22fIdml0+W1OKmvhYuTAuNDONRNRNQZ2l2kNRoNgJaetIeHh8UkMVdXV9x3332YO3eu7TMku9Taix5z1x3Q9HKRORsiou6p3UX6o48+AtCysth///d/c2i7BxNCcFY3EVEXsHp2d2pqamfkQQ7kyA8GnK+6CrWLErH3+MqdDhFRt2V1kQaAv//97/jb3/6G0tJSNDY2WpwrKCiwSWJkv1qHumPv8YW7qkO/QkRE1A5Wz+5evXo1Zs+eDV9fXxQWFiIyMhLe3t44e/Ysxo8f3xk5kh0xmwV2HLkEAJjEoW4iok5ldZF+//33sX79erz33ntwdXXFwoULkZWVhRdffBEGg6EzciQ7kneuGnpjPTzUzhhz9x1yp0NE1K1ZXaRLS0ulr1q5ubmhtrYWADBjxgyu3d0DtA51PzxMC5Wzk8zZEBF1b1YXaa1WK604NnDgQBw4cAAAUFJSAiGEbbMju9JkMmPn0Z+GukdwqJuIqLNZXaQfeOABbNu2DQAwe/ZsvPTSS3jwwQcxbdo0PPbYYzZPkOzHvu8u48erTfDp7Yrowd5yp0NE1O1ZXaTXr1+PJUuWAACSkpLwf//3f7jnnnvw5ptv4oMPPrA6gTVr1iAwMBBqtRpRUVHIy8u7afzWrVsRHBwMtVqN0NBQ7Ny50+K8EAIpKSnw8/ODm5sbYmNjcebMmete55///CeioqLg5uaGPn36ICEhwerce5rth1uGuh8J9YOzk9W/OkREZCWrPmmbm5uxfPly6PV66diTTz6J1atX44UXXoCrq6tVb75lyxYkJycjNTUVBQUFCAsLQ1xcHCoqKtqM379/P6ZPn445c+agsLAQCQkJSEhIQHFxsRSTnp6O1atXY+3atcjNzYW7uzvi4uJQX18vxfzjH//AjBkzMHv2bBQVFeHbb7/Fr3/9a6ty72nqm0z41/FyAJzVTUTUZYSV3N3dRUlJibWXtSkyMlIkJSVJz00mk/D39xdpaWltxk+dOlXEx8dbHIuKihLPPPOMEEIIs9kstFqtWLFihXS+pqZGqFQqsWnTJiGEEE1NTaJ///7iz3/+823lbjAYBABhMBhu63Ucxc4jZWLQqzvE6LRsYTKZ5U6HiMjudEZdsHrMcty4cdizZ89t/89BY2Mj8vPzERsbKx1TKpWIjY1FTk5Om9fk5ORYxANAXFycFF9SUgK9Xm8Ro9FoEBUVJcUUFBTg4sWLUCqVGDlyJPz8/DB+/HiL3jhdr3UZ0AnD/aBUXr8LGhER2Z7Vy0WNHz8er732Go4ePYrw8PDr1vCeNGlSu17n8uXLMJlM8PW1XFbS19cXJ0+ebPMavV7fZnzr8Hvrz5vFnD17FgDw+uuv43e/+x0CAwOxcuVK/OpXv8Lp06fRt2/fNt+7oaEBDQ0N0nOj0diudnYHtfVN2H2y5RYE1+omIuo6Vhfp3/72twCA3/3ud9edUygUMJlMt59VJzKbzQCAJUuWYMqUKQBaNg8ZMGAAtm7dimeeeabN69LS0vDGG290WZ72JOt4ORqazRh8hzuG+XvKnQ4RUY9h9XC32Wy+4cOaAu3j4wMnJyeUl5dbHC8vL4dWq23zGq1We9P41p83i/Hza9n7eOjQodJ5lUqFwYMHo7S09Ib5Llq0CAaDQXpcuHChPc3sFqQdr4b7Q6HgUDcRUVeR7Xs0rq6uCA8PR3Z2tnTMbDYjOzsb0dHRbV4THR1tEQ8AWVlZUrxOp4NWq7WIMRqNyM3NlWLCw8OhUqlw6tQpKaapqQnnzp3DoEGDbpivSqWCp6enxaMnqK5rxL4zlwFwARMioq4m6xZGycnJmDVrFiIiIhAZGYlVq1ahrq4Os2fPBgDMnDkT/fv3R1paGgBg/vz5GDNmDFauXIn4+Hhs3rwZhw4dwvr16wG0DLcvWLAAy5cvR1BQEHQ6HZYtWwZ/f3/pe9Cenp549tlnkZqaioCAAAwaNAgrVqwAADzxxBNd/5dg53YVX0KzWWCYvyeG3NFb7nSIiHoUWYv0tGnTUFlZiZSUFOj1eowYMQKZmZnSxK/S0lIolf/u7I8ePRobN27E0qVLsXjxYgQFBSEjIwMhISFSzMKFC1FXV4d58+ahpqYGMTExyMzMhFqtlmJWrFgBZ2dnzJgxA9euXUNUVBR2796NPn36dF3jHcS2nxYw4YQxIqKupxCCC253hNFohEajgcFg6LZD33pDPaLfzoYQwLevPYD+Xm5yp0REZLc6oy5wbUe6oR1HyiAEEDGoDws0EZEMrB7uvtH3gxUKBVQqldVLg5L9at2WkhPGiIjkYXWR9vLyuunXcAYMGICnnnoKqampFveTybGcu1yHoh8MUCqA8SF+cqdDRNQjWV2kN2zYgCVLluCpp55CZGQkACAvLw8ff/wxli5disrKSrz77rtQqVRYvHixzROmrrHjSEsv+v47fXCHh0rmbIiIeiari/THH3+MlStXYurUqdKxiRMnIjQ0FOvWrUN2djYGDhyIt956i0XagUkLmHBWNxGRbKwej96/fz9Gjhx53fGRI0dKm1jExMTcdPUusm8n9UacLr8CVycl4oa1vfobERF1PquLdEBAAD788MPrjn/44YcICAgAAFRVVfE7xw6sdcLYmLvvgMbNReZsiIh6LquHu99991088cQT2LVrF+69914AwKFDh3Dy5En8/e9/BwAcPHgQ06ZNs22m1CWEENhedAkAMIlD3UREsrK6SE+aNAknT57EunXrcPr0aQAt21dmZGQgMDAQAPDcc8/ZNEnqOocv1KC0+ircXJww7p5+cqdDRNSjdWhZUJ1Oh7ffftvWuZAdaJ0w9uBQX/RylXXVWCKiHq9Dn8I1NTXIy8tDRUWFtD9zq5kzZ9okMep6JrPAP49wqJuIyF5YXaS3b9+OxMREXLlyBZ6enhYLmygUChZpB5ZbUoWK2gZ4qp3xy7vukDsdIqIez+rZ3S+//DKefvppXLlyBTU1Nfjxxx+lR3V1dWfkSF2kdVb3+BA/uDpztTgiIrlZ/Ul88eJFvPjii+jVq1dn5EMyaWw2Y1exHgDX6iYishdWF+m4uDgcOnSoM3IhGe37rhI1V5vg01uF+wZ7y50OERGhA/ek4+Pj8corr+D48eMIDQ2Fi4vlYheTJk2yWXLUdbYdbhnqnjDcD07KG2+gQkREXcfqIj137lwAwJtvvnndOYVCAZPJdPtZUZe61mhC1vFyAFyrm4jInlhdpP/zK1fk+HafrEBdown9vdwwaqCX3OkQEdFPOIWXsK3oIoCWXvTN9gonIqKu1a6e9OrVqzFv3jyo1WqsXr36prEvvviiTRKjrmGsb8JXpyoBcAETIiJ7oxBCiFsF6XQ6HDp0CN7e3tDpdDd+MYUCZ8+etWmC9spoNEKj0cBgMMDT01PudDrs7/k/4L+3FuHOfr2R9dIv2ZMmIuqgzqgL7epJl5SUtPlncnyta3VP4lA3EZHd4T3pHqzqSgO+/e4yAM7qJiKyR1bP7jaZTNiwYQOys7Pb3GBj9+7dNkuOOtfOYj1MZoHQ/hrofNzlToeIiP6D1UV6/vz52LBhA+Lj4xESEsIhUge2/fC/h7qJiMj+WF2kN2/ejL/97W945JFHOiMf6iJlNdeQd65lQ5T44X4yZ0NERG2x+p60q6sr7rzzzs7IhbpQ677RkYF94e/lJnM2RETUlg5tVfmHP/wB7fjmFtmx1lndE7njFRGR3bJ6uHvfvn346quvsGvXLgwbNuy6DTY+/fRTmyVHnaPkch2OXjTASanAIyFaudMhIqIbsLpIe3l54bHHHuuMXKiLbP+pF33/nT7w7q2SORsiIroRq4p0c3Mzxo4di4ceeghaLXtgjkgIYbGACRER2S+r7kk7Ozvj2WefRUNDg02TWLNmDQIDA6FWqxEVFYW8vLybxm/duhXBwcFQq9UIDQ3Fzp07Lc4LIZCSkgI/Pz+4ubkhNjYWZ86cafO1GhoaMGLECCgUChw+fNhWTbJbJy7V4ruKK3B1VuKhYb5yp0NERDdh9cSxyMhIFBYW2iyBLVu2IDk5GampqSgoKEBYWBji4uJQUVHRZvz+/fsxffp0zJkzB4WFhUhISEBCQgKKi4ulmPT0dKxevRpr165Fbm4u3N3dERcXh/r6+uteb+HChfD37zk9ytZe9Ni774Cn2uUW0UREJCthpS1btojBgweL9957T+zfv18UFRVZPKwVGRkpkpKSpOcmk0n4+/uLtLS0NuOnTp0q4uPjLY5FRUWJZ555RgghhNlsFlqtVqxYsUI6X1NTI1Qqldi0aZPFdTt37hTBwcHi2LFjAoAoLCxsd94Gg0EAEAaDod3XyM1sNovRadli0Ks7xI6iMrnTISLqVjqjLlg9cezJJ58EYLklpUKhgBACCoUCJpOp3a/V2NiI/Px8LFq0SDqmVCoRGxuLnJycNq/JyclBcnKyxbG4uDhkZGQAaNkARK/XIzY2Vjqv0WgQFRWFnJwcKf/y8nLMnTsXGRkZ6NWr1y1zbWhosBjmNxqN7W6nvSgorcHFmmtwd3XCA8H95E6HiIhuweoibctdsC5fvgyTyQRfX8t7o76+vjh58mSb1+j1+jbj9Xq9dL712I1ihBB46qmn8OyzzyIiIgLnzp27Za5paWl444032tUue9U6q/vBob5wc3WSORsiIroVq4v0oEGDOiOPLvXee++htrbWogd/K4sWLbLowRuNRgQEBHRGep3CZBbY8dMqY5O4gAkRkUOwuki3On78OEpLS9HY2GhxfNKkSe1+DR8fHzg5OaG8vNzieHl5+Q2/4qXVam8a3/qzvLwcfn5+FjEjRowA0LJTV05ODlQqy+8IR0REIDExER9//PF176tSqa6LdyQHzlbh8pUGePVyQcydd8idDhERtYPVRfrs2bN47LHHcPToUeleNABpNyxr7km7uroiPDwc2dnZSEhIAACYzWZkZ2fj+eefb/Oa6OhoZGdnY8GCBdKxrKwsREdHAwB0Oh20Wi2ys7Olomw0GpGbm4vnnnsOALB69WosX75cur6srAxxcXHYsmULoqKi2p2/I9n2045X40O0cHXmNuJERI6gQ1tV6nQ6ZGdnQ6fTIS8vD1VVVXj55Zfx7rvvWp1AcnIyZs2ahYiICERGRmLVqlWoq6vD7NmzAQAzZ85E//79kZaWJr3/mDFjsHLlSsTHx2Pz5s04dOgQ1q9fD6DlfxYWLFiA5cuXIygoCDqdDsuWLYO/v7/0PwIDBw60yKF3794AgCFDhmDAgAFWt8HeNTabsau4Zah7IhcwISJyGFYX6ZycHOzevRs+Pj5QKpVQKpWIiYlBWloaXnzxRau/Qz1t2jRUVlYiJSUFer0eI0aMQGZmpjTxq7S0FErlv3t+o0ePxsaNG7F06VIsXrwYQUFByMjIQEhIiBSzcOFC1NXVYd68eaipqUFMTAwyMzOhVqutbW63sPd0JYz1zejnoUKUzlvudIiIqJ0UQli3nVWfPn1QUFAAnU6HIUOG4M9//jPGjh2L77//HqGhobh69Wpn5WpXjEYjNBoNDAYDPD095U7npl7cVIhtRWV4+n4dUiYOlTsdIqJuqTPqgtU96ZCQEBQVFUGn0yEqKgrp6elwdXXF+vXrMXjwYJskRbZztbEZWcdbJtpNDPO7RTQREdkTq4v00qVLUVdXBwB48803MWHCBPziF7+At7c3tmzZYvME6fZkn6jAtSYTBvbthREBXnKnQ0REVrC6SMfFxUl/vvPOO3Hy5ElUV1ejT58+0gxvsh+ta3VPDPPjvw8RkYPp8HdxvvvuO3zxxRe4du0a+vbta8ucyEYM15qw51QlAM7qJiJyRFYX6aqqKowbNw533XUXHnnkEVy61PLVnjlz5uDll1+2eYLUcV8c06PRZMZdvr0RrLXvyW1ERHQ9q4v0Sy+9BBcXF5SWllpsTDFt2jRkZmbaNDm6Pa1rdU9iL5qIyCFZfU/6X//6F7744ovrFv0ICgrC+fPnbZYY3Z7K2gZ8+91lAMCE4SzSRESOyOqedF1dXZtbO1ZXVzv02tbdza7iSzALIGyABoE+7nKnQ0REHWB1kf7FL36Bv/zlL9JzhUIBs9mM9PR0jB071qbJUce1rtXNCWNERI7L6uHu9PR0jBs3DocOHUJjYyMWLlyIY8eOobq6Gt9++21n5EhWulhzDYfO/wiFgkPdRESOzOqedEhICE6fPo2YmBg8+uijqKurw+TJk1FYWIghQ4Z0Ro5kpdYJY5GBfaHV9Mz1yomIuoMO7Set0WiwZMkSi2M//PAD5s2bJ+1GRfKRZnWPYC+aiMiR2Wxj4aqqKnz44Ye2ejnqoO8rr+BYmRHOSgXGh3CtbiIiR2azIk32oXXCWEyQD/q6u8qcDRER3Q4W6W5ECIHtR7iACRFRd8Ei3Y0cKzPibGUdVM5KPDjUV+50iIjoNrV74tjkyZNver6mpuZ2c6Hb1Dph7IHgfvBQu8icDRER3a52F2mNRnPL8zNnzrzthKhjzGaBHUdaNjvhUDcRUffQ7iL90UcfdWYedJsKSn/ExZpr6K1yxtjgfnKnQ0RENsB70t3Etp+Guh8a5gu1i5PM2RARkS2wSHcDzSYzdh5tGermWt1ERN0Hi3Q3kHO2CpevNKJPLxfE3OkjdzpERGQjLNLdQOsCJo+E+sHFif+kRETdBT/RHVxDswmZx/QAONRNRNTdsEg7uD2nKlFb3wytpxqRgX3lToeIiGyIRdrBtc7qnjDcD0qlQuZsiIjIllikHVhdQzO+PFEOgEPdRETdEYu0A/vyRDnqm8wY5N0LwwfcfEU4IiJyPCzSDqx1re5JYf5QKDjUTUTU3bBIO6iaq43Yc7oSANfqJiLqruyiSK9ZswaBgYFQq9WIiopCXl7eTeO3bt2K4OBgqNVqhIaGYufOnRbnhRBISUmBn58f3NzcEBsbizNnzkjnz507hzlz5kCn08HNzQ1DhgxBamoqGhsbO6V9nSGzWI8mk0Cw1gNBvh5yp0NERJ1A9iK9ZcsWJCcnIzU1FQUFBQgLC0NcXBwqKirajN+/fz+mT5+OOXPmoLCwEAkJCUhISEBxcbEUk56ejtWrV2Pt2rXIzc2Fu7s74uLiUF9fDwA4efIkzGYz1q1bh2PHjuH3v/891q5di8WLF3dJm21h+5GWoW5OGCMi6saEzCIjI0VSUpL03GQyCX9/f5GWltZm/NSpU0V8fLzFsaioKPHMM88IIYQwm81Cq9WKFStWSOdramqESqUSmzZtumEe6enpQqfTtTtvg8EgAAiDwdDua2yl3HhN6F7bIQa9ukOUVtV1+fsTEdH1OqMuyNqTbmxsRH5+PmJjY6VjSqUSsbGxyMnJafOanJwci3gAiIuLk+JLSkqg1+stYjQaDaKiom74mgBgMBjQt69jLAbyzyOXYBbAiAAvBPTtJXc6RETUSdq9n3RnuHz5MkwmE3x9fS2O+/r64uTJk21eo9fr24zX6/XS+dZjN4r5T9999x3ee+89vPvuuzfMtaGhAQ0NDdJzo9F4w9jO9vNZ3URE1H3Jfk9abhcvXsTDDz+MJ554AnPnzr1hXFpaGjQajfQICAjowiz/7UL1VRSU1kChaFlljIiIui9Zi7SPjw+cnJxQXl5ucby8vBxarbbNa7Ra7U3jW3+25zXLysowduxYjB49GuvXr79prosWLYLBYJAeFy5cuHUDO0HrhLH7dN7o56mWJQciIuoashZpV1dXhIeHIzs7WzpmNpuRnZ2N6OjoNq+Jjo62iAeArKwsKV6n00Gr1VrEGI1G5ObmWrzmxYsX8atf/Qrh4eH46KOPoFTe/K9CpVLB09PT4iGH7UWXAACTRnCom4iou5P1njQAJCcnY9asWYiIiEBkZCRWrVqFuro6zJ49GwAwc+ZM9O/fH2lpaQCA+fPnY8yYMVi5ciXi4+OxefNmHDp0SOoJKxQKLFiwAMuXL0dQUBB0Oh2WLVsGf39/JCQkAPh3gR40aBDeffddVFZWSvncqAdvD76rqMWJS0a4OCkwPsR+8yQiItuQvUhPmzYNlZWVSElJgV6vx4gRI5CZmSlN/CotLbXo5Y4ePRobN27E0qVLsXjxYgQFBSEjIwMhISFSzMKFC1FXV4d58+ahpqYGMTExyMzMhFrdMjyclZWF7777Dt999x0GDBhgkY8Qogta3THbDrcMdf8y6A549XKVORsiIupsCmHPVcmOGY1GaDQaGAyGLhn6FkJg7Ltf41zVVayaNgIJI/t3+nsSEVH7dUZd6PGzux1F8UUjzlVdhdpFiQeH+t76AiIicngs0g5iW9FFAMC4e3zhrpL9LgUREXUBFmkHYDYL7DjSMqt74nDO6iYi6ilYpB3AofM/4pKhHh4qZ/zq7jvkToeIiLoIi7QDaB3qjgvRQu3iJHM2RETUVVik7VyTyYydR1vWHOe2lEREPQuLtJ3b/30Vqusa4e3uivuHeMudDhERdSEWaTvXuoDJI6F+cHbiPxcRUU/CT307Vt9kwr+OcaibiKinYpG2Y1+fqkBtQzP8NGpEDOojdzpERNTFWKTtWOuOVxPD/KFUKmTOhoiIuhqLtJ260tCML0+07Ik9iUPdREQ9Eou0nco6rkdDsxk6H3cM85dn72oiIpIXi7Sd+vlQt0LBoW4iop6IRdoO/VjXiL2nKwEAk8L8ZM6GiIjkwiJth3YV69FsFrjHzxN39vOQOx0iIpIJi7Qd2l7UsoAJJ4wREfVsLNJ2ptxYjwMlVQCACcM51E1E1JOxSNuZHUcuQQhg1EAvBPTtJXc6REQkIxZpO7ONQ91ERPQTFmk7Ulp1FUUXaqBUAPHDWaSJiHo6Fmk7sv1ISy969BAf3OGhkjkbIiKSG4u0HWndlnIivxtNRERgkbYbp/S1OFVeCxcnBR4exiJNREQs0naj9bvRY+7qB00vF5mzISIie8AibQeEENKsbg51ExFRKxZpO3DkBwNKq6/CzcUJDw71lTsdIiKyEyzSdqC1Fx071Be9XJ1lzoaIiOwFi7TMTGaBHT999WoilwElIqKfYZGWWV5JNcqNDfBUO2PM3XfInQ4REdkRFmmZtS5g8nCIFipnJ5mzISIie2IXRXrNmjUIDAyEWq1GVFQU8vLybhq/detWBAcHQ61WIzQ0FDt37rQ4L4RASkoK/Pz84ObmhtjYWJw5c8Yiprq6GomJifD09ISXlxfmzJmDK1eu2LxtN9NkMmPX0UsAgElh/bv0vYmIyP7JXqS3bNmC5ORkpKamoqCgAGFhYYiLi0NFRUWb8fv378f06dMxZ84cFBYWIiEhAQkJCSguLpZi0tPTsXr1aqxduxa5ublwd3dHXFwc6uvrpZjExEQcO3YMWVlZ2LFjB/bu3Yt58+Z1ent/bt+Zy/jxahN8ervivsF9u/S9iYjIAQiZRUZGiqSkJOm5yWQS/v7+Ii0trc34qVOnivj4eItjUVFR4plnnhFCCGE2m4VWqxUrVqyQztfU1AiVSiU2bdokhBDi+PHjAoA4ePCgFLNr1y6hUCjExYsX25W3wWAQAITBYGhfQ9vw0uZCMejVHSIl42iHX4OIiOyDLerCf5K1J93Y2Ij8/HzExsZKx5RKJWJjY5GTk9PmNTk5ORbxABAXFyfFl5SUQK/XW8RoNBpERUVJMTk5OfDy8kJERIQUExsbC6VSidzc3Dbft6GhAUaj0eJxu5791RAkjR2Cx8MDbvu1iIio+5G1SF++fBkmkwm+vpYLePj6+kKv17d5jV6vv2l8689bxfTr18/ivLOzM/r27XvD901LS4NGo5EeAQG3X1jv8vXAK3HBCB2gue3XIiKi7kf2e9KOYtGiRTAYDNLjwoULcqdERETdnKxF2sfHB05OTigvL7c4Xl5eDq1W2+Y1Wq32pvGtP28V858T05qbm1FdXX3D91WpVPD09LR4EBERdSZZi7SrqyvCw8ORnZ0tHTObzcjOzkZ0dHSb10RHR1vEA0BWVpYUr9PpoNVqLWKMRiNyc3OlmOjoaNTU1CA/P1+K2b17N8xmM6KiomzWPiIiottisyloHbR582ahUqnEhg0bxPHjx8W8efOEl5eX0Ov1QgghZsyYIV577TUp/ttvvxXOzs7i3XffFSdOnBCpqanCxcVFHD367xnSb7/9tvDy8hKff/65OHLkiHj00UeFTqcT165dk2IefvhhMXLkSJGbmyv27dsngoKCxPTp09udd2fM4iMiIsfVGXVB9t0cpk2bhsrKSqSkpECv12PEiBHIzMyUJn6VlpZCqfx3h3/06NHYuHEjli5disWLFyMoKAgZGRkICQmRYhYuXIi6ujrMmzcPNTU1iImJQWZmJtRqtRTzySef4Pnnn8e4ceOgVCoxZcoUrF69uusaTkREdAsKIYSQOwlHZDQaodFoYDAYeH+aiIg6pS5wdjcREZGdkn2421G1DkDYYlETIiJyfK31wJYD1CzSHVRbWwsANlnUhIiIuo/a2lpoNLZZpIr3pDvIbDajrKwMHh4eUCgUHXoNo9GIgIAAXLhwodvc12abHAPb5BjYJsfQ2qbS0lIoFAr4+/tbTHi+HexJd5BSqcSAAQNs8lrdcXEUtskxsE2OgW1yDBqNxuZt4sQxIiIiO8UiTUREZKdYpGWkUqmQmpoKlUoldyo2wzY5BrbJMbBNjqEz28SJY0RERHaKPWkiIiI7xSJNRERkp1ikiYiI7BSLtIzWrFmDwMBAqNVqREVFIS8vT+6U2i0tLQ333nsvPDw80K9fPyQkJODUqVMWMfX19UhKSoK3tzd69+6NKVOmoLy8XKaMrfP2229DoVBgwYIF0jFHbM/FixfxX//1X/D29oabmxtCQ0Nx6NAh6bwQAikpKfDz84ObmxtiY2Nx5swZGTO+OZPJhGXLlkGn08HNzQ1DhgzB//zP/1gsw2jvbdq7dy8mTpwIf39/KBQKZGRkWJxvT/7V1dVITEyEp6cnvLy8MGfOHFy5cqULW2HpZm1qamrCq6++itDQULi7u8Pf3x8zZ85EWVmZxWs4Upv+07PPPguFQoFVq1ZZHLdFm1ikZbJlyxYkJycjNTUVBQUFCAsLQ1xcHCoqKuROrV327NmDpKQkHDhwAFlZWWhqasJDDz2Euro6Keall17C9u3bsXXrVuzZswdlZWWYPHmyjFm3z8GDB7Fu3ToMHz7c4rijtefHH3/E/fffDxcXF+zatQvHjx/HypUr0adPHykmPT0dq1evxtq1a5Gbmwt3d3fExcWhvr5exsxv7J133sEHH3yAP/7xjzhx4gTeeecdpKen47333pNi7L1NdXV1CAsLw5o1a9o83578ExMTcezYMWRlZWHHjh3Yu3cv5s2b11VNuM7N2nT16lUUFBRg2bJlKCgowKeffopTp05h0qRJFnGO1Kaf++yzz3DgwAH4+/tfd84mbbLZztRklcjISJGUlCQ9N5lMwt/fX6SlpcmYVcdVVFQIAGLPnj1CCCFqamqEi4uL2Lp1qxRz4sQJAUDk5OTIleYt1dbWiqCgIJGVlSXGjBkj5s+fL4RwzPa8+uqrIiYm5obnzWaz0Gq1YsWKFdKxmpoaoVKpxKZNm7oiRavFx8eLp59+2uLY5MmTRWJiohDC8doEQHz22WfS8/bkf/z4cQFAHDx4UIrZtWuXUCgU4uLFi12W+438Z5vakpeXJwCI8+fPCyEct00//PCD6N+/vyguLhaDBg0Sv//976VztmoTe9IyaGxsRH5+PmJjY6VjSqUSsbGxyMnJkTGzjjMYDACAvn37AgDy8/PR1NRk0cbg4GAMHDjQrtuYlJSE+Ph4i7wBx2zPtm3bEBERgSeeeAL9+vXDyJEj8ac//Uk6X1JSAr1eb9EmjUaDqKgou23T6NGjkZ2djdOnTwMAioqKsG/fPowfPx6AY7bp59qTf05ODry8vBARESHFxMbGQqlUIjc3t8tz7giDwQCFQgEvLy8Ajtkms9mMGTNm4JVXXsGwYcOuO2+rNnHtbhlcvnwZJpMJvr6+Fsd9fX1x8uRJmbLqOLPZjAULFuD+++9HSEgIAECv18PV1VX6j7CVr68v9Hq9DFne2ubNm1FQUICDBw9ed84R23P27Fl88MEHSE5OxuLFi3Hw4EG8+OKLcHV1xaxZs6S82/o9tNc2vfbaazAajQgODoaTkxNMJhPeeustJCYmAoBDtunn2pO/Xq9Hv379LM47Ozujb9++DtHG+vp6vPrqq5g+fbq0zrUjtumdd96Bs7MzXnzxxTbP26pNLNJ025KSklBcXIx9+/bJnUqHXbhwAfPnz0dWVhbUarXc6diE2WxGREQE/vd//xcAMHLkSBQXF2Pt2rWYNWuWzNl1zN/+9jd88skn2LhxI4YNG4bDhw9jwYIF8Pf3d9g29SRNTU2YOnUqhBD44IMP5E6nw/Lz8/GHP/wBBQUFHd4Fsb043C0DHx8fODk5XTczuLy8HFqtVqasOub555/Hjh078NVXX1nsCqbVatHY2IiamhqLeHttY35+PioqKjBq1Cg4OzvD2dkZe/bswerVq+Hs7AxfX1+Hag8A+Pn5YejQoRbH7rnnHpSWlgKAlLcj/R6+8soreO211/Dkk08iNDQUM2bMwEsvvYS0tDQAjtmmn2tP/lqt9roJps3NzaiurrbrNrYW6PPnzyMrK8tityhHa9M333yDiooKDBw4UPq8OH/+PF5++WUEBgYCsF2bWKRl4OrqivDwcGRnZ0vHzGYzsrOzER0dLWNm7SeEwPPPP4/PPvsMu3fvhk6nszgfHh4OFxcXizaeOnUKpaWldtnGcePG4ejRozh8+LD0iIiIQGJiovRnR2oPANx///3XfS3u9OnTGDRoEABAp9NBq9VatMloNCI3N9du23T16tXr9ul1cnKC2WwG4Jht+rn25B8dHY2amhrk5+dLMbt374bZbEZUVFSX59werQX6zJkz+PLLL+Ht7W1x3tHaNGPGDBw5csTi88Lf3x+vvPIKvvjiCwA2bFPH57vR7di8ebNQqVRiw4YN4vjx42LevHnCy8tL6PV6uVNrl+eee05oNBrx9ddfi0uXLkmPq1evSjHPPvusGDhwoNi9e7c4dOiQiI6OFtHR0TJmbZ2fz+4WwvHak5eXJ5ydncVbb70lzpw5Iz755BPRq1cv8de//lWKefvtt4WXl5f4/PPPxZEjR8Sjjz4qdDqduHbtmoyZ39isWbNE//79xY4dO0RJSYn49NNPhY+Pj1i4cKEUY+9tqq2tFYWFhaKwsFAAEL/73e9EYWGhNNO5Pfk//PDDYuTIkSI3N1fs27dPBAUFienTp8vVpJu2qbGxUUyaNEkMGDBAHD582OLzoqGhwSHb1Jb/nN0thG3axCIto/fee08MHDhQuLq6isjISHHgwAG5U2o3AG0+PvroIynm2rVr4re//a3o06eP6NWrl3jsscfEpUuX5EvaSv9ZpB2xPdu3bxchISFCpVKJ4OBgsX79eovzZrNZLFu2TPj6+gqVSiXGjRsnTp06JVO2t2Y0GsX8+fPFwIEDhVqtFoMHDxZLliyx+LC39zZ99dVXbf63M2vWLCFE+/KvqqoS06dPF7179xaenp5i9uzZora2VobWtLhZm0pKSm74efHVV185ZJva0laRtkWbuAsWERGRneI9aSIiIjvFIk1ERGSnWKSJiIjsFIs0ERGRnWKRJiIislMs0kRERHaKRZqIiMhOsUgTERHZKRZpIiIiO8UiTUSSyspKPPfccxg4cCBUKhW0Wi3i4uLw7bffAgAUCgUyMjLkTZKoB+F+0kQkmTJlChobG/Hxxx9j8ODBKC8vR3Z2NqqqquROjahH4trdRAQAqKmpQZ8+ffD1119jzJgx150PDAzE+fPnpeeDBg3CuXPnAACff/453njjDRw/fhz+/v6YNWsWlixZAmfnln6AQqHA+++/j23btuHrr7+Gn58f0tPT8fjjj3dJ24gcFYe7iQgA0Lt3b/Tu3RsZGRloaGi47vzBgwcBAB999BEuXbokPf/mm28wc+ZMzJ8/H8ePH8e6deuwYcMGvPXWWxbXL1u2DFOmTEFRURESExPx5JNP4sSJE53fMCIHxp40EUn+8Y9/YO7cubh27RpGjRqFMWPG4Mknn8Tw4cMBtPSIP/vsMyQkJEjXxMbGYty4cVi0aJF07K9//SsWLlyIsrIy6bpnn30WH3zwgRRz3333YdSoUXj//fe7pnFEDog9aSKSTJkyBWVlZdi2bRsefvhhfP311xg1ahQ2bNhww2uKiorw5ptvSj3x3r17Y+7cubh06RKuXr0qxUVHR1tcFx0dzZ400S1w4hgRWVCr1XjwwQfx4IMPYtmyZfjNb36D1NRUPPXUU23GX7lyBW+88QYmT57c5msRUcexJ01ENzV06FDU1dUBAFxcXGAymSzOjxo1CqdOncKdd9553UOp/PdHzIEDByyuO3DgAO65557ObwCRA2NPmogAAFVVVXjiiSfw9NNPY/jw4fDw8MChQ4eQnp6ORx99FEDLDO/s7Gzcf//9UKlU6NOnD1JSUjBhwgQMHDgQjz/+OJRKJYqKilBcXIzly5dLr79161ZEREQgJiYGn3zyCfLy8vDhhx/K1Vwih8CJY0QEAGhoaMDrr7+Of/3rX/j+++/R1NSEgIAAPPHEE1i8eDHc3Nywfft2JCcn49y5c+jfv7/0FawvvvgCb775JgoLC+Hi4oLg4GD85je/wdy5cwG0TBxbs2YNMjIysHfvXvj5+eGdd97B1KlTZWwxkf1jkSaiTtfWrHAiujXekyYiIrJTLNJERER2ihPHiKjT8a4aUcewJ01ERGSnWKSJiIjsFIs0ERGRnWKRJiIislMs0kRERHaKRZqIiMhOsUgTERHZKRZpIiIiO8UiTUREZKf+H2noFlcGZmfMAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "total_training_steps = len(train_loader) * n_epochs\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "plt.tight_layout(); plt.savefig(\"1.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8",
   "metadata": {},
   "source": [
    "## D.2 Cosine decay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5216214-de79-40cf-a733-b1049a73023c",
   "metadata": {},
   "source": [
    "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n",
    "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n",
    "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses,  which is crucial for stabilizing the training in its later stages\n",
    "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4e8d2068-a057-4abf-b478-f02cc37191f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "min_lr = 0.1 * initial_lr\n",
    "track_lrs = []\n",
    "\n",
    "lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "global_step = -1\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for input_batch, target_batch in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        global_step += 1\n",
    "    \n",
    "        # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "        if global_step < warmup_steps:\n",
    "            # Linear warmup\n",
    "            lr = initial_lr + global_step * lr_increment  \n",
    "        else:\n",
    "            # Cosine annealing after warmup\n",
    "            progress = ((global_step - warmup_steps) / \n",
    "                        (total_training_steps - warmup_steps))\n",
    "            lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "        \n",
    "        # Apply the calculated learning rate to the optimizer\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group[\"lr\"] = lr\n",
    "        track_lrs.append(optimizer.defaults[\"lr\"])\n",
    "    \n",
    "        # Calculate loss and update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0e779e33-8a44-4984-bb23-be0603dc4158",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABQLklEQVR4nO3deVxU9f7H8dcMywwqDAjKgIqSWi4gKCiilpUUmaaYN5dMzSzNrDQqS0utbv24Wt7KMpduV20xzW6pmVGGWyqC4r7gkgsqDqDIqmwz5/cHMV2uaKDAmYHP8/GYB8053zO8v2p8OOd8z/erURRFQQghhBA2R6t2ACGEEEJUTIq0EEIIYaOkSAshhBA2Soq0EEIIYaOkSAshhBA2Soq0EEIIYaOkSAshhBA2Soq0EEIIYaMc1Q5grywWC6mpqbi6uqLRaNSOI4QQQmWKopCbm4uvry9abfWcA0uRvkmpqam0aNFC7RhCCCFszNmzZ2nevHm1fJYU6Zvk6uoKlP5luLm5qZxGCCGE2nJycmjRooW1PlQHKdI3qewSt5ubmxRpIYQQVtV5C1QGjgkhhBA2Soq0EEIIYaOkSAshhBA2SvUiPW/ePFq1aoVerycsLIzExMQbtl+5ciXt2rVDr9cTGBjIunXryu3/7rvvuP/++/H09ESj0bB3795rPqOgoICJEyfi6elJo0aNGDx4MGlpadXZLSGEEOKWqVqkV6xYQXR0NDNnzmT37t0EBQURGRlJenp6he23b9/O8OHDGTt2LHv27CEqKoqoqCgOHjxobZOfn0+vXr2YNWvWdb/vCy+8wA8//MDKlSvZvHkzqampPPzww9XePyGEEOJWaBRFUdT65mFhYXTt2pWPP/4YKJ0gpEWLFjz33HO8+uqr17QfOnQo+fn5rF271rqte/fuBAcHs2DBgnJtT58+jb+/P3v27CE4ONi6PTs7myZNmrBs2TL+9re/AZCcnEz79u2Jj4+ne/fulcqek5ODwWAgOztbRncLIYSokbqg2iNYRUVFJCUlMXXqVOs2rVZLREQE8fHxFR4THx9PdHR0uW2RkZGsWrWq0t83KSmJ4uJiIiIirNvatWuHn5/fDYt0YWEhhYWF1vc5OTmV/p6i6iwWhXfWHeF4eh46Ry16J4c/vmppqHOkSSMdTd30NHXV4e2mx9tNRwNneaJQCFG3qPZT7eLFi5jNZry9vctt9/b2Jjk5ucJjTCZThe1NJlOlv6/JZMLZ2Rl3d/cqfU5MTAxvvvlmpb+PuDXxJy/x2dZTVTrG6KanrXcj2jRtRNumrrT1bkR7Hzca6aR4CyHsk/z0qqSpU6eWO4svm1lG1Iw1e1MBuPuOJtzfwUhhiZmCYguFJWZyC0pIzy0kLaeAjNxC0nMKyC8yY8opwJRTwG/HL1o/R6uBdkY3Qlt5ENKy9NXM3UXmWxdC2AXVirSXlxcODg7XjKpOS0vDaDRWeIzRaKxS++t9RlFREVlZWeXOpv/qc3Q6HTqdrtLfR9y8whIz6w5eAGD8Xa0Jb+35l8dkXy3mRHoeJ9JzOZ6Wx/H0PI6l5XIhu4DDF3I4fCGHz+PPANDM3YW772jCve2a0qO1Fy7ODjXaHyGEuFmqFWlnZ2dCQkKIi4sjKioKKB04FhcXx7PPPlvhMeHh4cTFxTF58mTrtvXr1xMeHl7p7xsSEoKTkxNxcXEMHjwYgKNHj5KSklKlzxE1Z/PRDHILSjC66enm37hSxxhcnKxnyv/NlF3ArjOZJJ25TNKZyxxKzeF81lW+Skjhq4QUdI5aerT25N723vQNMOLVSH4RE0LYDlUvd0dHRzN69GhCQ0Pp1q0bH3zwAfn5+YwZMwaAUaNG0axZM2JiYgCYNGkSvXv3Zs6cOfTr14/ly5eza9cuFi1aZP3MzMxMUlJSSE0tvVx69OhRoPQM2mg0YjAYGDt2LNHR0TRu3Bg3Nzeee+45wsPDKz2yW9Ss1ftK/+76d/LBQXtrl6WNBj39O/nSv5MvAFeKSoj//RIbktPZmJxOanYBG49msPFoBm+sOcSdbb2ICm7G/R29ZSCaEEJ1qv4UGjp0KBkZGcyYMQOTyURwcDCxsbHWwWEpKSnl1uTs0aMHy5Yt4/XXX2fatGm0bduWVatWERAQYG2zZs0aa5EHGDZsGAAzZ87kjTfeAOD9999Hq9UyePBgCgsLiYyM5JNPPqmFHou/kldYQtyR0lsaA4ObVfvnN3B2pE97b/q090ZRFI6m5bIhOZ3Ygyb2n8tm09EMNh3NoIGzA/d38GZI1xaE3+Yp97CFEKpQ9TlpeybPSdeM7/ec44UV+/D3asiGF3vXanH8PSOP1XtTWb33PGcuXbFub9u0ESPDWzKoczNc9U61lkcIYV9qoi5Ikb5JUqRrxpjFiWw8msGkPm154b7bVcmgKAp7z2bxbdI5vt9znitFZgAaOjvwcJfmjO7RkjZNq2+9WCFE3SBF2oZIka5+mflFdHvnV0osCnEv9qZ1k0ZqRyK3oJjvdp/n8/jT/J6RD4BGA5EdjEy8pw2BzQ0qJxRC2Io6NeOYEP9r3YELlFgUApq52USBBnDVOzG6RytGhbdk+++XWLL9NOsPpxF7yETsIRN33d6EiXe3Juy2v35MTAghqkqKtLAZZROYDAjyVTnJtTQaDT3beNGzjRfH0nKZv+l31uxLZcuxDLYcy6Bbq8a8FHlHpR8ZE0KIylB9qUohAFKzrpJ4OhONBh6ywSL93273duX9ocFsfPFuHg3zw9lBS+LpTIYsjGfskp0cNeWqHVEIUUdIkRY24Yc/no3u2qoxPgYXldNUjp9nA/5vUCBbptzDo2F+OGg1xCWn88CHW3hp5T7OZ11VO6IQws5JkRY2Yc0fRXpgsG2fRVfEaNDzf4MC+eWFu+gbYERR4Nukc9zz3ibe/TmZK0UlakcUQtgpKdJCdSfS8ziUmoOjVsODAT5qx7lprZs0Yv5jIXz3TA/C/BtTVGJh3sbf6TNnMz/uv4A8SCGEqCop0kJ1ZWfRd93eBI+GziqnuXVd/DxYPq47Cx4LoZm7CxeyC5i4bDePfZbAiXS5Xy2EqDwp0kJViqKwZu95wDZHdd8sjUbDAwFGfo3uzfN92uLsqGXbiUs88MFvxPx0hIJis9oRhRB2QIq0UNWB89mcvnQFvZOW+zp4qx2n2rk4OxB93+38+kJvItp7U2JRWLj5JH0//I2Ek5fUjieEsHFSpIWqVv/xbHREe28a6uruY/t+ng341+hQPh0VirebjlMX8xm6aAfTVx0kt6BY7XhCCBslRVqoxmxRWLu/bFR39a94ZYvu6+DNLy/0Zni3FgB8seMMke9vYePRdJWTCSFskRRpoZqEU5dIyynETe/IXbd7qR2n1hhcnIh5uBPLngzDr3EDUrMLGLN4J699f0Ae1xJClCNFWqimbAKTvgE+6BwdVE5T+3q08SJ28p2M6dkKgK8SUug/dyv7zmapmksIYTukSAtVFJVYWHfABNjnBCbVpYGzIzMf6shXT4ZhdNNz8mI+g+dv56O445SYLWrHE0KoTIq0UMWWYxlkXy2mqatOVpACev5xVt2vkw8lFoU5648xdNEOzmZeUTuaEEJFUqSFKlb/cam7fydfHLQaldPYBvcGznw8vDPvDw3CVedI0pnL9Jv7G+sPp6kdTQihEinSotblF5bw6x+FZ0A9vtRdEY1Gw6DOzVk36U6CWriTU1DCU5/v4v/WHaFYLn8LUe9IkRa17tcjaVwtNtPSswFBzQ1qx7FJLRo3YOX4cJ7o6Q/Aoi0nGbownlRZWUuIekWKtKh1a/6YwGRAkC8ajVzqvh5nRy0zHurAgsdCcNU7sjsliwfn/sYmeaZaiHpDirSoVZfzi9h8LAOo36O6q+KBACM/Pncngc0MZF0pZsySnczbeEJW1RKiHpAiLWrVTwdNlFgU2vu40aapq9px7IafZwO+nRDO8G5+KAq8+/NRJi7bTX6hTH4iRF0mRVrUqtV/rHglZ9FVp3N0IObhQP5vUCBODhrWHTDx8CfbOXMpX+1oQogaIkVa1JoL2VdJPJ0JwEN1aFnK2vZomB/Lx3WniauOo2m5PPTRVustBCFE3SJFWtSatfsuoCjQtZUHzdxd1I5j10JaNmbtc73o7Ff6mNaYxYks2XZK7VhCiGomRVrUmjX7/hzVLW6dt5ue5eO6MyS0ORYF3vjhMDNWH5TpRIWoQ6RIi1pxMiOPA+ezcdBqeDDQR+04dYbO0YFZgzsxtW87NBr4PP4MTyzdRY6sUS1EnSBFWtSKsrPoXm288GykUzlN3aLRaBjfuzULHgvBxcmBLccyGPzJdpn3W4g6QIq0qHGKolgnMJFR3TUnsqORlU+H4+2m43h6HlHztrEn5bLasYQQt0D1Ij1v3jxatWqFXq8nLCyMxMTEG7ZfuXIl7dq1Q6/XExgYyLp168rtVxSFGTNm4OPjg4uLCxERERw/frxcm2PHjjFw4EC8vLxwc3OjV69ebNy4sdr7JkodSs3h5MV8dI5a7u9oVDtOnRbQzMDqib3o6OvGpfwihn+6g7gjskCHEPZK1SK9YsUKoqOjmTlzJrt37yYoKIjIyEjS0yue9nD79u0MHz6csWPHsmfPHqKiooiKiuLgwYPWNrNnz2bu3LksWLCAhIQEGjZsSGRkJAUFBdY2/fv3p6SkhA0bNpCUlERQUBD9+/fHZDLVeJ/ro7JnoyPae9NI56hymrrPaNDzzfhwet/ehIJiC099votlCSlqxxJC3AxFRd26dVMmTpxofW82mxVfX18lJiamwvZDhgxR+vXrV25bWFiYMn78eEVRFMVisShGo1F59913rfuzsrIUnU6nfP3114qiKEpGRoYCKFu2bLG2ycnJUQBl/fr1lc6enZ2tAEp2dnalj6mPzGaL0v3/flVavrJW+enABbXj1CtFJWblxW/2Ki1fWau0fGWtMufnZMVisagdS4g6qybqgmpn0kVFRSQlJREREWHdptVqiYiIID4+vsJj4uPjy7UHiIyMtLY/deoUJpOpXBuDwUBYWJi1jaenJ3fccQeff/45+fn5lJSUsHDhQpo2bUpISEh1d7Pe23k6kwvZBbjqHbn7jiZqx6lXnBy0vPu3Tjx/bxsA5m44wZRv98uSl0LYEdWuPV68eBGz2Yy3t3e57d7e3iQnJ1d4jMlkqrB92WXqsq83aqPRaPj111+JiorC1dUVrVZL06ZNiY2NxcPD47p5CwsLKSwstL7PycmpZE/rt9V/jOp+oKMRvZODymnqH41GQ/T9d+Bt0DN91UFWJp0jM7+IeSO6yN+HEHZA9YFjtU1RFCZOnEjTpk357bffSExMJCoqioceeogLFy5c97iYmBgMBoP11aJFi1pMbZ+KSiysO1D6ZzowuJnKaeq3EWEtWTgyFJ2jlrjkdB5fnEieLM4hhM1TrUh7eXnh4OBAWlr5kadpaWkYjRWPADYajTdsX/b1Rm02bNjA2rVrWb58OT179qRLly588sknuLi4sHTp0uvmnTp1KtnZ2dbX2bNnq9bhemjriQyyrhTj1UhHeGtPtePUe/d18GbpE91opHNkx8lMRny6g8v5RWrHEkLcgGpF2tnZmZCQEOLi4qzbLBYLcXFxhIeHV3hMeHh4ufYA69evt7b39/fHaDSWa5OTk0NCQoK1zZUrpRM8aLXlu67VarFYrn+vTqfT4ebmVu4lbqzs2ej+nXxw0GpUTiMAut/mybKnwvBo4MS+c9kMWRhPWk7BXx8ohFCFqpe7o6Oj+fTTT1m6dClHjhxhwoQJ5OfnM2bMGABGjRrF1KlTre0nTZpEbGwsc+bMITk5mTfeeINdu3bx7LPPAqX33yZPnszbb7/NmjVrOHDgAKNGjcLX15eoqCigtNB7eHgwevRo9u3bx7Fjx3j55Zc5deoU/fr1q/U/g7rqapGZXw6XXtEYIBOY2JROzd35Zvyfk578bcF2Ui7J7GRC2CJVi/TQoUN57733mDFjBsHBwezdu5fY2FjrwK+UlJRy94l79OjBsmXLWLRoEUFBQXz77besWrWKgIAAa5spU6bw3HPPMW7cOLp27UpeXh6xsbHo9Xqg9DJ7bGwseXl53HvvvYSGhrJ161ZWr15NUFBQ7f4B1GG/HknjSpGZFo1d6NzCXe044n+09Xbl26d74Ne4AWczr/K3Bds5lpardiwhxP/QKIqiqB3CHuXk5GAwGMjOzpZL3xV4cukufj2SxsR7WvNyZDu144jrSM8p4LHPEjiWlod7AyeWjulGkPxSJcRNqYm6UO9Gd4ual32lmM3HSmeNk1Hdtq2pm54V48IJauFO1pViHv10BwknL6kdSwjxBynSotr9dPACxWaFdkZXbvd2VTuO+AseDZ356skwwm/zJL/IzOOLdxL/uxRqIWyBFGlR7cqWpZQBY/ajkc6RxWO6ctftTbhabGbMkkS2n7iodiwh6j0p0qJapeUUEP/H5dKHOkmRtid6JwcWjQzh7jtKF+YYs2QnW49LoRZCTVKkRbVau/8CigIhLT1o0biB2nFEFemdHFg4MoR72zWlsMTC2KU72XIsQ+1YQtRbUqRFtVrzx7KUA4LkLNpe6RwdmP9YFyLalxbqJz/fxaajFS8fK4SoWVKkRbU5fTGffeeycdBqeDDQR+044hboHB34ZEQI93XwpqjEwrjPk9iYLIVaiNomRVpUm7IBYz1ae9LEVadyGnGrnB21zHu0C5EdvSkyWxj/RRJxR9L++kAhRLWRIi2qhaIorJZL3XWOs6OWjx/tQt8AI0VmC09/KYVaiNokRVpUi8MXcvg9Ix9nRy2RARWvYibsk5ODlrnDO9Mv0Idis8KEL3fLYDIhaslNFemSkhJ+/fVXFi5cSG5u6Xy/qamp5OXlVWs4YT/KLnXfe0dT3PROKqcR1c3JQcsHw4Ktl77HfbFLJjwRohZUuUifOXOGwMBABg4cyMSJE8nIKP2NetasWbz00kvVHlDYPotF4Yc/lqUcKBOY1FlODlo+Gt6Fe9s1paC49PGspDOZascSok6rcpGeNGkSoaGhXL58GRcXF+v2QYMGXbPWs6gfklIuk5pdQCOdI/e0a6p2HFGDnB21fDKiC73aeHGlyMzj/97J/nNZascSos6qcpH+7bffeP3113F2di63vVWrVpw/f77aggn7UTZgLLKjEb2Tg8ppRE3TOzmwaFQI3fwbk1tYwsjPEjmcmqN2LCHqpCoXaYvFgtlsvmb7uXPncHWVxRTqm2KzhXUHTIDM1V2fNHB25N+Pd6WznzvZV4sZ+VkCx2U9aiGqXZWL9P33388HH3xgfa/RaMjLy2PmzJk8+OCD1ZlN2IGtJy6SmV+EZ0Nnerb2VDuOqEWNdI4sGdONgGZuXMov4tF/JXDqYr7asYSoU6pcpOfMmcO2bdvo0KEDBQUFPProo9ZL3bNmzaqJjMKGlQ0Y69fJB0cHeaKvvjG4OPHFE2G0M7qSkVvIo5/u4GzmFbVjCVFnaBRFUap6UElJCStWrGDfvn3k5eXRpUsXRowYUW4gWV2Xk5ODwWAgOzsbNzc3teOo4mqRmdC315NfZOY/E8IJadlY7UhCJRfzChm6MJ7fM/Jp6dmAlU+H09RVr3YsIWpVTdSFKhfpLVu20KNHDxwdHcttLykpYfv27dx1113VEszWSZGGH/dfYOKy3TRzd2HrK/eg0WjUjiRUlJZTwOD52zl3+SrtjK6sGBeOoYE8My/qj5qoC1W+PnnPPfeQmXnts5HZ2dncc8891RJK2AfrNKDBvlKgBd5uer56MowmrjqSTbmMWZLIlaIStWMJYdeqXKQVRanwB/KlS5do2LBhtYQSti/7ajGbjpZOZCNzdYsyLT0b8sXYbhhcnNidksX4L5IoLLn2aRAhROU4/nWTUg8//DBQOpr78ccfR6f7c5Ujs9nM/v376dGjR/UnFDbp54MmiswWbvduRDujPHon/tTO6MbiMV157F8J/Hb8IpO+3svHj3aWgYVC3IRK/19jMBgwGAwoioKrq6v1vcFgwGg0Mm7cOL788suazCpsSNlc3QOC5FK3uFYXPw8WjQzF2UFL7CETU787gMVS5TGqQtR7lT6TXrx4MVA6s9hLL70kl7brsfTcArb/fhGAAUHNVE4jbFWvtl7MHR7MM1/tZmXSOdxcnHi9X3v5pU6IKqjy9aeZM2dKga7nftx/AYsCwS3c8fNsoHYcYcMeCPBh1uBOAHy29RQfbTihciIh7Eulz6T/27fffss333xDSkoKRUVF5fbt3r27WoIJ27VaVrwSVfBIaAtyCkr4+9rD/HP9MTwaODEyvJXasYSwC1U+k547dy5jxozB29ubPXv20K1bNzw9PTl58iR9+/atiYzChqRcusLes1loNaWzjAlRGWN7+fN8n7YAzFhziB/3X1A5kRD2ocpF+pNPPmHRokV89NFHODs7M2XKFNavX8/zzz9PdnZ2TWQUNmTNvtJno3u09pIZpUSVvBDRlkfD/FAUeGHFXrafuKh2JCFsXpWLdEpKivVRKxcXF3JzS1e+GTlyJF9//XX1phM2579HdQtRFRqNhr8PDOCBjkaKzBbGfZHEwfPyi70QN1LlIm00Gq0zjvn5+bFjxw4ATp06xU1MAy7sSLIph2NpeTg7aIkMMKodR9ghB62GD4YFE+bfmLzCEh5fvJMzl2TlLCGup8pF+t5772XNmjUAjBkzhhdeeIH77ruPoUOHMmjQoCoHmDdvHq1atUKv1xMWFkZiYuIN269cuZJ27dqh1+sJDAxk3bp15fYrisKMGTPw8fHBxcWFiIgIjh8/fs3n/Pjjj4SFheHi4oKHhwdRUVFVzl7flA0Yu/uOJhhcZE5mcXP0Tg58OjqU9j5uXMwrZNS/E8nILVQ7lhA2qcpFetGiRbz22msATJw4kX//+9+0b9+et956i/nz51fps1asWEF0dDQzZ85k9+7dBAUFERkZSXp6eoXtt2/fzvDhwxk7dix79uwhKiqKqKgoDh48aG0ze/Zs5s6dy4IFC0hISKBhw4ZERkZSUFBgbfOf//yHkSNHMmbMGPbt28e2bdt49NFHq/pHUa8oisIa66hueTZa3Bo3vRNLx3SlRWMXzly6wuOLE8ktKFY7lhC2R6mC4uJi5c0331TOnj1blcOuq1u3bsrEiROt781ms+Lr66vExMRU2H7IkCFKv379ym0LCwtTxo8fryiKolgsFsVoNCrvvvuudX9WVpai0+mUr7/+2tqHZs2aKf/6179uKXt2drYCKNnZ2bf0OfZi1+lLSstX1iodpv+kXC0qUTuOqCNOZuQpXd76RWn5ylpl+KJ4paBY/m0J+1UTdaFKZ9KOjo7Mnj2bkpJbX9mmqKiIpKQkIiIirNu0Wi0RERHEx8dXeEx8fHy59gCRkZHW9qdOncJkMpVrYzAYCAsLs7bZvXs358+fR6vV0rlzZ3x8fOjbt2+5s3FxrbKz6MiORvRODiqnEXWFv1dDlozpRkNnB7b/fonoFfswy/ShQlhV+XJ3nz592Lx58y1/44sXL2I2m/H29i633dvbG5PJVOExJpPphu3Lvt6ozcmTJwF44403eP3111m7di0eHh7cfffdFS7BWaawsJCcnJxyr/qixGzhxwOlz7U+JBOYiGoW2NzAwpGhODlo+PHABd784ZAMQhXiD1Wecaxv3768+uqrHDhwgJCQkGumCB0wYEC1hasJFosFgNdee43BgwcDpfOSN2/enJUrVzJ+/PgKj4uJieHNN9+stZy2ZPvvl7iYV0Tjhs70auOldhxRB/Vq68U/hwTz/PI9fB5/hqauOp69t63asYRQXZWL9DPPPAPAP//5z2v2aTQazObKrR3r5eWFg4MDaWlp5banpaVhNFb8eI/RaLxh+7KvaWlp+Pj4lGsTHBwMYN3eoUMH636dTsdtt91GSkrKdfNOnTqV6Oho6/ucnBxatGjxV92sE8pGdT8YaMRJlhsUNeShIF8u5RXyxg+Hee+XY3g20jG8m5/asYRQVZV/4losluu+KlugAZydnQkJCSEuLq7cZ8fFxREeHl7hMeHh4eXaA6xfv97a3t/fH6PRWK5NTk4OCQkJ1jYhISHodDqOHj1qbVNcXMzp06dp2bLldfPqdDrc3NzKveqDgmIzPx8qvVUgo7pFTXu8pz8T72kNwGvfHyDuSNpfHCFE3abqaVF0dDSffvopS5cu5ciRI0yYMIH8/HzGjBkDwKhRo5g6daq1/aRJk4iNjWXOnDkkJyfzxhtvsGvXLp599lmg9Ex+8uTJvP3226xZs4YDBw4watQofH19rc9Bu7m58fTTTzNz5kx++eUXjh49yoQJEwB45JFHavcPwA5sTE4nr7AEX4OeED8PteOIeuCl++9gSGhzLAo8u2wPe89mqR1JCNXc1CpY1WXo0KFkZGQwY8YMTCYTwcHBxMbGWgd+paSkoNX++XtEjx49WLZsGa+//jrTpk2jbdu2rFq1ioCAAGubKVOmkJ+fz7hx48jKyqJXr17Exsai1/85z/S7776Lo6MjI0eO5OrVq4SFhbFhwwY8PKQI/a+yaUAfCvZFq5V1gEXN02g0vDMokLScQjYfy+CJJTv5bkIPWnnJErmi/tEoMozypuTk5GAwGMjOzq6zl75zCooJfftXikos/Ph8Lzr6GtSOJOqR/MIShi3awYHz2bT0bMB/JvTAq5FO7VhCXFdN1AUZBSSu65dDaRSVWGjdpCEdfOrmLyLCdjXUOfLvx/+clWzskp1cKbr1ORqEsCdSpMV1rd5buizlwOBmaDRyqVvUviauOpaM6YZHAyf2ncvm2WV7KDFb1I4lRK2pcpH+3wk9yl65ubkUFRXVREahgozcQrb/fgmQZSmFulo3acS/RndF56hlQ3I601cflMlORL1R5SLt7u6Oh4fHNS93d3dcXFxo2bIlM2fOtE4aIuzTugMXMFsUgpobZMCOUF1ISw/mDu+MVgNfJ57low0n1I4kRK2ocpFesmQJvr6+TJs2jVWrVrFq1SqmTZtGs2bNmD9/PuPGjWPu3Ln84x//qIm8opZYR3XLWbSwEZEdjbw5oCMA/1x/jG92nVU5kRA1r8qPYC1dupQ5c+YwZMgQ67aHHnqIwMBAFi5cSFxcHH5+frzzzjtMmzatWsOK2nE28wpJZy6j0UiRFrZlZHgrUrMLmL/pd6Z+d4CmrjruvqOp2rGEqDFVPpPevn07nTt3vmZ7586drStN9erV64ZTbArb9sP+0rPo7v6eeLvp/6K1ELVrSuQdDOrcDLNF4ZmvdnPgXLbakYSoMVUu0i1atOCzzz67Zvtnn31mncv60qVLMjGIHStblnKgrHglbJBGo2HW4E70auPFlSIzY5bs5GzmFbVjCVEjqny5+7333uORRx7hp59+omvXrgDs2rWL5ORkvv32WwB27tzJ0KFDqzepqBVHTbkkm3JxctDQN8Dnrw8QQgXOjlrmP9aFIQt3cORCDqP/nci3E3rQuKGz2tGEqFZVPpMeMGAAycnJ9O3bl8zMTDIzM+nbty/Jycn0798fgAkTJlS4SpawfWv2lT4b3fv2phgaOKmcRojrc9U7sWRMV5q5u3DyYj5PLt3J1aLKL/IjhD2QaUFvUl2cFlRRFO56dyNnM68yd3hneT5a2IXjabkMnr+dnIIS7u/gzfzHQnCQeeaFCmqiLtzUAhtZWVkkJiaSnp5+zfPQo0aNqpZgovbtOZvF2cyrNHB2IKK9jJgV9qGttyv/Gt2Vxz5L4JfDabyx5hBvDewos+SJOqHKRfqHH35gxIgR5OXl4ebmVu5/BI1GI0XajpUNGLuvgzcNnFVdIE2IKunm35gPhgYzcdluvthxBh93Pc/c3UbtWELcsirfk37xxRd54oknyMvLIysri8uXL1tfmZmZNZFR1IISs4W1+y8AMqpb2KcHA32Y3q8DALNjj/L9nnMqJxLi1lW5SJ8/f57nn3+eBg0a1EQeoZIdJzO5mFeIewMnerVponYcIW7KE738eepOfwBeXrmfrccvqpxIiFtT5SIdGRnJrl27aiKLUFHZilcPBvrg7CiLown7NbVvex4K8qXEovD0l0kcSpXJToT9qvKNx379+vHyyy9z+PBhAgMDcXIq/5jOgAEDqi2cqB0FxWZiD5kAWfFK2D+tVsN7j3QiI7eAHSczGbN4J98904PmHnL1T9ifKj+CpdVe/yxLo9FgNteP5xTr0iNYsQdNPP1lEkY3PdtfvRetPL4i6oDsq8UMWRDP0bRcWjdpyH8m9MC9gUx2ImpOTdSFKl/XtFgs133VlwJd1/xgXfHKRwq0qDMMLk4seaIrPgY9v2fk8+TSXRQUy88oYV/k5mM9l1tQzK9H0gAYGNxM5TRCVC8fgwtLxnTDVe/IrjOXmbx8L2aLzN8k7Eel7knPnTuXcePGodfrmTt37g3bPv/889USTNSO9YfTKCyxcJtXQzr62vdleyEqcofRlU9HhTLqs0RiD5n4+9rDzHyog0x2IuxCpe5J+/v7s2vXLjw9PfH397/+h2k0nDx5sloD2qq6ck969L8T2Xwsg8kRbZkccbvacYSoMWv3p/Lssj0ATO3bjvG9W6ucSNQ1qk0LeurUqQr/W9i3S3mFbD1R+hypjOoWdV3/Tr6Ysgt4+8cjxPyUjNGgl1s8wubJPel6bN2BC5gtCoHNDNzWpJHacYSocU/eeRtje5VeDXxp5T62nZDJToRtq/Jz0mazmSVLlhAXF1fhAhsbNmyotnCiZq35Y1S3nEWL+uS1B9tjyingx/0XGP9FEt+MD6eDjMcQNqrKRXrSpEksWbKEfv36ERAQIIMv7NT5rKvsPH0ZjQb6B/moHUeIWqPVavjnkCAu5haScCqTMUsS+e6ZnjRzd1E7mhDXqHKRXr58Od988w0PPvhgTeQRtaTs2ehurRrjY5AfTqJ+0Tk6sGhUKI8s2M6xtDxG/zuRb58Ol8lOhM2p8j1pZ2dn2rSRJeDsXdmylDJwRtRXBhcnlozphtFNz4n0PMZ9niSTnQibc1NLVX744YdUcTZRYUNOpOdy+EIOjloNfQOMascRQjW+7i4seaIrrjpHEk9nEv2NTHYibEuVL3dv3bqVjRs38tNPP9GxY8drFtj47rvvqi2cqBllZ9G9b2+CR0O5vCfqt3ZGNxaOCuHxf+9k3QETTV1lshNhO6pcpN3d3Rk0aFBNZBG1QFEUVpeN6g6WUd1CAPRo7cV7Q4J4/us9LNl+Gl93PePukslOhPqqdLm7pKSEe+65h5iYGBYvXlzh62bMmzePVq1aodfrCQsLIzEx8YbtV65cSbt27dDr9QQGBrJu3bpy+xVFYcaMGfj4+ODi4kJERATHjx+v8LMKCwsJDg5Go9Gwd+/em8pvT/afy+bMpSu4ODkQ0d5b7ThC2IwBQb683q89AP+3Ltm6xroQaqpSkXZ0dOTpp5+msLCw2gKsWLGC6OhoZs6cye7duwkKCiIyMpL09PQK22/fvp3hw4czduxY9uzZQ1RUFFFRURw8eNDaZvbs2cydO5cFCxaQkJBAw4YNiYyMpKCg4JrPmzJlCr6+9eeMcvUfl7ojOnjTUFflCylC1GlP3nkbT/T8c7KT7TLZiVCbUkW9e/dWvv/++6oedl3dunVTJk6caH1vNpsVX19fJSYmpsL2Q4YMUfr161duW1hYmDJ+/HhFURTFYrEoRqNReffdd637s7KyFJ1Op3z99dfljlu3bp3Srl075dChQwqg7Nmzp9K5s7OzFUDJzs6u9DFqKzFblK5vr1davrJWWX/IpHYcIWyS2WxRnvkqSWn5ylolYEascjjVfv4fF+qqibpQ5dHdzzzzDC+++CIff/wx8fHx7N+/v9yrKoqKikhKSiIiIsK6TavVEhERQXx8fIXHxMfHl2sPEBkZaW1/6tQpTCZTuTYGg4GwsLByn5mWlsZTTz3FF198QYMGDf4ya2FhITk5OeVe9ibh5CXScwsxuDhx1+1N1I4jhE3SajXMeSSIbv6NyS0s4fHFiZzPuqp2LFFPVfl657Bhw4DyS1JqNBoURUGj0WA2V/45w4sXL2I2m/H2Ln9v1Nvbm+Tk5AqPMZlMFbY3mUzW/WXbrtdGURQef/xxnn76aUJDQzl9+vRfZo2JieHNN9+sVL9sVdk0oH0DjDg7yrTtQlyP3smBT0eG8sjC0slORn6WwLdP96CxPA0halmVi3RdWAXro48+Ijc3l6lTp1b6mKlTpxIdHW19n5OTQ4sWLWoiXo0oLDGz7sAFQEZ1C1EZhgZOLH2iG4M/2c7JjHzGLE5k2VPdZSyHqFVV/tfWsmXLavvmXl5eODg4kJaWVm57WloaRmPFk2wYjcYbti/7mpaWho+PT7k2wcHBQOkiIPHx8eh0unKfExoayogRI1i6dOk131en013T3p5sOXaRnIISmrrqCPP3VDuOEHbBx+DC52PDeGTBdvady+bpL5P4bHRXuRIlas1N/0s7fPgwsbGxrFmzptyrKpydnQkJCSEuLs66zWKxEBcXR3h4eIXHhIeHl2sPsH79emt7f39/jEZjuTY5OTkkJCRY28ydO5d9+/axd+9e9u7da32Ea8WKFbzzzjtV6oO9KHuc5KEgXxy0MkmDEJXVpmkjFo/pRgNnB347fpHob/ZikVnJRC2p8pn0yZMnGTRoEAcOHLDeiwass/NU5Z40QHR0NKNHjyY0NJRu3brxwQcfkJ+fz5gxYwAYNWoUzZo1IyYmBihdhat3797MmTOHfv36sXz5cnbt2sWiRYusOSZPnszbb79N27Zt8ff3Z/r06fj6+hIVFQWAn59fuQyNGpWupdy6dWuaN29e1T8Sm5dfWMKvR0qvPsiylEJUXXALdxaODOGJJTtZu/8Cng2deWNAR5mVTNS4Kp9JT5o0CX9/f9LT02nQoAGHDh1iy5YthIaGsmnTpioHGDp0KO+99x4zZswgODiYvXv3Ehsbax34lZKSwoULF6zte/TowbJly1i0aBFBQUF8++23rFq1ioCAAGubKVOm8NxzzzFu3Di6du1KXl4esbGx6PX6KuerC9YfTqOg2EIrzwZ0am5QO44QdunOtk2YMyQYjQaWxp/how0n1I4k6gGNolRtpQwvLy82bNhAp06dMBgMJCYmcscdd7BhwwZefPFF9uzZU1NZbUpOTg4Gg4Hs7Gzc3Gx7wfgnluxkQ3I6z9/bhuj771A7jhB2bcm2U7zxw2EA3o4K4LHu1TdOR9i3mqgLVT6TNpvNuLq6AqUFOzW19LGeli1bcvTo0WoJJarP5fwithzLAGRUtxDV4fGe/jx/b+lyvdNXH7Q+NSFETajyPemAgAD27duHv78/YWFhzJ49G2dnZxYtWsRtt91WExnFLVh38AIlFoUOPm60aeqqdhwh6oQX7rudi/lFLEtIYfLyvRhcnOjZxkvtWKIOqvKZ9Ouvv47FYgHgrbfe4tSpU9x5552sW7eOuXPnVntAcWvK5uoeKGfRQlQbjUbD3wcG0DfASJHZwrjPd3HgXLbasUQdVOV70hXJzMzEw8OjXo10tId70qlZV+k5awOKAttevZdm7i5qRxKiTiksMTNm8U62/34Jz4bOrHw6nNuaNFI7llCJTdyTLnPixAl+/vlnrl69SuPGjasljKhea/enoijQrVVjKdBC1ACdowMLR4YQ0MyNS/lFjPwskVSZ51tUoyoX6UuXLtGnTx9uv/12HnzwQevjUWPHjuXFF1+s9oDi5pXN1f2QXOoWosa46p1YMqYb/l4NOZ91lcf+lcDFvOpbzlfUb1Uu0i+88AJOTk6kpKSUWz1q6NChxMbGVms4cfN+z8jj4PkcHLUa+gX6/PUBQoib5tVIx5dPhuFr0HPyYj4jP0sk+0qx2rFEHVDlIv3LL78wa9asa2bmatu2LWfOnKm2YOLWrPljwFivtl6yco8QtaCZuwtfPdUdr0Y6jlzIYcySRPILS9SOJexclYt0fn5+hesvZ2Zm2vUCFHWJoijWS90yqluI2uPv1ZAvxnbDTe/I7pQsxn2xi4Liqk2VLMR/q3KRvvPOO/n888+t7zUaDRaLhdmzZ3PPPfdUazhxcw6ez+HUxXx0jlru61DxamJCiJrR3seNpU+ULsix7cQlnvt6D8Vmi9qxhJ2q8mQms2fPpk+fPuzatYuioiKmTJnCoUOHyMzMZNu2bTWRUVRR2YpXER28aSRr3wpR6zr7efCv0aE8vngn6w+n8dLKfbw/JBitrEAnqqjKZ9IBAQEcO3aMXr16MXDgQPLz83n44YfZs2cPrVu3romMogrMFoUf9pde6pYVr4RQT4/WXswf0QVHrYbVe1OZvvog1TAthahnbuo0y2Aw8Nprr5Xbdu7cOcaNG2ddMlKoI/FUJmk5hbjqHbn7jiZqxxGiXuvT3pt/Dg1m0vI9fJWQQkOdI1P7tqtXEz+JW3PTk5n8r0uXLvHZZ59V18eJm1Q2YKxvgBGdo4PKaYQQA4J8iRkUCMCiLSd575ejckYtKq3airRQX1GJxboiz8DgZiqnEUKUGdbNjzce6gDAvI2/88Gvx1VOJOyFFOk65LfjGWRfLaaJq47ut3mqHUcI8V8e7+nP6/3aA/Bh3HE+3iCFWvw1KdJ1SNmKV/07+eAgo0iFsDlP3nkbrzzQDoD3fjnGgs2/q5xI2LpKDxx7+OGHb7g/KyvrVrOIW3ClqIT1h9MAGdUthC2bcHdrSswW5qw/xj9+SsZRq+HJO29TO5awUZUu0gaD4S/3jxo16pYDiZuz/nAaV4vN+DVuQHALd7XjCCFu4Lk+bSmxKHwYd5y3fzyCk4OW0T1aqR1L2KBKF+nFixfXZA5xi37Y9+ez0fJ4hxC2b3JEW0osFuZt/J2Zaw7h6KBhRFhLtWMJGyP3pOuArCtFbD6WAchc3ULYC41Gw0v338H4u0ovdb/2/UGWJ6aonErYGinSdcBPB00UmxXaGV1p6+2qdhwhRCVpNBpe7duOJ3r6A/Dqdwf4Iv60uqGETZEiXQeUzdUtz0YLYX80Gg3T+7dnbK/SQj199SE+23pK5VTCVkiRtnOm7AISTmUC8FCQj8pphBA3Q6PR8Hq/9ky4u3T9g7+vPcz8TfJ4lpAibffW7k9FUSC0pQfNPa5d51sIYR80Gg1TIu9gUp+2AMyKTWZunEx4Ut9JkbZzZXN1D5ABY0LYPY1Gwwv33c7LkXcA8M/1x3jvZ5nruz6TIm3HTl3MZ/+5bBy0Gh4MlEvdQtQVE+9pY51C9OONJ4j5KVkKdT0lRdqOrfljGtCebbzwaqRTOY0Qojo9eedtvDmgI1C6etabPxyWQl0PSZG2U4qisHpf6ahumQZUiLppdI9W/N+gQDQaWLL9NFO/O4DZIoW6PpEibacOpeZwMiMfZ0ctkR291Y4jhKghj4b5MXtwJ7QaWL7zLBO/2k1BsVntWKKW2ESRnjdvHq1atUKv1xMWFkZiYuIN269cuZJ27dqh1+sJDAxk3bp15fYrisKMGTPw8fHBxcWFiIgIjh//c5Tk6dOnGTt2LP7+/ri4uNC6dWtmzpxJUVFRjfSvJpRNA9qnXVNc9U4qpxFC1KRHQlvwyYguODtoiT1kYszineQWFKsdS9QC1Yv0ihUriI6OZubMmezevZugoCAiIyNJT0+vsP327dsZPnw4Y8eOZc+ePURFRREVFcXBgwetbWbPns3cuXNZsGABCQkJNGzYkMjISAoKCgBITk7GYrGwcOFCDh06xPvvv8+CBQuYNm1arfT5VlksinVUt0wDKkT98ECAD0vGdKWhswPxJy/x6KcJXMwrVDuWqGEaReWRCGFhYXTt2pWPP/4YAIvFQosWLXjuued49dVXr2k/dOhQ8vPzWbt2rXVb9+7dCQ4OZsGCBSiKgq+vLy+++CIvvfQSANnZ2Xh7e7NkyRKGDRtWYY53332X+fPnc/LkyUrlzsnJwWAwkJ2djZubW1W7fUsST2UyZGE8rjpHdr4egd7JoVa/vxBCPQfOZTN6cSKZ+UXc5tWQz8d2kzkSbERN1AVVz6SLiopISkoiIiLCuk2r1RIREUF8fHyFx8THx5drDxAZGWltf+rUKUwmU7k2BoOBsLCw634mlBbyxo0bX3d/YWEhOTk55V5qKZsGNDLAKAVaiHomsLmBb58Op5m7Cycv5vO3+fEcS8tVO5aoIaoW6YsXL2I2m/H2Lj/wydvbG5PJVOExJpPphu3LvlblM0+cOMFHH33E+PHjr5s1JiYGg8FgfbVo0eLGnashxWYL6w5cAGRUtxD11W1NGvHthHDaNm2EKaeARxbEszvlstqxRA1Q/Z602s6fP88DDzzAI488wlNPPXXddlOnTiU7O9v6Onv2bC2m/NPW4xe5fKUYr0bO9GjtqUoGIYT6fAwufDM+nM5+7mRfLWbEpwlsPFrxWB5hv1Qt0l5eXjg4OJCWllZue1paGkajscJjjEbjDduXfa3MZ6ampnLPPffQo0cPFi1adMOsOp0ONze3ci81lA0Y6xfog6NDvf8dS4h6zaOhM189GcZdtzfharGZJ5fuYlmCrEldl6j6U97Z2ZmQkBDi4uKs2ywWC3FxcYSHh1d4THh4eLn2AOvXr7e29/f3x2g0lmuTk5NDQkJCuc88f/48d999NyEhISxevBit1vYL3tUiMz8fKr1kP0CWpRRCAA2cHfnXqFAGd2mO2aIw7fsDzIpNxiKTntQJjmoHiI6OZvTo0YSGhtKtWzc++OAD8vPzGTNmDACjRo2iWbNmxMTEADBp0iR69+7NnDlz6NevH8uXL2fXrl3WM2GNRsPkyZN5++23adu2Lf7+/kyfPh1fX1+ioqKAPwt0y5Ytee+998jIyLDmud4ZvC2IS07jSpGZ5h4udPFzVzuOEMJGODtqee+RTvg1bsD7vx5j/qbfOZt5hfceCZLBpXZO9SI9dOhQMjIymDFjBiaTieDgYGJjY60Dv1JSUsqd5fbo0YNly5bx+uuvM23aNNq2bcuqVasICAiwtpkyZQr5+fmMGzeOrKwsevXqRWxsLHq9Hig98z5x4gQnTpygefPm5fLY8ty4q/+Yq3tAkC8ajUblNEIIW6LRaJgU0ZbmHi68+t1+1u6/gCm7gE9HheLR0FnteOImqf6ctL2q7eeks68U0/WdXykyW4idfCftjOrcExdC2L7tJy4y/sskcgtKaOXZgH+NDqVNU1e1Y9V5de45aVF5sYcuUGS2cIe3qxRoIcQN9WjjxXcTetDcw4XTl64waN52Gfltp6RI24myUd0DZBpQIUQltPV2ZfXEnnRr1ZjcwhLGLtnJp1tO2vQtPXEtKdJ2ID2ngO2/XwJkAhMhROV5NtLx5ZNhDOvaAosC76w7wksr98sqWnZEirQdWLv/AooCnf3cadFY5ugVQlSes6OWmIcDeeOhDjhoNfxn9zmGf7qDtJwCtaOJSpAibQdWl614JWfRQoiboNFoeLynP0vGdMVN78ielCz6zf2N+D+u0AnbJUXaxp25lM++s1loNdCvkxRpIcTNu7NtE9Y824t2Rlcu5hXx2GcJLNz8u9yntmFSpG3cmj+eje7ZxosmrjqV0wgh7F0rr4Z8/0xPHu7cDLNFIeanZCZ8uZvcgmK1o4kKSJG2YYqiWC91PySXuoUQ1cTF2YE5Q4L4e1QATg4aYg+ZGPjxNpJN6i3BKyomRdqGHbmQy4n0PJwdtTwQYLvTlQoh7I9Go2Fk95Z8Mz4cH4OekxfzGfDxNr6IPy2Xv22IFGkbVvZs9D13NMFN76RyGiFEXdTZz4O1z/XinjuaUFRiYfrqQ4z/IomsK0VqRxNIkbZZFovCD2WjumXFKyFEDfJspOPfj3dlev8OODlo+OVwGn0//I2EkzL6W21SpG3U7pTLnM+6SiOdI/e2a6p2HCFEHafRaBjby5/vn+mJv1dDLmQXMPzTHcz55ShFJRa149VbUqRtVNmKV/d39Jal5oQQtSagmYG1z/VicJfmWBT4aMMJouZt48gFGVSmBinSNqjYbGHdgQuATAMqhKh9DXWOzBkSxMePdsajgROHL+Qw4OOtzNt4ghKznFXXJinSNmjbiYtcyi/Cs6EzPdt4qR1HCFFP9e/kyy8v9Oa+Dt4UmxXe/fkogxfEcyI9V+1o9YYUaRtUNqr7wUAfnBzkr0gIoZ4mrjoWjQzhn0OCcNU7su9sFg9+uJUPfj1GYYks1FHTpALYmIJiMz8fNAEwUJalFELYAI1Gw8NdmrP+hd7cfUcTiswWPvj1OH0/lPm/a5oUaRuzITmd/CIzzdxd6OLnoXYcIYSwMhr0LH68Kx8/2pkmrjpOZuQz/NMdvPjNPjLz5bnqmiBF2sas3nseKJ0GVKvVqJxGCCHK02g09O/ky6/RvRnZvSUaDfxn9znunbOJz+NPy8CyaiZF2obkFBSz8WgGIKO6hRC2zeDixN+jAvhuQg/aGV3JulLMjNWHeODD39h0NF3teHWGFGkb8vNBE0UlFto0bUR7H1e14wghxF8qm1b071EBeDRw4kR6Ho8v3snofydyPE1Ggd8qKdI2pGxU98AgXzQaudQthLAPjg5aRnZvyaaX7+GpO/1xctCw+VgGD3z4G6/+Zz/nLl9RO6LdkiJtIzJyC9l24iIgy1IKIeyTwcWJ1/p1YP0Lvbm/gzdmi8LynWe5571NvPb9AS5kX1U7ot2RIm0jftyfikWBoBbutPJqqHYcIYS4aa28GrJoVCj/mRBOrzZeFJsVvkpIoffsTbyx5hBpOQVqR7QbUqRtRNmlbhkwJoSoK0JaNubLJ8NYMa47Yf6NKTJbWLL9NL1mbeDFb/bJfOCVIEXaBpzNvMLulCw0Gniok4/acYQQolqF3ebJ8nHdWfZkGN1aNabYrPCf3efo++FvjPwsgc3HMlAURe2YNslR7QDiz7Po8Ns8aeqmVzmNEEJUP41GQ482XvRo48WelMv867dT/HTwAr8dv8hvxy/StmkjhnXzY1DnZjRu6Kx2XJuhUeTXl5uSk5ODwWAgOzsbNze3W/qsyPe3cDQtl1mDAxna1a+aEgohhG07m3mFf287xYqdZ7lSVDoPuLODlvs6ejM0tAW92njZ1aRO1VkXykiRvknV9ZeRbMrhgQ9+w8lBw67X7sPQwKkaUwohhO3LvlrMmr3nWb7zLIdS/7xP3czdhf6dfHggwEhwC3ebfzS1Joq0XO5W2Zq9pZe6776jqRRoIUS9ZHBxYmR4K0aGt+Lg+Wy+2XWWVXvOcz7rKgu3nGThlpP4GvREBhjpG+BDSEsPHOzoDPtW2MTAsXnz5tGqVSv0ej1hYWEkJibesP3KlStp164der2ewMBA1q1bV26/oijMmDEDHx8fXFxciIiI4Pjx4+XaZGZmMmLECNzc3HB3d2fs2LHk5eVVe99uRFEUGdUthBD/JaCZgbcGBpD4WgTzHu1C/04+NHR2IDW7gMXbTjNkYTxd/r6e8V/s4vP405xIz6vTg85Uv9y9YsUKRo0axYIFCwgLC+ODDz5g5cqVHD16lKZNm17Tfvv27dx1113ExMTQv39/li1bxqxZs9i9ezcBAQEAzJo1i5iYGJYuXYq/vz/Tp0/nwIEDHD58GL2+dGBW3759uXDhAgsXLqS4uJgxY8bQtWtXli1bVqnc1XFZI+nMZQbP304DZweSXr8PF2eHm/ocIYSoywqKzWw5lkHsQRPrj6SRW1BSbr+3m44wf08Cmxno2MyNjr4GDC61f2WyTt6TDgsLo2vXrnz88ccAWCwWWrRowXPPPcerr756TfuhQ4eSn5/P2rVrrdu6d+9OcHAwCxYsQFEUfH19efHFF3nppZcAyM7OxtvbmyVLljBs2DCOHDlChw4d2LlzJ6GhoQDExsby4IMPcu7cOXx9//qstjr+Mt5Yc4gl208TFezLB8M639RnCCFEfVJitrD/fDbxv19i24mL7DpzmaKSa1feaunZgA4+brT0bIhf4wbWl4+7HieHmrmIXOfuSRcVFZGUlMTUqVOt27RaLREREcTHx1d4THx8PNHR0eW2RUZGsmrVKgBOnTqFyWQiIiLCut9gMBAWFkZ8fDzDhg0jPj4ed3d3a4EGiIiIQKvVkpCQwKBBg6qxl9cX3tqTM5fyiercrFa+nxBC2DtHBy1d/Dzo4ufBxHvaUFBsJunMZfakXObg+RwOnM/mfNZVzly6wplL184ZrtWU3gMve7n98TK4OPH3gQE2d69b1SJ98eJFzGYz3t7e5bZ7e3uTnJxc4TEmk6nC9iaTybq/bNuN2vzvpXRHR0caN25sbfO/CgsLKSwstL7Pybn1mXIiOxqJ7Gi85c8RQoj6Su/kQM82XvRs42Xddjm/iEOpOSSbcjibeYWUzCucvXyVs5lXKCyxcPlKMZevFJf7HGdHLf83KLC24/8lGd1dSTExMbz55ptqxxBCCPEXPBo606utF73aepXbbrEoXMwvJOtKMdlXi8n+42tOQTHF5msvmdsCVYu0l5cXDg4OpKWllduelpaG0VjxGabRaLxh+7KvaWlp+Pj4lGsTHBxsbZOeXn5R8pKSEjIzM6/7fadOnVruMntOTg4tWrSoRC+FEELYAq1WQ1NXPU1d7WdmR1UfwXJ2diYkJIS4uDjrNovFQlxcHOHh4RUeEx4eXq49wPr1663t/f39MRqN5drk5OSQkJBgbRMeHk5WVhZJSUnWNhs2bMBisRAWFlbh99XpdLi5uZV7CSGEEDVKUdny5csVnU6nLFmyRDl8+LAybtw4xd3dXTGZTIqiKMrIkSOVV1991dp+27ZtiqOjo/Lee+8pR44cUWbOnKk4OTkpBw4csLb5xz/+obi7uyurV69W9u/frwwcOFDx9/dXrl69am3zwAMPKJ07d1YSEhKUrVu3Km3btlWGDx9e6dzZ2dkKoGRnZ1fDn4IQQgh7VxN1QfV70kOHDiUjI4MZM2ZgMpkIDg4mNjbWOvArJSUFrfbPE/4ePXqwbNkyXn/9daZNm0bbtm1ZtWqV9RlpgClTppCfn8+4cePIysqiV69exMbGWp+RBvjqq6949tln6dOnD1qtlsGDBzN37tza67gQQgjxF1R/Ttpe1cTzcEIIIexXTdQFm5gWVAghhBDXkiIthBBC2Cgp0kIIIYSNUn3gmL0qu5VfHTOPCSGEsH9l9aA6h3pJkb5Jubm5ADKhiRBCiHJyc3MxGAzV8lkyuvsmWSwWUlNTcXV1RaO5uQnZy2YtO3v2bJ0ZIS59sg/SJ/sgfbIPZX1KSUlBo9Hg6+tb7tHhWyFn0jdJq9XSvHnzavmsujiDmfTJPkif7IP0yT4YDIZq75MMHBNCCCFslBRpIYQQwkZJkVaRTqdj5syZ6HQ6taNUG+mTfZA+2Qfpk32oyT7JwDEhhBDCRsmZtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEiraN68ebRq1Qq9Xk9YWBiJiYlqR6q0mJgYunbtiqurK02bNiUqKoqjR4+Wa1NQUMDEiRPx9PSkUaNGDB48mLS0NJUSV80//vEPNBoNkydPtm6zx/6cP3+exx57DE9PT1xcXAgMDGTXrl3W/YqiMGPGDHx8fHBxcSEiIoLjx4+rmPjGzGYz06dPx9/fHxcXF1q3bs3f//73ctMw2nqftmzZwkMPPYSvry8ajYZVq1aV21+Z/JmZmYwYMQI3Nzfc3d0ZO3YseXl5tdiL8m7Up+LiYl555RUCAwNp2LAhvr6+jBo1itTU1HKfYU99+l9PP/00Go2GDz74oNz26uiTFGmVrFixgujoaGbOnMnu3bsJCgoiMjKS9PR0taNVyubNm5k4cSI7duxg/fr1FBcXc//995Ofn29t88ILL/DDDz+wcuVKNm/eTGpqKg8//LCKqStn586dLFy4kE6dOpXbbm/9uXz5Mj179sTJyYmffvqJw4cPM2fOHDw8PKxtZs+ezdy5c1mwYAEJCQk0bNiQyMhICgoKVEx+fbNmzWL+/Pl8/PHHHDlyhFmzZjF79mw++ugjaxtb71N+fj5BQUHMmzevwv2VyT9ixAgOHTrE+vXrWbt2LVu2bGHcuHG11YVr3KhPV65cYffu3UyfPp3du3fz3XffcfToUQYMGFCunT316b99//337NixA19f32v2VUufFKGKbt26KRMnTrS+N5vNiq+vrxITE6NiqpuXnp6uAMrmzZsVRVGUrKwsxcnJSVm5cqW1zZEjRxRAiY+PVyvmX8rNzVXatm2rrF+/Xundu7cyadIkRVHssz+vvPKK0qtXr+vut1gsitFoVN59913rtqysLEWn0ylff/11bUSssn79+ilPPPFEuW0PP/ywMmLECEVR7K9PgPL9999b31cm/+HDhxVA2blzp7XNTz/9pGg0GuX8+fO1lv16/rdPFUlMTFQA5cyZM4qi2G+fzp07pzRr1kw5ePCg0rJlS+X999+37quuPsmZtAqKiopISkoiIiLCuk2r1RIREUF8fLyKyW5ednY2AI0bNwYgKSmJ4uLicn1s164dfn5+Nt3HiRMn0q9fv3K5wT77s2bNGkJDQ3nkkUdo2rQpnTt35tNPP7XuP3XqFCaTqVyfDAYDYWFhNtunHj16EBcXx7FjxwDYt28fW7dupW/fvoB99um/VSZ/fHw87u7uhIaGWttERESg1WpJSEio9cw3Izs7G41Gg7u7O2CffbJYLIwcOZKXX36Zjh07XrO/uvokc3er4OLFi5jNZry9vctt9/b2Jjk5WaVUN89isTB58mR69uxJQEAAACaTCWdnZ+v/hGW8vb0xmUwqpPxry5cvZ/fu3ezcufOaffbYn5MnTzJ//nyio6OZNm0aO3fu5Pnnn8fZ2ZnRo0dbc1f079BW+/Tqq6+Sk5NDu3btcHBwwGw288477zBixAgAu+zTf6tMfpPJRNOmTcvtd3R0pHHjxnbRx4KCAl555RWGDx9unefaHvs0a9YsHB0def755yvcX119kiItbtnEiRM5ePAgW7duVTvKTTt79iyTJk1i/fr16PV6teNUC4vFQmhoKP/3f/8HQOfOnTl48CALFixg9OjRKqe7Od988w1fffUVy5Yto2PHjuzdu5fJkyfj6+trt32qT4qLixkyZAiKojB//ny149y0pKQkPvzwQ3bv3n3TqyBWllzuVoGXlxcODg7XjAxOS0vDaDSqlOrmPPvss6xdu5aNGzeWWxXMaDRSVFREVlZWufa22sekpCTS09Pp0qULjo6OODo6snnzZubOnYujoyPe3t521R8AHx8fOnToUG5b+/btSUlJAbDmtqd/hy+//DKvvvoqw4YNIzAwkJEjR/LCCy8QExMD2Gef/ltl8huNxmsGmJaUlJCZmWnTfSwr0GfOnGH9+vXlVouytz799ttvpKen4+fnZ/15cebMGV588UVatWoFVF+fpEirwNnZmZCQEOLi4qzbLBYLcXFxhIeHq5is8hRF4dlnn+X7779nw4YN+Pv7l9sfEhKCk5NTuT4ePXqUlJQUm+xjnz59OHDgAHv37rW+QkNDGTFihPW/7ak/AD179rzmsbhjx47RsmVLAPz9/TEajeX6lJOTQ0JCgs326cqVK9es0+vg4IDFYgHss0//rTL5w8PDycrKIikpydpmw4YNWCwWwsLCaj1zZZQV6OPHj/Prr7/i6elZbr+99WnkyJHs37+/3M8LX19fXn75ZX7++WegGvt08+PdxK1Yvny5otPplCVLliiHDx9Wxo0bp7i7uysmk0ntaJUyYcIExWAwKJs2bVIuXLhgfV25csXa5umnn1b8/PyUDRs2KLt27VLCw8OV8PBwFVNXzX+P7lYU++tPYmKi4ujoqLzzzjvK8ePHla+++kpp0KCB8uWXX1rb/OMf/1Dc3d2V1atXK/v371cGDhyo+Pv7K1evXlUx+fWNHj1aadasmbJ27Vrl1KlTynfffad4eXkpU6ZMsbax9T7l5uYqe/bsUfbs2aMAyj//+U9lz5491pHOlcn/wAMPKJ07d1YSEhKUrVu3Km3btlWGDx+uVpdu2KeioiJlwIABSvPmzZW9e/eW+3lRWFhol32qyP+O7laU6umTFGkVffTRR4qfn5/i7OysdOvWTdmxY4fakSoNqPC1ePFia5urV68qzzzzjOLh4aE0aNBAGTRokHLhwgX1QlfR/xZpe+zPDz/8oAQEBCg6nU5p166dsmjRonL7LRaLMn36dMXb21vR6XRKnz59lKNHj6qU9q/l5OQokyZNUvz8/BS9Xq/cdtttymuvvVbuh72t92njxo0V/r8zevRoRVEql//SpUvK8OHDlUaNGilubm7KmDFjlNzcXBV6U+pGfTp16tR1f15s3LjRLvtUkYqKdHX0SVbBEkIIIWyU3JMWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEVUZGBhMmTMDPzw+dTofRaCQyMpJt27YBoNFoWLVqlbohhahHZD1pIYTV4MGDKSoqYunSpdx2222kpaURFxfHpUuX1I4mRL0kc3cLIQDIysrCw8ODTZs20bt372v2t2rVijNnzljft2zZktOnTwOwevVq3nzzTQ4fPoyvry+jR4/mtddew9Gx9DxAo9HwySefsGbNGjZt2oSPjw+zZ8/mb3/7W630TQh7JZe7hRAANGrUiEaNGrFq1SoKCwuv2b9z504AFi9ezIULF6zvf/vtN0aNGsWkSZM4fPgwCxcuZMmSJbzzzjvljp8+fTqDBw9m3759jBgxgmHDhnHkyJGa75gQdkzOpIUQVv/5z3946qmnuHr1Kl26dKF3794MGzaMTp06AaVnxN9//z1RUVHWYyIiIujTpw9Tp061bvvyyy+ZMmUKqamp1uOefvpp5s+fb23TvXt3unTpwieffFI7nRPCDsmZtBDCavDgwaSmprJmzRoeeOABNm3aRJcuXViyZMl1j9m3bx9vvfWW9Uy8UaNGPPXUU1y4cIErV65Y24WHh5c7Ljw8XM6khfgLMnBMCFGOXq/nvvvu47777mP69Ok8+eSTzJw5k8cff7zC9nl5ebz55ps8/PDDFX6WEOLmyZm0EOKGOnToQH5+PgBOTk6YzeZy+7t06cLRo0dp06bNNS+t9s8fMTt27Ch33I4dO2jfvn3Nd0AIOyZn0kIIAC5dusQjjzzCE088QadOnXB1dWXXrl3Mnj2bgQMHAqUjvOPi4ujZsyc6nQ4PDw9mzJhB//798fPz429/+xtarZZ9+/Zx8OBB3n77bevnr1y5ktDQUHr16sVXX31FYmIin332mVrdFcIuyMAxIQQAhYWFvPHGG/zyyy/8/vvvFBcX06JFCx555BGmTZuGi4sLP/zwA9HR0Zw+fZpmzZpZH8H6+eefeeutt9izZw9OTk60a9eOJ598kqeeegooHTg2b948Vq1axZYtW/Dx8WHWrFkMGTJExR4LYfukSAshalxFo8KFEH9N7kkLIYQQNkqKtBBCCGGjZOCYEKLGyV01IW6OnEkLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNur/Adfp2YVinxzFAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Step\")\n",
    "plt.plot(range(total_training_steps), track_lrs)\n",
    "plt.tight_layout(); plt.savefig(\"2.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7512808-b48d-4146-86a1-5931b1e3aec1",
   "metadata": {},
   "source": [
    "## D.3 Gradient clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21",
   "metadata": {},
   "source": [
    "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n",
    "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n",
    "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n",
    "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n",
    "- Specifically, it's the L2 norm, also known as the Euclidean norm\n",
    "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n",
    "$$\n",
    "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44838a6-4322-47b2-a935-c00d3a88355f",
   "metadata": {},
   "source": [
    "- The L2 norm is calculated similarly for matrices.\n",
    "- Let's assume our gradient matrix is:\n",
    "$$\n",
    "G = \\begin{bmatrix}\n",
    "1 & 2 \\\\\n",
    "2 & 4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "- And we want to clip these gradients with a `max_norm` of 1.\n",
    "\n",
    "- First, we calculate the L2 norm of these gradients:\n",
    "$$\n",
    "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n",
    "$$\n",
    "\n",
    "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n",
    "\n",
    "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n",
    "$$\n",
    "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n",
    "\\frac{1}{5} & \\frac{2}{5} \\\\\n",
    "\\frac{2}{5} & \\frac{4}{5}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb0c3c1-2cff-46f5-8127-24412184428c",
   "metadata": {},
   "source": [
    "- Let's see this in action\n",
    "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e199e1ff-58c4-413a-855e-5edbe9292649",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import calc_loss_batch\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b60f3a-15ec-4846-838d-fdef3df99899",
   "metadata": {},
   "source": [
    "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n",
    "- Let's define a utility function to calculate the highest gradient based on all model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0411)\n"
     ]
    }
   ],
   "source": [
    "def find_highest_gradient(model):\n",
    "    max_grad = None\n",
    "    for param in model.parameters():\n",
    "        if param.grad is not None:\n",
    "            grad_values = param.grad.data.flatten()\n",
    "            max_grad_param = grad_values.max()\n",
    "            if max_grad is None or max_grad_param > max_grad:\n",
    "                max_grad = max_grad_param\n",
    "    return max_grad\n",
    "\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f",
   "metadata": {},
   "source": [
    "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0185)\n"
     ]
    }
   ],
   "source": [
    "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "print(find_highest_gradient(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62c2af0-dac3-4742-be4b-4292c6753099",
   "metadata": {},
   "source": [
    "## D.4 The modified training function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76715332-94ec-4185-922a-75cb420819d5",
   "metadata": {},
   "source": [
    "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "46eb9c84-a293-4016-a523-7ad726e171e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import evaluate_model, generate_and_print_sample\n",
    "\n",
    "\n",
    "def train_model(model, train_loader, val_loader, optimizer, device, n_epochs,\n",
    "                eval_freq, eval_iter, start_context, tokenizer, warmup_steps,\n",
    "                initial_lr=3e-05, min_lr=1e-6):\n",
    "\n",
    "    train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n",
    "    tokens_seen, global_step = 0, -1\n",
    "\n",
    "    # Retrieve the maximum learning rate from the optimizer\n",
    "    peak_lr = optimizer.defaults[\"lr\"]\n",
    "\n",
    "    # Calculate the total number of iterations in the training process\n",
    "    total_training_steps = len(train_loader) * n_epochs\n",
    "\n",
    "    # Calculate the learning rate increment during the warmup phase\n",
    "    lr_increment = (peak_lr - initial_lr) / warmup_steps\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        model.train()\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            global_step += 1\n",
    "\n",
    "            # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n",
    "            if global_step < warmup_steps:\n",
    "                # Linear warmup\n",
    "                lr = initial_lr + global_step * lr_increment  \n",
    "            else:\n",
    "                # Cosine annealing after warmup\n",
    "                progress = ((global_step - warmup_steps) / \n",
    "                            (total_training_steps - warmup_steps))\n",
    "                lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n",
    "\n",
    "            # Apply the calculated learning rate to the optimizer\n",
    "            for param_group in optimizer.param_groups:\n",
    "                param_group[\"lr\"] = lr\n",
    "            track_lrs.append(lr)  # Store the current learning rate\n",
    "\n",
    "            # Calculate and backpropagate the loss\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward()\n",
    "\n",
    "            # Apply gradient clipping after the warmup phase to avoid exploding gradients\n",
    "            if global_step > warmup_steps:\n",
    "                torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
    "            \n",
    "            optimizer.step()\n",
    "            tokens_seen += input_batch.numel()\n",
    "\n",
    "            # Periodically evaluate the model on the training and validation sets\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader,\n",
    "                    device, eval_iter\n",
    "                )\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                track_tokens_seen.append(tokens_seen)\n",
    "                # Print the current losses\n",
    "                print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        # Generate and print a sample from the model to monitor progress\n",
    "        generate_and_print_sample(\n",
    "            model, tokenizer, device, start_context\n",
    "        )\n",
    "\n",
    "    return train_losses, val_losses, track_tokens_seen, track_lrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Iter 000000): Train loss 10.934, Val loss 10.939\n",
      "Ep 1 (Iter 000005): Train loss 9.151, Val loss 9.461\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 2 (Iter 000010): Train loss 7.949, Val loss 8.184\n",
      "Ep 2 (Iter 000015): Train loss 6.362, Val loss 6.876\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,, the,,,,,,,,, the,,,,,,,,,,, the,,,,,,,,\n",
      "Ep 3 (Iter 000020): Train loss 5.851, Val loss 6.607\n",
      "Ep 3 (Iter 000025): Train loss 5.750, Val loss 6.634\n",
      "Every effort moves you. \"I\"I and I had to the to the to the and the of the to the of the to Gisburn, and the of the the of the of the to the to the of the of the of the to the of\n",
      "Ep 4 (Iter 000030): Train loss 5.225, Val loss 6.944\n",
      "Ep 4 (Iter 000035): Train loss 4.304, Val loss 6.512\n",
      "Every effort moves you know   \"--and--and--I                 \", and, and, and, and I had been, and, and \" it.   \n",
      "Ep 5 (Iter 000040): Train loss 3.736, Val loss 6.383\n",
      "Every effort moves you know the picture to have the picture--his--his, the donkey of a little: \"strong, with a little of the donkey, in the picture--as, with a little of his painting, the donkey, the donkey, with a little\n",
      "Ep 6 (Iter 000045): Train loss 2.395, Val loss 6.244\n",
      "Ep 6 (Iter 000050): Train loss 2.948, Val loss 6.279\n",
      "Every effort moves you?\"     I, and he had a little the in a flash that he was a little the fact, and in the picture. Gisburn's my unexpected discovery; and as I had the picture--the. He was his\n",
      "Ep 7 (Iter 000055): Train loss 2.316, Val loss 6.169\n",
      "Ep 7 (Iter 000060): Train loss 1.003, Val loss 6.343\n",
      "Every effort moves you?\"  \"Yes--I glanced after him, so inevitably the last word. Gisburn's past! The women had been his pictures I remember getting off a prodigious phrase about the honour being _mine_--because he didn't say\n",
      "Ep 8 (Iter 000065): Train loss 0.860, Val loss 6.348\n",
      "Ep 8 (Iter 000070): Train loss 1.117, Val loss 6.375\n",
      "Every effort moves you?\" \"I that my hostess was \"interesting\": on that point I could have given Miss Croft the fact, and Mrs. \"I must have Jack himself, I had again run over from Monte Carlo; and Mrs. Gis\n",
      "Ep 9 (Iter 000075): Train loss 0.367, Val loss 6.498\n",
      "Ep 9 (Iter 000080): Train loss 0.289, Val loss 6.612\n",
      "Every effort moves you?\" \" on--forming, as it were, so inevitably the background of the house.\"  \" went on groping and muddling; then I looked at the donkey again. I may be pardoned the bull--that I found\n",
      "Ep 10 (Iter 000085): Train loss 0.263, Val loss 6.700\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 11 (Iter 000090): Train loss 0.151, Val loss 6.788\n",
      "Ep 11 (Iter 000095): Train loss 0.097, Val loss 6.805\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 12 (Iter 000100): Train loss 0.081, Val loss 6.832\n",
      "Ep 12 (Iter 000105): Train loss 0.089, Val loss 6.900\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 13 (Iter 000110): Train loss 0.045, Val loss 6.911\n",
      "Ep 13 (Iter 000115): Train loss 0.047, Val loss 6.903\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 14 (Iter 000120): Train loss 0.038, Val loss 6.907\n",
      "Ep 14 (Iter 000125): Train loss 0.040, Val loss 6.912\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n",
      "Ep 15 (Iter 000130): Train loss 0.041, Val loss 6.915\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"  He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "\n",
    "peak_lr = 5e-4\n",
    "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "n_epochs = 15\n",
    "train_losses, val_losses, tokens_seen, lrs = train_model(\n",
    "    model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n",
    "    eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n",
    "    tokenizer=tokenizer, warmup_steps=warmup_steps, \n",
    "    initial_lr=1e-5, min_lr=1e-5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53",
   "metadata": {},
   "source": [
    "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n",
    "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n",
    "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n",
    "  - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n",
    "  - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7",
   "metadata": {},
   "source": [
    "- A quick check that the learning rate behaves as intended"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEmCAYAAACdy8LUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABUvUlEQVR4nO3deVxUVf8H8M8Mw8ywzQyIzIDKYu6IyiIjWppBYVJupUmkiCZm9mjZolbqr3p6XOspfcylTKzMrXIJNSM0V3YUFxAtcUMHBGSGfZk5vz+QqUlURme4M/B9v173RXPv9858D9p8veeeew6PMcZACCGEEKvC5zoBQgghhBiPCjghhBBihaiAE0IIIVaICjghhBBihaiAE0IIIVaICjghhBBihaiAE0IIIVaICjghhBBihQRcJ9Ca6XQ6XL9+HU5OTuDxeFynQwghhAOMMZSVlcHDwwN8vumum6mAm9H169fRqVMnrtMghBBiAa5evYqOHTua7P2ogJuRk5MTgIY/NIlEwnE2hBBCuKDRaNCpUyd9TTAVKuBm1NhtLpFIqIATQkgbZ+pbqTSIjRBCCLFCVMAJIYQQK0QFnBBCCLFCnBfwVatWwdvbG2KxGEqlEqmpqfeM3759O3r06AGxWAw/Pz/s3bvX4DhjDAsWLIC7uzvs7OwQFhaGCxcuGMR8/PHHGDhwIOzt7SGTyZr8nCtXriAiIgL29vZwc3PD22+/jfr6+odqKyGEEGIqnBbwrVu3Yvbs2Vi4cCEyMzPRt29fhIeHo7CwsMn448ePIzIyElOmTMGJEycwatQojBo1CmfOnNHHLF26FCtWrMCaNWuQkpICBwcHhIeHo7q6Wh9TW1uLsWPHYvr06U1+jlarRUREBGpra3H8+HFs3LgRcXFxWLBggWl/AYQQQsiDYhwKDg5mM2bM0L/WarXMw8ODLVq0qMn4cePGsYiICIN9SqWSTZs2jTHGmE6nYwqFgi1btkx/vLS0lIlEIrZ58+Y73m/Dhg1MKpXesX/v3r2Mz+czlUql37d69WomkUhYTU1Ns9unVqsZAKZWq5t9DiGEkNbFXLWAs8fIamtrkZGRgXnz5un38fl8hIWFISkpqclzkpKSMHv2bIN94eHh2LlzJwAgLy8PKpUKYWFh+uNSqRRKpRJJSUkYP358s3JLSkqCn58f5HK5wedMnz4dZ8+ehb+/f5Pn1dTUoKamRv9ao9E06/PIg/nfgQvYe1oFkS0fYoGN/qe90AbtJSLIncRwk4ggl4ghdxLDQyaGwIbzu0aEEGISnBXwoqIiaLVagyIJAHK5HOfOnWvyHJVK1WS8SqXSH2/cd7eY5rjb5/z9M5qyaNEifPDBB83+HPLgispr8GnCeehY888RCvh4pL0juskd0U3uhK5ujvDrKIW71M58iRJCiJnQRC4mNG/ePIMegsbZd4jp7Tujgo4BPRROmP1kN1TX61Bdp0VNnRblNVrcLKtBQVk1CjXVKNDUQKWpRm29Djk3NMi5Ydgz0kFmh/7ezgjydkF/bxd0dXMEn09z1xNCLBtnBdzV1RU2NjYoKCgw2F9QUACFQtHkOQqF4p7xjT8LCgrg7u5uENOvX79m56ZQKO4YDd/4uXfLDQBEIhFEIlGzP4c8uPis6wCA5wI64infu/+ZNNLpGK7dqsL5gjKcLyzDeVUZcgvKcb6gDPmlVcg/WYWdJxve09neFkO7uyG0pxyDu7nCSWxr1rYQQsiD4KyAC4VCBAYGIjExEaNGjQLQsHpXYmIiXnvttSbPCQkJQWJiIl5//XX9voSEBISEhAAAfHx8oFAokJiYqC/YGo0GKSkpdx1xfrfP+fjjj1FYWAg3Nzf950gkEvTq1cv4xhKTKtRUI/VSCQBgeB/3+0Q34PN58GxnD8929gjr9dftkYqaepy8WorUvBKkXy7BiSuluFVZh59O5OOnE/mwteFB6dMOYT3dMLyPO9ycxGZpEyGEGIvTLvTZs2cjOjoaQUFBCA4OxmeffYaKigrExMQAACZOnIgOHTpg0aJFAIBZs2ZhyJAh+OSTTxAREYEtW7YgPT0d69atA9Awz+zrr7+Of//73+jatSt8fHwwf/58eHh46P+RADQ8411SUoIrV65Aq9Xi5MmTAIAuXbrA0dERTz31FHr16oUJEyZg6dKlUKlUeP/99zFjxgy6wrYAe0/fAGNAgKcMHWQPd//aQSTAoC6uGNTFFQBQp9Uh4/ItJOYUIDGnEBeLKnD0jyIc/aMIH8Zn47Gu7TEmoAOe6qWAndDGFM0hhJAHY9Ix7Q9g5cqVzNPTkwmFQhYcHMySk5P1x4YMGcKio6MN4rdt28a6devGhEIh8/X1ZXv27DE4rtPp2Pz585lcLmcikYiFhoay3Nxcg5jo6GgG4I7t4MGD+phLly6xp59+mtnZ2TFXV1f25ptvsrq6OqPaRo+Rmcfzq48xrznxbP2Ri2b/rD8Ly9iXh/9ko1YdZV5z4vWb74Jf2JvbTrL0S8VMp9OZPQ9CiPUyVy3gMcaMGMdLjKHRaCCVSqFWq2k1MhO5oa5CyKID4PGApLmhUEhbrks7r6gCO07kY8eJa7haUqXf79dBikkDvfFMX3eIBHRVTggxZK5aQAXcjKiAm976o3n4KD4bwd4u2PZKCCc5MMaQfvkWtqZdxe6s66it1wEAXB2FeDHYEy8N8IKbhO6VE0IamKsW0KwWxKrEn2oYKR7RzMFr5sDj8dDf2wXLx/ZF8rxQvB3eHQqJGEXltVhx4A88uvQg5u88g/zSqvu/GSGEPCC6AjcjugI3rWu3KvHokoPg8YCUd0MtakR4nVaHX88W4Otjeci4fAsAYGvDw/OBnfDq44+gk4s9xxkSQrhCV+Ckzdt7+gYAQOnjYlHFGwBsbfiI6OOOH14JweapAxDSuR3qtAybU6/g8eW/450fsnCdrsgJISZEBZxYjT2nGgr4M308OM7k7ng8HkIeaYfNsQOwbVoIHuvqCq2OYVv6NQxd/juW/nIOmuo6rtMkhLQCVMCJVbhSXImsa2rwecCw3vefec0SBPu44NspSvw4fSCCfVxQU6/DF7//iceX/Y6Nxy+hTqvjOkVCiBWjAk6swp7b3ecDH3GFq6N1TaYT6OWMrbEDsG5CIDq3d0BJRS0W7j6Lp/57GAfPFXKdHiHESlEBJ1bBEkafPwwej4enfBXY//pgfDSqN1wdhcgrqkBMXBqmfZtOI9YJIUajAk4sXl5RBc5e18CGz8OwZixcYslsbfiYMMALB996HFMf84ENn4f9ZwsQ9skhrD30J3WrE0KajQo4sXh7bl99D+riCmcHIcfZmIaT2BbvRfTCnpmPIsjLGVV1Wizadw4RK44g/fZCLYQQci9UwInFi9ePPrfO7vN76aGQYNu0ECx7vg9cHIQ4X1COsWuT8OHP2aiq1XKdHiHEglEBJxbtj8IynFOVwdaGh/Be1t19fjd8Pg9jgzrhwJtDMDawIxgDvj6Wh+ErjiCNrsYJIXdBBZxYtMar78e6tofU3pbjbMxLZi/EsrF9sSGmPxQSMfKKKjCOrsYJIXdBBZxYtD2tuPv8boZ2d8P+NwZjXNBfV+MRK47g9DU116kRQiwIFXBisXJVZbhQWA6hDR9hveRcp9OipHa2WPr8X1fjF4sqMGb1Maw7/Cd0Olq+gBBCBZxYsMbR50O6t4dE3Lq7z+9maHc3/PL6Yxjmq0CdluE/e88hekMqCjXVXKdGCOEYFXBikRhjrXr0uTFk9kKsfikAi8b4QWzLx5ELRRj2+REk5hRwnRohhENUwIlFyrlRhotFFRAJ+Ajt2ba6z5vC4/EQGeyJ+H89hl7uEpRU1GLKxnQs2puDepr8hZA2iQo4sUiNU6cO7e4GR5GA42wsRxc3R+yYMRCTB/kAANYevogXv0pBYRl1qRPS1lABJxaHMaZfvOSZvm27+7wpIoENFjzbC19EBcBRJEBqXgkiVhxFah49M05IW0IFnFicM/kaXC6uhJ2tDZ7o4cZ1OhZruJ87dr02CN3kjrhZVoPIL5Px5eGLYIxGqRPSFlABJxYn/nRD9/kTPd1gL6Tu83t5pL0jds4YhFH9PKDVMXy8Nwf/2nyCJn4hpA2gAk4sCmPsr8lb/Kj7vDnshQL894V++GikLwR8HuJP3cDYtcdxnZYoJaRVowJOLErWNTWu3aqCvdAGQ6n7vNl4PB4mhHhj08tKuDgIcSZfgxH/O4aMy7e4To0QYiZUwIlFic9q6D4P6ymH2NaG42ysj7JzO+yaMQg9FE4oKq9B5Lpk/JBxjeu0CCFmQAWcWAyd7m+jz9v45C0Po5OLPX6cPhDhvnLUanV4a3sW/rM3h6ZgJaSVoQJOLMaJq7dwQ10NJ5EAg7u15zodq+YgEmB1VCBmhXYFAKw7fBEzvs9EdR0NbiOktaACTizGz1kNV99P9qLuc1Pg83l448lu+Hx8Pwht+Nh3RoXIL5NRXF7DdWqEEBOgAk4sgk7HsPd293kEdZ+b1Mh+HfDtlGBI7Wxx4kopRn9xHBdvlnOdFiHkIVEBJxYh7VIJCstqIBEL8FhX6j43NWXndvjp1YHo5GKHKyWVGLP6OM3cRoiVowJOLELj4LVwXwWEAvpraQ6PtHfEjlcHoV8nGUor6/DS+hT8ckbFdVqEkAfE+TflqlWr4O3tDbFYDKVSidTU1HvGb9++HT169IBYLIafnx/27t1rcJwxhgULFsDd3R12dnYICwvDhQsXDGJKSkoQFRUFiUQCmUyGKVOmoLzcsEtx//79GDBgAJycnNC+fXs899xzuHTpkknaTAxpdQx7TzcUEuo+Ny9XRxE2Tx2Ap3rJUVuvw6ubMrAl9QrXaRFCHgCnBXzr1q2YPXs2Fi5ciMzMTPTt2xfh4eEoLCxsMv748eOIjIzElClTcOLECYwaNQqjRo3CmTNn9DFLly7FihUrsGbNGqSkpMDBwQHh4eGorv5rtaaoqCicPXsWCQkJiI+Px+HDhxEbG6s/npeXh5EjR+KJJ57AyZMnsX//fhQVFWHMmDHm+2W0YSl5xSgqr4HM3haDurhynU6rZye0wRdRARjfvxN0DJj702n878AFmkOdEGvDOBQcHMxmzJihf63VapmHhwdbtGhRk/Hjxo1jERERBvuUSiWbNm0aY4wxnU7HFAoFW7Zsmf54aWkpE4lEbPPmzYwxxrKzsxkAlpaWpo/Zt28f4/F4LD8/nzHG2Pbt25lAIGBarVYfs3v3bsbj8VhtbW2z26dWqxkAplarm31OWzTvp1PMa048m/NDFteptCk6nY4t++Uc85oTz7zmxLOFu84wrVbHdVqEtDrmqgWcXYHX1tYiIyMDYWFh+n18Ph9hYWFISkpq8pykpCSDeAAIDw/Xx+fl5UGlUhnESKVSKJVKfUxSUhJkMhmCgoL0MWFhYeDz+UhJSQEABAYGgs/nY8OGDdBqtVCr1fj2228RFhYGW1tb0/wCCACgXqvT34d9po8Hx9m0LTweD2+Fd8f/PdsLABB3/BJe33oStfU6jjMjhDQHZwW8qKgIWq0WcrncYL9cLodK1fTAGpVKdc/4xp/3i3FzM5xjWyAQwMXFRR/j4+ODX3/9Fe+++y5EIhFkMhmuXbuGbdu23bNNNTU10Gg0Bhu5t6SLxSipqEU7ByEGdHbhOp02adIgH3w+vh8EfB52Z13Hq5syaMIXQqwA54PYLJFKpcLUqVMRHR2NtLQ0HDp0CEKhEM8///w97xMuWrQIUqlUv3Xq1KkFs7ZOjSuPDeutgMCG/jpyZWS/DvgqOggiAR+/5RRi6jfptCQpIRaOs29MV1dX2NjYoKCgwGB/QUEBFApFk+coFIp7xjf+vF/MPwfJ1dfXo6SkRB+zatUqSKVSLF26FP7+/hg8eDC+++47JCYm6rvZmzJv3jyo1Wr9dvXq1fv9Gtq0Oq0Ov5yl0eeW4vHuboiLCYa90AZHLhQh+utUlNfUc50WIeQuOCvgQqEQgYGBSExM1O/T6XRITExESEhIk+eEhIQYxANAQkKCPt7HxwcKhcIgRqPRICUlRR8TEhKC0tJSZGRk6GMOHDgAnU4HpVIJAKisrASfb/irsbGx0ed4NyKRCBKJxGAjd3fsjyKUVtbB1VEEpU87rtMhAEIeaYdvpyjhJBYg9VIJor5Kgbqyjuu0CCFNMemQOCNt2bKFiUQiFhcXx7Kzs1lsbCyTyWRMpVIxxhibMGECmzt3rj7+2LFjTCAQsOXLl7OcnBy2cOFCZmtry06fPq2PWbx4MZPJZGzXrl3s1KlTbOTIkczHx4dVVVXpY4YNG8b8/f1ZSkoKO3r0KOvatSuLjIzUH09MTGQ8Ho998MEH7Pz58ywjI4OFh4czLy8vVllZ2ez20Sj0e3tz20nmNSeezd95+v7BpEWdvlbK+n2wn3nNiWdPf3aYFZVVc50SIVbLXLWA0wLOGGMrV65knp6eTCgUsuDgYJacnKw/NmTIEBYdHW0Qv23bNtatWzcmFAqZr68v27Nnj8FxnU7H5s+fz+RyOROJRCw0NJTl5uYaxBQXF7PIyEjm6OjIJBIJi4mJYWVlZQYxmzdvZv7+/szBwYG1b9+ejRgxguXk5BjVNirgd1ddV896L/yFec2JZykXi7lOhzTh3A0NC/wogXnNiWdhn/zOCtRV9z+JEHIHc9UCHmM0e4O5aDQaSKVSqNVq6k7/h8ScAkzZmA65RISkuaHg83lcp0SacPFmOaK+SsENdTW829lj09QB6CCz4zotQqyKuWoBDfslnIi/Pfp8uJ87FW8L1rm9I7ZNC0EnFztcKq7EuDVJuFpSyXVahBBQASccqK7TIiG74UmBZ2j0ucXr5GKPbdNC0NnVAfmlVYj8MhnXblERJ4RrVMBJizt0/ibKa+rhIRXDv5Mz1+mQZnCX2mFz7AD4uDrg2q0qvPhlCq6XVnGdFiFtGhVw0uL2UPe5VZJLxNg8dQC82tnjSkklIr9Mhkpdff8TCSFmQQWctKiqWi1+y7ndfd6X5j63NgppQxHv5GKHy8UNRbxQQ0WcEC5QASct6vfcQlTWatHR2Q59O0q5Toc8AA+ZHTbfHo2eV1SB8V8mo7CMijghLY0KOGlRjaPPI/q4g8ej7nNr1dHZHltiG4r4xZsVePHLFBSV13CdFiFtChVw0mIqa+uReK6h+/xZWjrU6nVyscf3U5Vwl4rxR2E5XvwyGcVUxAlpMVTASYtJzClEdZ0OXu3s4etBE9u0Bl7tHPD91AGQS0Q4X9Aw6UtpZS3XaRHSJlABJy2mcfT5M9R93qr4uDYU8fZOIpxTlSF6QxqtYkZIC6ACTlpEeU09DuY2LOMa4Ufd563NI+0dsellJWT2tsi6WoqpG9NRXUfriRNiTg9UwOvr6/Hbb79h7dq1KCsrAwBcv34d5eXlJk2OtB6JOQWoqdehc3sH9HR34jodYgbd5E74ZnIwHEUCJF0sxqubMlGnvfvyu4SQh2N0Ab98+TL8/PwwcuRIzJgxAzdv3gQALFmyBG+99ZbJEyStw89Zt7vP/aj7vDXr01GG9dFBEAn4OHCuEG9sPQmtjtZLIsQcjC7gs2bNQlBQEG7dugU7u79WJRo9ejQSExNNmhxpHdRVdTh8vuEfejR5S+un7NwOayYEwtaGh/hTN/DejtOgRQ8JMT2jC/iRI0fw/vvvQygUGuz39vZGfn6+yRIjrcdv2QWo1erQ1c0R3eTUfd4WDO3uhs9e8AefB2xJu4p/78mhIk6IiRldwHU6HbTaOwenXLt2DU5O9OVM7hR/6joA4Bl69rtNiejjjsXP9QEArD+ah88TL3CcESGti9EF/KmnnsJnn32mf83j8VBeXo6FCxdi+PDhpsyNtALqyjocuVAEoOELnbQt44I6YeGzvQAAn/12AV8duchxRoS0HgJjT/jkk08QHh6OXr16obq6Gi+++CIuXLgAV1dXbN682Rw5Eiu2/6wK9TqGHgondHFz5DodwoGYQT4or67HJwnn8e89OXC2F+K5wI5cp0WI1TO6gHfs2BFZWVnYunUrsrKyUF5ejilTpiAqKspgUBshABB/+q/JW0jb9doTXaCuqsNXR/Pwzo+n4Oxgiyd6yLlOixCrxmNGjiw5fPgwBg4cCIHAsPbX19fj+PHjGDx4sEkTtGYajQZSqRRqtRoSSdubOrSkohb9P/4NWh3Dwbceh4+rA9cpEQ7pdAxvbs/CjhP5ENvysellJQK9XLhOixCzM1ctMPoe+NChQ1FSUnLHfrVajaFDh5okKdI67D+rglbH4OshoeJNwOfzsPT5Pni8e3tU1+kwOS4d5wvKuE6LEKtldAFnjDU5EUdxcTEcHOhLmvyFRp+Tf7K14eOLqAD4e8qgrqrDxPWpyC+t4jotQqxSs++BjxkzBkDDqPNJkyZBJBLpj2m1Wpw6dQoDBw40fYbEKhWV1yDpz2IAQIQf3f8mf7EXCvB1dH+MXZuEPwrLMWF9Cn54ZSBcHIT3P5kQotfsK3CpVAqpVArGGJycnPSvpVIpFAoFYmNj8d1335kzV2JF9p1RQceAvh2l8Gxnz3U6xMI4OwjxzeRguEvFuHizAjFxaaigFcwIMUqzr8A3bNgAoGHGtbfeeou6y8k97aHuc3IfHjI7fDslGM+vSULW1VJM35SJryYGQSigRRIJaQ6j/09ZuHAhFW9yT4WaaqTkNQx0HE6Pj5F76OLmhK8n9YedrQ0On7+Jt3/Igo4WPyGkWYx+DhwAfvjhB2zbtg1XrlxBbW2twbHMzEyTJEas174zKjAGBHjK0EFGcwOQewvwdMbqlwLw8sZ07Dp5HQqpGPOe7sl1WoRYPKOvwFesWIGYmBjI5XKcOHECwcHBaNeuHS5evIinn37aHDkSK9M4+jyCus9JMz3e3U0/b/raQxex8fglbhMixAoYXcC/+OILrFu3DitXroRQKMQ777yDhIQEzJw5E2q12hw5EiuiUlcj7dItADT6nBjn+cCOePPJbgCA//v5LH45o+I4I0Ism9EF/MqVK/rHxezs7FBW1jARw4QJE2gudII9t6dO7e/tDIVUzHE2xNq89kQXRAZ7gjFg1pYTyLh856RRhJAGRhdwhUKhn4nN09MTycnJAIC8vDxa75fQ5C3kofB4PHw00hehPdxQU6/DlI3p+PNmOddpEWKRjC7gTzzxBHbv3g0AiImJwRtvvIEnn3wSL7zwAkaPHm3yBIn1uHarEieulILHA57ureA6HWKlBDZ8rHzRH307SlFaWYfor1NRWFbNdVqEWByjC/i6devw3nvvAQBmzJiBr7/+Gj179sSHH36I1atXG53AqlWr4O3tDbFYDKVSidTU1HvGb9++HT169IBYLIafnx/27t1rcJwxhgULFsDd3R12dnYICwvDhQsXDGJKSkoQFRUFiUQCmUyGKVOmoLy8/I73Wb58Obp16waRSIQOHTrg448/Nrp9bcne293nSh8XuEmo+5w8OHuhAOsn9YdXO3tcu1WFKXHpNNELIf/EjFBXV8c++OADdvXqVWNOu6stW7YwoVDIvv76a3b27Fk2depUJpPJWEFBQZPxx44dYzY2Nmzp0qUsOzubvf/++8zW1padPn1aH7N48WImlUrZzp07WVZWFhsxYgTz8fFhVVVV+phhw4axvn37suTkZHbkyBHWpUsXFhkZafBZ//rXv1j37t3Zrl272MWLF1l6ejr79ddfjWqfWq1mAJharTbqPGs1YuUR5jUnnn2TdInrVEgrkXeznPl/+CvzmhPPJq5PYbX1Wq5TIsRo5qoFRhVwxhhzcHBgeXl5Jvnw4OBgNmPGDP1rrVbLPDw82KJFi5qMHzduHIuIiDDYp1Qq2bRp0xhjjOl0OqZQKNiyZcv0x0tLS5lIJGKbN29mjDGWnZ3NALC0tDR9zL59+xiPx2P5+fn6GIFAwM6dO/dQ7WtLBfxyUQXzmhPPfObGs5tl1VynQ1qRzMslrPv7e5nXnHj29vaTTKfTcZ0SIUYxVy0wugs9NDQUhw4deugr/9raWmRkZCAsLEy/j8/nIywsDElJSU2ek5SUZBAPAOHh4fr4vLw8qFQqgxipVAqlUqmPSUpKgkwmQ1BQkD4mLCwMfD4fKSkpAICff/4ZnTt3Rnx8PHx8fODt7Y2XX365yWVU/66mpgYajcZgaysaR5+HPNIOro6i+0QT0nz+ns5YGRkAPg/Yln4NnydeuP9JhLQBRs/E9vTTT2Pu3Lk4ffo0AgMD75hWdcSIEc16n6KiImi1WsjlcoP9crkc586da/IclUrVZLxKpdIfb9x3rxg3NzeD4wKBAC4uLvqYixcv4vLly9i+fTu++eYbaLVavPHGG3j++edx4MCBu7Zp0aJF+OCDD+7X9FaJRp8Tc3qylxwfjuyN93eewWe/XYCH1A7j+nfiOi1COGV0AX/11VcBAJ9++ukdx3g8HrRa7cNnxTGdToeamhp888036NatYWKJ9evXIzAwELm5uejevXuT582bNw+zZ8/Wv9ZoNOjUqfV/yeQVVeDsdQ1s+DyE+9Loc2IeLw3wwg11FVYd/BPzdpyGm0SEx7u73f9EQlopo7vQdTrdXTdjirerqytsbGxQUFBgsL+goAAKRdNFQKFQ3DO+8ef9YgoLCw2O19fXo6SkRB/j7u4OgUCgL94A0LNnw9zMV65cuWubRCIRJBKJwdYWNK48NqiLK63pTMzqrae6Y4x/B2h1DDM2ZSL7etu5TUXIP3G2bp9QKERgYCASExP1+3Q6HRITExESEtLkOSEhIQbxAJCQkKCP9/HxgUKhMIjRaDRISUnRx4SEhKC0tBQZGRn6mAMHDkCn00GpVAIABg0ahPr6evz555/6mPPnzwMAvLy8HqbZrVL8qYb738/Q1KnEzHg8HhY/1wchnduholaLyXFpUKnpGXHSRpl0SJyRtmzZwkQiEYuLi2PZ2dksNjaWyWQyplKpGGOMTZgwgc2dO1cff+zYMSYQCNjy5ctZTk4OW7hwYZOPkclkMrZr1y526tQpNnLkyCYfI/P392cpKSns6NGjrGvXrgaPkWm1WhYQEMAGDx7MMjMzWXp6OlMqlezJJ580qn1tYRT6hYIy5jUnnnV5dw8rrajlOh3SRpRW1LInlh9kXnPi2bDPDrOy6jquUyLkrizmMTJTW7lyJfP09GRCoZAFBwez5ORk/bEhQ4aw6Ohog/ht27axbt26MaFQyHx9fdmePXsMjut0OjZ//nwml8uZSCRioaGhLDc31yCmuLiYRUZGMkdHRyaRSFhMTAwrKysziMnPz2djxoxhjo6OTC6Xs0mTJrHi4mKj2tYWCvhnCeeZ15x4FrMhletUSBtzpbiCBX7U8Ix49NcprI6eEScWyly1gMcYTWBuLhqNBlKpFGq1utXeD3/y00O4UFiOT8b2xXOBHblOh7QxJ6+WYvy6JFTX6RCl9MS/R/UGj8fjOi1CDJirFnB2D5xYv/MFZbhQWA6hDR9P+srvfwIhJtavkwyfveAPHg/YlHIFXx65yHVKhLQYowv4PycqadzKyspQW1trjhyJhYrPahh9Prhbe0jEthxnQ9qqYb0VeG94w1Mi/9l7Tj8nPyGtndEFXCaTwdnZ+Y5NJpPBzs4OXl5eWLhwIXQ6nTnyJRaCMaYfff5sXxp9Trg15VEfRIc0PCHyxtaTyLh8i+OMCDE/owt4XFwcPDw88O6772Lnzp3YuXMn3n33XXTo0AGrV69GbGwsVqxYgcWLF5sjX2Ihcm6U4WJRBUQCPkJ7Uvc54RaPx8OCZ/9aR3zqN+m4XFzBdVqEmJXRM7Ft3LgRn3zyCcaNG6ff9+yzz8LPzw9r165FYmIiPD098fHHH+Pdd981abLEcjROnTq0uxscRUb/NSLE5Gz4PKyI9McL65JwJl+DmA1p+OnVgZDZ0+RCpHUy+gr8+PHj8Pf3v2O/v7+/fsGQRx999J4zlhHrxhjTL14S0Ye6z4nlcBAJ8HV0f3hIxbhYVIHYbzJQU2/90zsT0hSjC3inTp2wfv36O/avX79eP+93cXExnJ2dHz47YpHO5GtwubgSYls+QnvSXNTEsrhJxPg6pj+cRAKkXirBOz+cAj0tS1ojo/s+ly9fjrFjx2Lfvn3o378/ACA9PR3nzp3DDz/8AABIS0vDCy+8YNpMicWIP93QfR7aQw57IXWfE8vTQyHBFy8FIGZDGnadvA5PF3u8+VTTixARYq0eaCKXvLw8rF27Vj8/ePfu3TFt2jR4e3ubOj+r1honcmGM4bGlB3HtVhVWRwXgaZr/nFiwrWlXMOfH0wCApc/3wbig1r86ILE85qoFD3T55OPjQ6PM26isa2pcu1UFe6ENLeVILN4L/T1xpaQSqw7+iXd/Og0PqR0e7erKdVqEmMQDFfDS0lKkpqaisLDwjue9J06caJLEiGVqnLwlrKccdkIbjrMh5P7efLI7rpZUYXfWdUz/LgM/TB+I7gonrtMi5KEZXcB//vlnREVFoby8HBKJxGDeYR6PRwW8FdPpmH6WKxp9TqwFn8/DsrF9cENdhbRLtzA5Lg07Xh0IN4mY69QIeShGj0J/8803MXnyZJSXl6O0tBS3bt3SbyUlJebIkViIE1dv4bq6Go4iAYZ0a891OoQ0m0hgg3UTguDj6oD80ipM2ZiOytp6rtMi5KEYXcDz8/Mxc+ZM2NvbmyMfYsEap059spccYlvqPifWxdlBiA2T+sPFQYjT+WrM3HwCWh09Xkasl9EFPDw8HOnp6ebIhViwv3efP0Pd58RKebs64MuJgRAK+PgtpxAfxWdznRIhD8zoe+ARERF4++23kZ2dDT8/P9jaGq5CNWLECJMlRyxH+uVbKNDUwEkswGNdqfucWK9ALxf8d1w/zPg+E3HHL8HTxR6TH/XhOi1CjGZ0AZ86dSoA4MMPP7zjGI/Hg1ZL0xa2Ro1zn4f7KiAU0DLyxLpF9HHHtVs9sGjfOXy0JxsdnO0Q7qvgOi1CjGL0N7FOp7vrRsW7ddLqGPaeVgGg7nPSesQO7owopScYA2ZtOYGTV0u5TokQo9ClFLmvlLxiFJXXQGZvi0FdaBIM0jrweDx8MMIXj3dvj+o6HV7emIarJZVcp0VIszWrC33FihWIjY2FWCzGihUr7hk7c+ZMkyRGLEfj6PNhvgrY2tC/+UjrIbDh438vBmDcmiRk39Bg0oZU/DR9EKT2tvc/mRCONWsudB8fH6Snp6Ndu3bw8bn7YA8ej4eLFy+aNEFr1hrmQq/X6hD8n0SUVNTi2ynBNICNtEoqdTVGrToGlaYaAzq7YOPkYIgE9KgkMQ1O50LPy8tr8r9J65d0sRglFbVwcRAipHM7rtMhxCwUUjE2xPTH2DVJSL5Ygnk/nsYn4/oazDRJiKWh/lByT3sau897KyCg7nPSivV0l2BVVABs+Dz8dCIfn/12geuUCLknox8j02q1iIuLQ2JiYpOLmRw4cMBkyRFu1Wl1+OUsjT4nbceQbu3x71G9Me+n0/g88QI6udjj+cCOXKdFSJOMLuCzZs1CXFwcIiIi0Lt3b+piasWO/VGE0so6uDqKoPSh7nPSNkQGe+JqSSW++P1PzP3xFNylYnr6glgkowv4li1bsG3bNgwfPtwc+RAL0jj6fLifAjZ8+ocaaTveeqo7rt6qws9Z1/HKdxn4cfpAdJPTEqTEshh9U1MoFKJLly7myIVYkNp6Hfbf7j6P8KPuc9K28Pk8LHu+D4K8nFFWXY+YDWkoLKvmOi1CDDzQcqKff/45mvH0GbFiRy7cRFl1PdycROjv7cJ1OoS0OLGtDdZN/NsSpHG0BCmxLEZ3oR89ehQHDx7Evn374Ovre8diJj/99JPJkiPc2aPvPncHn7rPSRvlcnsJ0tFfHLu9BOlJrJ0QSLeUiEUw+gpcJpNh9OjRGDJkCFxdXSGVSg02Yv2q67T4NbsAAPBsX+o+J22bt6sDvooOur0EaQEtQUoshlFX4PX19Rg6dCieeuopKBS0ck9rdfj8TZTX1MNdKoZ/J2eu0yGEc7QEKbFERl2BCwQCvPLKK6ipqTFpEqtWrYK3tzfEYjGUSiVSU1PvGb99+3b06NEDYrEYfn5+2Lt3r8FxxhgWLFgAd3d32NnZISwsDBcuGE7KUFJSgqioKEgkEshkMkyZMgXl5eVNft4ff/wBJycnyGSyh2qntWgcfR5B3eeE6EX0cce8p3sAAD7ak60f5EkIV4zuQg8ODsaJEydMlsDWrVsxe/ZsLFy4EJmZmejbty/Cw8NRWFjYZPzx48cRGRmJKVOm4MSJExg1ahRGjRqFM2fO6GOWLl2KFStWYM2aNUhJSYGDgwPCw8NRXf3XKNKoqCicPXsWCQkJiI+Px+HDhxEbG3vH59XV1SEyMhKPPfaYydpsyapqtfgtp6H7/Jm+HhxnQ4hloSVIiUVhRtq6dSvr3LkzW7lyJTt+/DjLysoy2IwVHBzMZsyYoX+t1WqZh4cHW7RoUZPx48aNYxEREQb7lEolmzZtGmOMMZ1OxxQKBVu2bJn+eGlpKROJRGzz5s2MMcays7MZAJaWlqaP2bdvH+PxeCw/P9/gvd955x320ksvsQ0bNjCpVGpU29RqNQPA1Gq1Uedxae+p68xrTjwbtDiR6XQ6rtMhxOLU1WtZ9NcpzGtOPAv86Fd2pbiC65SIhTNXLTD6Cnz8+PHIy8vDzJkzMWjQIPTr1w/+/v76n8aora1FRkYGwsLC9Pv4fD7CwsKQlJTU5DlJSUkG8QAQHh6uj8/Ly4NKpTKIkUqlUCqV+pikpCTIZDIEBQXpY8LCwsDn85GSkqLfd+DAAWzfvh2rVq1qVntqamqg0WgMNmuj7z7v406z7BHShMYlSHu5S1BUXovor1NRUlHLdVqkDTL6MTJTrkZWVFQErVYLuVxusF8ul+PcuXNNnqNSqZqMV6lU+uON++4V4+bmZnBcIBDAxcVFH1NcXIxJkybhu+++a/byb4sWLcIHH3zQrFhLVFlbj8Rzt7vP/aj7nJC7cRQJsCGmP8Z8cRwXiyowZWMavn95AOyEtAQpaTlGX4F7eXndc2stpk6dihdffBGDBw9u9jnz5s2DWq3Wb1evXjVjhqaXmFOI6jodvNrZo3cH61y/nJCWIpeIsXFyf0jtbHHiSin+tfkE6rW6+59IiIkYfQXeKDs7G1euXEFtrWHX0YgRI5r9Hq6urrCxsUFBQYHB/oKCgrs+pqZQKO4Z3/izoKAA7u7uBjH9+vXTx/xzkFx9fT1KSkr05x84cAC7d+/G8uXLATSMbNfpdBAIBFi3bh0mT558R24ikQgikai5zbc4e/42+py6zwm5vy5uTvgqOghRX6Xgt5wCzN91Fv8ZTYs8kZZh9BX4xYsX0bdvX/Tu3RsRERH6UeCjR4/G6NGjjXovoVCIwMBAJCYm6vfpdDokJiYiJCSkyXNCQkIM4gEgISFBH+/j4wOFQmEQo9FokJKSoo8JCQlBaWkpMjIy9DEHDhyATqeDUqkE0HCf/OTJk/rtww8/hJOTE06ePGl0O61BeU09DuY2/KPmmT7UfU5Ic/X3dsGK8f3A4wGbU6/gfwf+4Dol0lYYO+rtmWeeYSNHjmQ3b95kjo6OLDs7mx05coQFBwezw4cPGz2KbsuWLUwkErG4uDiWnZ3NYmNjmUwmYyqVijHG2IQJE9jcuXP18ceOHWMCgYAtX76c5eTksIULFzJbW1t2+vRpfczixYuZTCZju3btYqdOnWIjR45kPj4+rKqqSh8zbNgw5u/vz1JSUtjRo0dZ165dWWRk5F3zbO2j0HeeuMa85sSzocsO0uhzQh7AxuN5zGtOPPOaE8+2pl7hOh1iQcxVC4wu4O3atdM/LiaRSNi5c+cYY4wlJiayfv36PVASK1euZJ6enkwoFLLg4GCWnJysPzZkyBAWHR1tEL9t2zbWrVs3JhQKma+vL9uzZ4/BcZ1Ox+bPn8/kcjkTiUQsNDSU5ebmGsQUFxezyMhI5ujoyCQSCYuJiWFlZWV3zbG1F/ApcWnMa048+2T/Oa5TIcRqLdmXw7zmxLPO8/awAzkFXKdDLIS5agGPMeOWFXN2dkZmZiZ8fHzwyCOP4KuvvsLQoUPx559/ws/PD5WVleboKLBKGo0GUqkUarW62SPZuaCprkPQR7+hVqvD/tcHo7uC1j0m5EEwxvDm9iz8lJkPO1sbbIkdgL6dZFynRThmrlpg9D3w3r17IysrCwCgVCqxdOlSHDt2DB9++CE6d+5sssRIy0k4W4BarQ5d3RypeBPyEHg8HpY81wePdXVFVZ0Wk+PScKmoguu0SCtldAF///33odM1PCrx4YcfIi8vD4899hj27t2LFStWmDxBYn57Tv81eQsh5OHY2vCx+qVA+HpIUFxRi+gNqSgqN+36EYQAgNFd6E0pKSmBs7MzPTrxD9bQha6urEPQxwmo0zL8NnswurjRFTghplBYVo3nVh/H1ZIq9OkoxeapA+AgeuAnd4kVs5gu9EZ//PEH9u/fj6qqKri4uJgsIdKy9merUKdl6KFwouJNiAm5OYmxMSYYzva2OHVNjVc3ZaKOJnohJmR0AS8uLkZoaCi6deuG4cOH48aNhu7XKVOm4M033zR5gsS8Guc+f4a6zwkxuc7tHbF+Un+Ibfk4dP4m3t6eBZ3uoTs9CQHwAAX8jTfegK2tLa5cuQJ7e3v9/hdeeAG//PKLSZMj5lVSUYtjfxQBACJo8hZCzCLA0xmrowIh4POw8+R1fBifDRPcuSTE+AL+66+/YsmSJejYsaPB/q5du+Ly5csmS4yY3/6zKmh1DL4eEvi4OnCdDiGt1tAeblg+ti8AIO74Jayk2dqICRhdwCsqKgyuvBuVlJRY9TzgbVH8qesAaOpUQlrCKP8OWPhsLwDApwnn8W3SJW4TIlbP6AL+2GOP4ZtvvtG/5vF40Ol0WLp0KYYOHWrS5Ij5FJXXIOnPYgANi5cQQswvZpAPZoZ2BQAs2H0Wu7Ouc5wRsWZGP9OwdOlShIaGIj09HbW1tXjnnXdw9uxZlJSU4NixY+bIkZjBvjMq6BjQt6MUnu3u7FEhhJjHG2FdcauiFt8mX8bsrSchEQvweHc3rtMiVuiBZmI7f/48Hn30UYwcORIVFRUYM2YMTpw4gUceecQcORIz2HO7+5wmbyGkZfF4PHwwwhfP9vVAvY5h+neZyLh8i+u0iBV6oFkFpFIp3nvvPYN9165dQ2xsLNatW2eSxIj5FGqqkZJXAgAYTt3nhLQ4Pp+HT8b2hbqqDofP38TkuDRsfyUE3eQ0FwNpvgeeyOWfiouLsX79elO9HTGjfWdUYAzw95ShozN1nxPCBaGAjzUvBcDfUwZ1VR1e+ioFl4tp3nTSfCYr4MR60OhzQiyDvVCADZP6o7vcCYVlNXjxyxTkl1ZxnRaxElTA2xiVuhpplxrutw33U3CcDSFEZi/Ety8Ho7OrA/JLq/DSVykoLKvmOi1iBaiAtzGNK4/193aGu9SO42wIIUDDvOnfvaxEB5kd8ooq8NJXKSipqOU6LWLhmj2IbcyYMfc8Xlpa+rC5kBagH31Og9cIsSgeMjtsnjoAY9cex/mCckz8OgWbXh4AqZ0t16kRC9XsK3CpVHrPzcvLCxMnTjRnruQh5ZdWIfNKKXg8Gn1OiCXybGePTS8PQDsHIc7kaxCzIRUVNfVcp0UsVLOvwDds2GDOPEgL2Ht75bFgbxe4ScQcZ0MIaUoXN0d8O0WJyC+TkXmlFC9vTMeGmP4Q29pwnRqxMHQPvA3Rjz7vS6PPCbFkvTwk2Dg5GA5CGyRdLMb07zJQW09riRNDVMDbiKsllci6pgafBwzzpdHnhFi6fp1k+Pr2WuIHc29i5uYTqNNSESd/oQLeRsTf7j4PeaQd2jvRqnGEWANl53ZYNyEIQhs+fjmrwr++pyJO/kIFvI2I148+p+5zQqzJ4G7tsXZioL6Iv/Z9JhVxAoAKeJuQV1SBs9c1sOHzMKw3dZ8TYm2GdndrKOICPvafLcBr32fSPXFCBbwtaHz2e+Aj7eDiIOQ4G0LIgxja3Q3rJvxVxP+1mYp4W0cFvA1ovP/9LM19TohVe/wfRZyuxNs2KuCt3B+F5TinKoOtDQ/hNPqcEKv3eHc3fDkxCEIBH79mF2AGFfE2iwp4K7fn9tX3o11cIbWnKRkJaQ2GdGuvL+IJVMTbLCrgrRwtHUpI6/TPIj79uwxU12m5Tou0ICrgrdj5gjJcKCyH0IaPJ33lXKdDCDGxId3a46uJQRAJ+Eg8V4gpG9No7vQ2hAp4Kxaf1XD1Pbhbe0jE1H1OSGs0uFt7/bSrx/4oxoT1KVBX1XGdFmkBVMBbKcYY4m+v/f1MH1p5jJDWbEDndvjuZSUkYgEyr5TixS+TUVxew3VaxMwsooCvWrUK3t7eEIvFUCqVSE1NvWf89u3b0aNHD4jFYvj5+WHv3r0GxxljWLBgAdzd3WFnZ4ewsDBcuHDBIKakpARRUVGQSCSQyWSYMmUKysvL9cd///13jBw5Eu7u7nBwcEC/fv2wadMm0zXazHJulOHizQoIBXyE9aLuc0JaO39PZ2yJDUE7ByHOXtfghXXJKNBUc50WMSPOC/jWrVsxe/ZsLFy4EJmZmejbty/Cw8NRWFjYZPzx48cRGRmJKVOm4MSJExg1ahRGjRqFM2fO6GOWLl2KFStWYM2aNUhJSYGDgwPCw8NRXf3XX+aoqCicPXsWCQkJiI+Px+HDhxEbG2vwOX369MGPP/6IU6dOISYmBhMnTkR8fLz5fhkmtOd0Q/f50O7t4Shq9qqxhBAr1stDgm2vhEAhEeOPwnI8v+Y4LhVVcJ0WMRfGseDgYDZjxgz9a61Wyzw8PNiiRYuajB83bhyLiIgw2KdUKtm0adMYY4zpdDqmUCjYsmXL9MdLS0uZSCRimzdvZowxlp2dzQCwtLQ0fcy+ffsYj8dj+fn5d811+PDhLCYmptltU6vVDABTq9XNPscUdDodG7z0APOaE892n7x7ewghrdOV4gr22JKG74DAj35lp6+Vcp1Sm2auWsDpFXhtbS0yMjIQFham38fn8xEWFoakpKQmz0lKSjKIB4Dw8HB9fF5eHlQqlUGMVCqFUqnUxyQlJUEmkyEoKEgfExYWBj6fj5SUlLvmq1ar4eLictfjNTU10Gg0BhsXzl7X4HJxJcS2fDzRw42THAgh3OnkYo8fpoegp7sEReW1GL8uGcf/LOI6LWJinBbwoqIiaLVayOWG92jlcjlUKlWT56hUqnvGN/68X4ybm2FhEwgEcHFxuevnbtu2DWlpaYiJiblrexYtWgSpVKrfOnXqdNdYc/r59rPfoT3kcKDuc0LaJDcnMbZOG4ABnV1QXlOPSV+n4ZczN7hOi5gQ5/fArcHBgwcRExODL7/8Er6+vneNmzdvHtRqtX67evVqC2bZgDGmn30tgkafE9KmScS2iIsJxjBfBWq1Ory6KRObUi5znRYxEU4LuKurK2xsbFBQUGCwv6CgAApF0/N2KxSKe8Y3/rxfzD8HydXX16OkpOSOzz106BCeffZZ/Pe//8XEiRPv2R6RSASJRGKwtbSsa2pcu1UFe6ENhnan7nNC2jqxrQ1WRQUgMtgTOga8t+MMlu/PBWOM69TIQ+K0gAuFQgQGBiIxMVG/T6fTITExESEhIU2eExISYhAPAAkJCfp4Hx8fKBQKgxiNRoOUlBR9TEhICEpLS5GRkaGPOXDgAHQ6HZRKpX7f77//joiICCxZssRghLola5y8JbSnHHZCG46zIYRYAhs+D/8Z3RszQ7sCAP538A+8vvUkaupp6lWrZtIhcQ9gy5YtTCQSsbi4OJadnc1iY2OZTCZjKpWKMcbYhAkT2Ny5c/Xxx44dYwKBgC1fvpzl5OSwhQsXMltbW3b69Gl9zOLFi5lMJmO7du1ip06dYiNHjmQ+Pj6sqqpKHzNs2DDm7+/PUlJS2NGjR1nXrl1ZZGSk/viBAweYvb09mzdvHrtx44Z+Ky4ubnbbWnoUularYyH/+Y15zYlnv5y50SKfSQixLlvTrrBH5u1hXnPi2dg1x9mtihquU2r1zFULOC/gjDG2cuVK5unpyYRCIQsODmbJycn6Y0OGDGHR0dEG8du2bWPdunVjQqGQ+fr6sj179hgc1+l0bP78+UwulzORSMRCQ0NZbm6uQUxxcTGLjIxkjo6OTCKRsJiYGFZWVqY/Hh0dzQDcsQ0ZMqTZ7WrpAp5+qZh5zYlnvgt+YVW19S3ymYQQ63Pk/E3We8EvzGtOPBu6/CC7XFTBdUqtmrlqAY8xuhFiLhqNBlKpFGq1ukXuh3/w81lsOHYJo/074L8v9DP75xFCrFeuqgwxG1JxXV2Ndg5CrJsYiECvuz8mSx6cuWoBjUJvJXQ6hr009zkhpJm6K5ywY8Yg9O4gQXFFLSLXpWB7ess/OUMeHBXwViL98i0UaGrgJBbg0a6uXKdDCLECcokYW2NDEO4rR61Wh7d/OIWP4rNRr9VxnRppBirgrUT87clbwn0VEAlo9DkhpHkcRAKsjgrErNsj1NcfzcPkjem0JKkVoALeCmh1DHtPN8wgR5O3EEKMxefz8MaT3fBFVADsbG1w+PxNjF51DH8Ult//ZMIZKuCtQEpeMYrKayC1s8WjXaj7nBDyYIb7ueOH6SHoILPDxaIKjFp1TD+2hlgeKuCtQOPUqcN8FbC1oT9SQsiD8/WQYtdrg6D0aZhD/dVNmfgoPht1dF/c4tC3vZWr1+rwy5mG7vNn+lL3OSHk4bk6irDpZSWmDekMoOG+eOS6ZBRoqjnOjPwdFXArl3yxBMUVtXBxECKkczuu0yGEtBICGz7mPd0Ta14KhJNIgPTLtxCx4ggtS2pBqIBbucbR58N6KyCg7nNCiIkN663A7n89ih4KJxSV1yLqqxR88msudalbAPrGt2J1Wh1+OXu7+9yPus8JIebh4+qAHa8OwrigjmAMWHngD4xbm4SrJZVcp9amUQG3Ysf+KEJpZR1cHYVQUvc5IcSM7IQ2WPp8X6yM9IeTWIATV0ox/PMj2HUyn+vU2iwq4FYs/vbo86d7u8OGz+M4G0JIW/BsXw/snfkYAr2cUVZTj1lbTmL2tpPQVNPELy2NCriVqq3XYX9j9zlN3kIIaUGdXOyxNXYAZoV2BZ8H/JSZj/D/HsbvuYVcp9amUAG3Ukcu3ERZdT3cnEQI8qYVhAghLUtgw8cbT3bDtmkh8G5njxvqakzakIZ3fsiiq/EWQgXcSjVO3jLcj7rPCSHcCfJ2wb5ZgzF5kA94PGBb+jW6Gm8hVMCtUHWdFr9mFwCg7nNCCPfshDZY8GwvbI01vBp/fcsJFJbR5C/mQgXcCh0+fxPlNfVwl4oR4OnMdTqEEAIACPYxvBrfefI6Qpcfwsbjl6DVMa7Ta3WogFuhxtHnEX7u4FP3OSHEgjReje+aMQh9OkpRVlOPhbvPYuSqozh5tZTr9FoVKuBWprpOi99yGrrPaelQQoil6tNRhh2vDsK/R/WGRCzAmXwNRn9xDHN+OEVzqpsIFXArc/BcISprteggs0O/TjKu0yGEkLuy4fPw0gAvHHjrcTwX0DCL29b0q3h82e/4NOE8KmrquU7RqlEBtzLxt9fmfaaPO3g86j4nhFg+V0cRPhnXFz9OD0GApwxVdVqsSLyAIct+x6aUy6inedUfCBVwK1JZW48DOQ2PZjzTx4PjbAghxDiBXi74cfpArI4KgHc7exSV1+C9HWfw1H8P46fMa1TIjUQF3IocOFeIqjotPF3s0buDhOt0CCHEaDweD0/7uePXN4bg/57tBWd7W1wsqsDsbVkI/fQQtqVfpZXOmokKuBWJz6Luc0JI6yAU8DFpkA+OzHkCc4b1gIuDEJeLK/HOD6cwdHlD13p1nZbrNC0ajzFGD+eZiUajgVQqhVqthkTycFfM5TX1CPwoATX1OuyZ+Sh8PaQmypIQQrhXWVuPTclXsPbwRRSV1wAAZPa2GN/fExNCvNBBZsdxhg/OlLXg7+gK3Eok5hSgpl6Hzq4O6OVO3eeEkNbFXijA1MGdceSdoVjwTC90dLZDaWUd1hz6E48tOYDp32Ug5WIx6JrzLwKuEyDN8/Pt7vMI6j4nhLRidkIbTH7UB9EDvZGYU4C445dw/M9i7Dujwr4zKvi4OmC0fweM9u+ATi72XKfLKepCNyNTdZtoqusQ9NFvqNXqsP/1weiucDJhloQQYtlyVWXYmHQJOzLzUfW3++LB3i4YE9ABw3orILMXcpjhvZmrC50KuBmZ6g/tx4xreHN7Frq4OSLhjcF0BU4IaZMqaurxyxkVdpzIx7E/i9BYvWz4PAR6OSOspxtCe8rxSHtHbhP9B3MVcOpCtwJ7aPIWQgiBg0iA5wI74rnAjrihrsKuk9ex80Q+zqnKkJpXgtS8Evxn7zn4uDpgSLf2CPZxQZCXM9wkYq5TNwu6AjcjU/yrS11Zh6CPE1CnZfht9mB0caPuc0II+burJZVIzClA4rlCJF8sRp3WsKx5tbNHkJcLArxk6KFwQle5EyRi2xbLr1WPQl+1ahW8vb0hFouhVCqRmpp6z/jt27ejR48eEIvF8PPzw969ew2OM8awYMECuLu7w87ODmFhYbhw4YJBTElJCaKioiCRSCCTyTBlyhSUl5cbxJw6dQqPPfYYxGIxOnXqhKVLl5qmwUbYn61CnZahh8KJijchhDShk4s9Jg3ywbdTlMic/yRWRwUgOsQLvdwl4PGAy8WV+DHzGt7bcQbPrU5Cn//7FSGLEjHx61R8FJ+N9UfzEH/qOlLzSnC5uAJVtdbx/DnnXehbt27F7NmzsWbNGiiVSnz22WcIDw9Hbm4u3Nzc7og/fvw4IiMjsWjRIjzzzDP4/vvvMWrUKGRmZqJ3794AgKVLl2LFihXYuHEjfHx8MH/+fISHhyM7OxticUNXSlRUFG7cuIGEhATU1dUhJiYGsbGx+P777wE0/IvpqaeeQlhYGNasWYPTp09j8uTJkMlkiI2NbbHfz9+XDiWEEHJvTmJbPO3njqdvf2dqquuQefkW0i/dQta1UlwoKIdKU40b6obt8PmbTb6PUMCHk0gAB5EA04Z0RpTSqyWb0Sycd6ErlUr0798f//vf/wAAOp0OnTp1wr/+9S/MnTv3jvgXXngBFRUViI+P1+8bMGAA+vXrhzVr1oAxBg8PD7z55pt46623AABqtRpyuRxxcXEYP348cnJy0KtXL6SlpSEoKAgA8Msvv2D48OG4du0aPDw8sHr1arz33ntQqVQQChtGN86dOxc7d+7EuXPnmtW2h+02KauuQ8BHDd3nB996HD6uDka/ByGEEEPqqjr8UViGXFU5/rxZjgJNNQo1NSgoq0aBphrVdYZTub4f0RMvP9b5gT+vVQ5iq62tRUZGBubNm6ffx+fzERYWhqSkpCbPSUpKwuzZsw32hYeHY+fOnQCAvLw8qFQqhIWF6Y9LpVIolUokJSVh/PjxSEpKgkwm0xdvAAgLCwOfz0dKSgpGjx6NpKQkDB48WF+8Gz9nyZIluHXrFpydne/IraamBjU1NfrXGo3GuF/IPziJbXHo7aE49kcRFW9CCDERqZ0tAr1cEOjlcscxxhjKauqhqapDRY0W5TX1FjsLHKf3wIuKiqDVaiGXyw32y+VyqFSqJs9RqVT3jG/8eb+Yf3bPCwQCuLi4GMQ09R5//4x/WrRoEaRSqX7r1KlT0w03gofMDmODHv59CCGE3B+Px4NEbIuOzvbornBCoJczFFLLHMVuEYPYWot58+ZBrVbrt6tXr3KdEiGEkFaK0wLu6uoKGxsbFBQUGOwvKCiAQqFo8hyFQnHP+Maf94spLCw0OF5fX4+SkhKDmKbe4++f8U8ikQgSicRgI4QQQsyB0wIuFAoRGBiIxMRE/T6dTofExESEhIQ0eU5ISIhBPAAkJCTo4318fKBQKAxiNBoNUlJS9DEhISEoLS1FRkaGPubAgQPQ6XRQKpX6mMOHD6Ours7gc7p3797k/W9CCCGkRTGObdmyhYlEIhYXF8eys7NZbGwsk8lkTKVSMcYYmzBhAps7d64+/tixY0wgELDly5eznJwctnDhQmZra8tOnz6tj1m8eDGTyWRs165d7NSpU2zkyJHMx8eHVVVV6WOGDRvG/P39WUpKCjt69Cjr2rUri4yM1B8vLS1lcrmcTZgwgZ05c4Zt2bKF2dvbs7Vr1za7bWq1mgFgarX6YX5FhBBCrJi5agHnBZwxxlauXMk8PT2ZUChkwcHBLDk5WX9syJAhLDo62iB+27ZtrFu3bkwoFDJfX1+2Z88eg+M6nY7Nnz+fyeVyJhKJWGhoKMvNzTWIKS4uZpGRkczR0ZFJJBIWExPDysrKDGKysrLYo48+ykQiEevQoQNbvHixUe2iAk4IIcRctYDz58BbM3M9+0cIIcR6tOqpVAkhhBBiHM6nUm3NGjs3HnZCF0IIIdarsQaYusObCrgZlZWVAYBJJnQhhBBi3crKyiCVSk32fnQP3Ix0Oh2uX78OJyenB17HW6PRoFOnTrh69WqruY/e2trU2toDtL42tbb2ANQma9DYnitXroDH48HDwwN8vunuXNMVuBnx+Xx07NjRJO/VGieGaW1tam3tAVpfm1pbewBqkzWQSqVmaQ8NYiOEEEKsEBVwQgghxApRAbdwIpEICxcuhEgk4joVk2ltbWpt7QFaX5taW3sAapM1MHd7aBAbIYQQYoXoCpwQQgixQlTACSGEECtEBZwQQgixQlTACSGEECtEBdzCrVq1Ct7e3hCLxVAqlUhNTeU6pWZZtGgR+vfvDycnJ7i5uWHUqFHIzc01iKmursaMGTPQrl07ODo64rnnnkNBQQFHGRtn8eLF4PF4eP311/X7rLE9+fn5eOmll9CuXTvY2dnBz88P6enp+uOMMSxYsADu7u6ws7NDWFgYLly4wGHG96bVajF//nz4+PjAzs4OjzzyCD766CODOagtuU2HDx/Gs88+Cw8PD/B4POzcudPgeHNyLykpQVRUFCQSCWQyGaZMmYLy8vIWbIWhe7Wprq4Oc+bMgZ+fHxwcHODh4YGJEyfi+vXrBu9hTW36p1deeQU8Hg+fffaZwX5TtIkKuAXbunUrZs+ejYULFyIzMxN9+/ZFeHg4CgsLuU7tvg4dOoQZM2YgOTkZCQkJqKurw1NPPYWKigp9zBtvvIGff/4Z27dvx6FDh3D9+nWMGTOGw6ybJy0tDWvXrkWfPn0M9ltbe27duoVBgwbB1tYW+/btQ3Z2Nj755BM4OzvrY5YuXYoVK1ZgzZo1SElJgYODA8LDw1FdXc1h5ne3ZMkSrF69Gv/73/+Qk5ODJUuWYOnSpVi5cqU+xpLbVFFRgb59+2LVqlVNHm9O7lFRUTh79iwSEhIQHx+Pw4cPIzY2tqWacId7tamyshKZmZmYP38+MjMz8dNPPyE3NxcjRowwiLOmNv3djh07kJycDA8PjzuOmaRNJl1dnJhUcHAwmzFjhv61VqtlHh4ebNGiRRxm9WAKCwsZAHbo0CHGGGOlpaXM1taWbd++XR+Tk5PDALCkpCSu0ryvsrIy1rVrV5aQkMCGDBnCZs2axRizzvbMmTOHPfroo3c9rtPpmEKhYMuWLdPvKy0tZSKRiG3evLklUjRaREQEmzx5ssG+MWPGsKioKMaYdbUJANuxY4f+dXNyz87OZgBYWlqaPmbfvn2Mx+Ox/Pz8Fsv9bv7ZpqakpqYyAOzy5cuMMett07Vr11iHDh3YmTNnmJeXF/vvf/+rP2aqNtEVuIWqra1FRkYGwsLC9Pv4fD7CwsKQlJTEYWYPRq1WAwBcXFwAABkZGairqzNoX48ePeDp6WnR7ZsxYwYiIiIM8gassz27d+9GUFAQxo4dCzc3N/j7++PLL7/UH8/Ly4NKpTJok1QqhVKptNg2DRw4EImJiTh//jwAICsrC0ePHsXTTz8NwDrb1Kg5uSclJUEmkyEoKEgfExYWBj6fj5SUlBbP+UGo1WrweDzIZDIA1tkmnU6HCRMm4O2334avr+8dx03VJlrMxEIVFRVBq9VCLpcb7JfL5Th37hxHWT0YnU6H119/HYMGDULv3r0BACqVCkKhUP8/aSO5XA6VSsVBlve3ZcsWZGZmIi0t7Y5j1tieixcvYvXq1Zg9ezbeffddpKWlYebMmRAKhYiOjtbn3dTfQUtt09y5c6HRaNCjRw/Y2NhAq9Xi448/RlRUFABYZZsaNSd3lUoFNzc3g+MCgQAuLi4W3z6gYRzJnDlzEBkZqV/8wxrbtGTJEggEAsycObPJ46ZqExVwYnYzZszAmTNncPToUa5TeWBXr17FrFmzkJCQALFYzHU6JqHT6RAUFIT//Oc/AAB/f3+cOXMGa9asQXR0NMfZPZht27Zh06ZN+P777+Hr64uTJ0/i9ddfh4eHh9W2qa2oq6vDuHHjwBjD6tWruU7ngWVkZODzzz9HZmbmAy8j3VzUhW6hXF1dYWNjc8co5oKCAigUCo6yMt5rr72G+Ph4HDx40GBpVYVCgdraWpSWlhrEW2r7MjIyUFhYiICAAAgEAggEAhw6dAgrVqyAQCCAXC63qvYAgLu7O3r16mWwr2fPnrhy5QoA6PO2pr+Db7/9NubOnYvx48fDz88PEyZMwBtvvIFFixYBsM42NWpO7gqF4o5BrvX19SgpKbHo9jUW78uXLyMhIcFg6U1ra9ORI0dQWFgIT09P/XfF5cuX8eabb8Lb2xuA6dpEBdxCCYVCBAYGIjExUb9Pp9MhMTERISEhHGbWPIwxvPbaa9ixYwcOHDgAHx8fg+OBgYGwtbU1aF9ubi6uXLlike0LDQ3F6dOncfLkSf0WFBSEqKgo/X9bU3sAYNCgQXc82nf+/Hl4eXkBAHx8fKBQKAzapNFokJKSYrFtqqysBJ9v+LVmY2MDnU4HwDrb1Kg5uYeEhKC0tBQZGRn6mAMHDkCn00GpVLZ4zs3RWLwvXLiA3377De3atTM4bm1tmjBhAk6dOmXwXeHh4YG3334b+/fvB2DCNj342Dtiblu2bGEikYjFxcWx7OxsFhsby2QyGVOpVFyndl/Tp09nUqmU/f777+zGjRv6rbKyUh/zyiuvME9PT3bgwAGWnp7OQkJCWEhICIdZG+fvo9AZs772pKamMoFAwD7++GN24cIFtmnTJmZvb8++++47fczixYuZTCZju3btYqdOnWIjR45kPj4+rKqqisPM7y46Opp16NCBxcfHs7y8PPbTTz8xV1dX9s477+hjLLlNZWVl7MSJE+zEiRMMAPv000/ZiRMn9COym5P7sGHDmL+/P0tJSWFHjx5lXbt2ZZGRkVw16Z5tqq2tZSNGjGAdO3ZkJ0+eNPiuqKmpsco2NeWfo9AZM02bqIBbuJUrVzJPT08mFApZcHAwS05O5jqlZgHQ5LZhwwZ9TFVVFXv11VeZs7Mzs7e3Z6NHj2Y3btzgLmkj/bOAW2N7fv75Z9a7d28mEolYjx492Lp16wyO63Q6Nn/+fCaXy5lIJGKhoaEsNzeXo2zvT6PRsFmzZjFPT08mFotZ586d2XvvvWdQDCy5TQcPHmzy/5vo6GjGWPNyLy4uZpGRkczR0ZFJJBIWExPDysrKOGhNg3u1KS8v767fFQcPHrTKNjWlqQJuijbRcqKEEEKIFaJ74IQQQogVogJOCCGEWCEq4IQQQogVogJOCCGEWCEq4IQQQogVogJOCCGEWCEq4IQQQogVogJOCCGEWCEq4ISQu7p58yamT58OT09PiEQiKBQKhIeH49ixYwAAHo+HnTt3cpskIW0ULSdKCLmr5557DrW1tdi4cSM6d+6MgoICJCYmori4mOvUCGnz6AqcENKk0tJSHDlyBEuWLMHQoUPh5eWF4OBgzJs3DyNGjNAvjTh69GjweDz9awDYtWsXAgICIBaL0blzZ3zwwQeor6/XH+fxeFi9ejWefvpp2NnZoXPnzvjhhx/0x2tra/Haa6/B3d0dYrEYXl5e+iVBCSENqIATQprk6OgIR0dH7Ny5EzU1NXccT0tLAwBs2LABN27c0L8+cuQIJk6ciFmzZiE7Oxtr165FXFwcPv74Y4Pz58+fj+eeew5ZWVmIiorC+PHjkZOTAwBYsWIFdu/ejW3btiE3NxebNm0y+AcCIQSgxUwIIXf1448/YurUqaiqqkJAQACGDBmC8ePHo0+fPgAarqR37NiBUaNG6c8JCwtDaGgo5s2bp9/33Xff4Z133sH169f1573yyitYvXq1PmbAgAEICAjAF198gZkzZ+Ls2bP47bffwOPxWqaxhFgZugInhNzVc889h+vXr2P37t0YNmwYfv/9dwQEBCAuLu6u52RlZeHDDz/UX8E7Ojpi6tSpuHHjBiorK/VxISEhBueFhITor8AnTZqEkydPonv37pg5cyZ+/fVXs7SPEGtGBZwQck9isRhPPvkk5s+fj+PHj2PSpElYuHDhXePLy8vxwQcf4OTJk/rt9OnTuHDhAsRicbM+MyAgAHl5efjoo49QVVWFcePG4fnnnzdVkwhpFaiAE0KM0qtXL1RUVAAAbG1todVqDY4HBAQgNzcXXbp0uWPj8//6yklOTjY4Lzk5GT179tS/lkgkeOGFF/Dll19i69at+PHHH1FSUmLGlhFiXegxMkJIk4qLizF27FhMnjwZffr0gZOTE9LT07F06VKMHDkSAODt7Y3ExEQMGjQIIpEIzs7OWLBgAZ555hl4enri+eefB5/PR1ZWFs6cOYN///vf+vffvn07goKC8Oijj2LTpk1ITU3F+vXrAQCffvop3N3d4e/vDz6fj+3bt0OhUEAmk3HxqyDEMjFCCGlCdXU1mzt3LgsICGBSqZTZ29uz7t27s/fff59VVlYyxhjbvXs369KlCxMIBMzLy0t/7i+//MIGDhzI7OzsmEQiYcHBwWzdunX64wDYqlWr2JNPPslEIhHz9vZmW7du1R9ft24d69evH3NwcGASiYSFhoayzMzMFms7IdaARqETQlpcU6PXCSHGoXvghBBCiBWiAk4IIYRYIRrERghpcXTnjpCHR1fghBBCiBWiAk4IIYRYISrghBBCiBWiAk4IIYRYISrghBBCiBWiAk4IIYRYISrghBBCiBWiAk4IIYRYISrghBBCiBX6fzhtZ3SXK1BAAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5, 3))\n",
    "plt.plot(range(len(lrs)), lrs)\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.xlabel(\"Steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f85b01-859b-4454-a3a3-c7ef593735a6",
   "metadata": {},
   "source": [
    "- And a quick look at the loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "445d8155-6eae-4b50-a381-d0820ebc27cc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/ipykernel_36856/3589549395.py:5: UserWarning: The figure layout has changed to tight\n",
      "  plt.tight_layout(); plt.savefig(\"3.pdf\")\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABZdUlEQVR4nO3deXxM1/vA8c9Mlsm+iMiCxBYiQexKqrTyFapaVKlfqnRTO9WqamvpglqqipbSb/m21dKNqrWhtqp9V8QWu4g1q2wz5/fHMDH2Jcmdief9et3XzD13e06Geeaee+49OqWUQgghhBA2R691AEIIIYS4OUnSQgghhI2SJC2EEELYKEnSQgghhI2SJC2EEELYKEnSQgghhI2SJC2EEELYKEnSQgghhI2SJC2EEELYKEnSQtiwI0eOoNPp2L59u9ahCCE0IElaiEKm0+luOw0fPlzrEIUQNspR6wCEKO5Onz5teT9nzhyGDh1KQkKCpczDw0OLsIQQdkDOpIUoZIGBgZbJ29sbnU5nmS9VqhTjx4+nTJkyGAwGatasyZIlS265L6PRyMsvv0x4eDjHjh0D4Pfff6d27dq4uLhQoUIFPvjgA/Ly8izb6HQ6vv76a9q2bYubmxthYWHMnz/fsvzixYvExcXh7++Pq6srYWFhzJgx45Yx/PLLL1SvXh1XV1f8/PyIiYkhIyPDsvzrr7+matWquLi4EB4ezpdffmm1/fHjx+nQoQM+Pj6UKFGCZ555hiNHjliWd+3alTZt2jBu3DiCgoLw8/OjV69e5Obm3vXfXIhiQwkhisyMGTOUt7e3ZX78+PHKy8tL/fjjj2rfvn3q7bffVk5OTmr//v1KKaUSExMVoLZt26aysrJU27ZtVa1atVRycrJSSqnVq1crLy8vNXPmTHXo0CH1559/qnLlyqnhw4dbjgGoMmXKqB9++EEdOHBA9e3bV3l4eKjz588rpZTq1auXqlmzptq0aZNKTExU8fHxav78+TeN/9SpU8rR0VGNHz9eJSYmqp07d6ovvvhCpaWlKaWU+v7771VQUJD69ddf1eHDh9Wvv/6qSpQooWbOnKmUUionJ0dVrVpVvfzyy2rnzp1qz5496v/+7/9UlSpVVHZ2tlJKqS5duigvLy/VvXt3tXfvXvXHH38oNzc3NW3atIL9MISwA5KkhShC1yfp4OBgNWLECKt16tWrp3r27KmUyk/Sa9asUc2aNVOPPvqounTpkmXdZs2aqZEjR1pt/91336mgoCDLPKDef/99y3x6eroC1OLFi5VSSrVu3Vq99NJLdxX/li1bFKCOHDly0+UVK1ZUP/zwg1XZRx99pBo2bGiJrUqVKspkMlmWZ2dnK1dXV7V06VKllDlJh4aGqry8PMs6zz33nOrYseNdxShEcSLXpIXQSGpqKqdOnSI6OtqqPDo6mh07dliVderUiTJlyvDXX3/h6upqKd+xYwdr165lxIgRljKj0UhWVhaZmZm4ubkBUKNGDctyd3d3vLy8SE5OBqBHjx48++yzbN26lebNm9OmTRsaNWp005ijoqJo1qwZ1atXJzY2lubNm9O+fXt8fX3JyMjg0KFDvPLKK7z22muWbfLy8vD29rbEe/DgQTw9Pa32m5WVxaFDhyzzkZGRODg4WOaDgoLYtWvXbf6aQhRPkqSFsANPPvkk33//PevWreOJJ56wlKenp/PBBx/Qrl27G7ZxcXGxvHdycrJaptPpMJlMALRs2ZKjR4+yaNEi4uPjadasGb169WLcuHE37NPBwYH4+Hj++ecf/vzzTyZNmsR7773Hhg0bLD8Ipk+fToMGDW7Y7mq8derUYdasWTfs29/f/67iFeJhIklaCI14eXkRHBzM2rVradKkiaV87dq11K9f32rdHj16UK1aNZ5++mkWLlxoWb927dokJCRQqVKlB4rF39+fLl260KVLFxo3bszAgQNvmqTBnDCjo6OJjo5m6NChhIaGMnfuXAYMGEBwcDCHDx8mLi7uptvWrl2bOXPmUKpUKby8vB4oZiEeBpKkhdDQwIEDGTZsGBUrVqRmzZrMmDGD7du33/RMs0+fPhiNRp566ikWL17Mo48+ytChQ3nqqacICQmhffv26PV6duzYwe7du/n444/vKoahQ4dSp04dIiMjyc7OZsGCBVStWvWm627YsIHly5fTvHlzSpUqxYYNGzh79qxl/Q8++IC+ffvi7e1NixYtyM7OZvPmzVy8eJEBAwYQFxfH2LFjeeaZZ/jwww8pU6YMR48e5bfffuPtt9+mTJky9//HFKIYkiQthIb69u1LSkoKb775JsnJyURERDB//nzCwsJuun7//v0xmUw8+eSTLFmyhNjYWBYsWMCHH37I6NGjcXJyIjw8nFdfffWuY3B2dmbw4MEcOXIEV1dXGjduzOzZs2+6rpeXF6tXr2bChAmkpqYSGhrKp59+SsuWLQF49dVXcXNzY+zYsQwcOBB3d3eqV69O//79AXBzc2P16tUMGjSIdu3akZaWRunSpWnWrJmcWQtxEzqllNI6CCGEEELcSB5mIoQQQtgoSdJCCCGEjZIkLYQQQtgoSdJCCCGEjZIkLYQQQtgoSdJCCCGEjZIkfRtffPEF5cqVw8XFhQYNGrBx40ZN41m9ejWtW7cmODgYnU7HvHnzrJYrpRg6dChBQUG4uroSExPDgQMHrNa5cOECcXFxeHl54ePjwyuvvEJ6errVOjt37qRx48a4uLhQtmxZxowZc0MsP//8M+Hh4bi4uFC9enUWLVr0QHUbNWoU9erVw9PTk1KlStGmTRurMZfB/HznXr164efnh4eHB88++yxnzpyxWufYsWO0atUKNzc3SpUqxcCBA62GbQRYuXIltWvXxmAwUKlSJWbOnHlDPAX52U+ZMoUaNWrg5eWFl5cXDRs2ZPHixXZfr5v55JNP0Ol0lvui7bl+w4cPR6fTWU3h4eF2X6+rTp48yQsvvICfnx+urq5Ur16dzZs3W5bb6/dJuXLlbvjcdDodvXr1Auzwc9N2fA/bNXv2bOXs7Ky++eYb9e+//6rXXntN+fj4qDNnzmgW06JFi9R7772nfvvtNwWouXPnWi3/5JNPlLe3t5o3b57asWOHevrpp1X58uXV5cuXLeu0aNFCRUVFqfXr16s1a9aoSpUqqU6dOlmWp6SkqICAABUXF6d2796tfvzxR+Xq6qq++uoryzpr165VDg4OasyYMWrPnj3q/fffV05OTmrXrl33XbfY2Fg1Y8YMtXv3brV9+3b15JNPqpCQEJWenm5Zp3v37qps2bJq+fLlavPmzeqRRx5RjRo1sizPy8tT1apVUzExMWrbtm1q0aJFqmTJkmrw4MGWdQ4fPqzc3NzUgAED1J49e9SkSZOUg4ODWrJkiWWdgv7s58+frxYuXKj279+vEhIS1LvvvqucnJzU7t277bpe19u4caMqV66cqlGjhurXr5+l3F7rN2zYMBUZGalOnz5tmc6ePWv39VJKqQsXLqjQ0FDVtWtXtWHDBnX48GG1dOlSdfDgQcs69vp9kpycbPWZxcfHK0CtWLFCKWV/n5sk6VuoX7++6tWrl2XeaDSq4OBgNWrUKA2jynd9kjaZTCowMFCNHTvWUnbp0iVlMBjUjz/+qJRSas+ePQpQmzZtsqyzePFipdPp1MmTJ5VSSn355ZfK19fXMravUkoNGjRIValSxTLfoUMH1apVK6t4GjRooF5//fUCq19ycrIC1KpVqyx1cXJyUj///LNlnb179ypArVu3Till/hGj1+tVUlKSZZ0pU6YoLy8vS33efvttFRkZaXWsjh07qtjYWMt8UXz2vr6+6uuvvy429UpLS1NhYWEqPj5eNWnSxJKk7bl+w4YNU1FRUTddZs/1Usr8f/rRRx+95fLi9H3Sr18/VbFiRWUymezyc5Pm7pvIyclhy5YtxMTEWMr0ej0xMTGsW7dOw8huLTExkaSkJKuYvb29adCggSXmdevW4ePjQ926dS3rxMTEoNfr2bBhg2Wdxx57DGdnZ8s6sbGxJCQkcPHiRcs61x7n6joF+bdJSUkBoESJEgBs2bKF3Nxcq+OGh4cTEhJiVb/q1asTEBBgFVdqair//vvvXcVe2J+90Whk9uzZZGRk0LBhw2JTr169etGqVasbYrD3+h04cIDg4GAqVKhAXFwcx44dKxb1mj9/PnXr1uW5556jVKlS1KpVi+nTp1uWF5fvk5ycHL7//ntefvlldDqdXX5ukqRv4ty5cxiNRqsPCSAgIICkpCSNorq9q3HdLuakpCRKlSpltdzR0ZESJUpYrXOzfVx7jFutU1B/G5PJRP/+/YmOjqZatWqWYzo7O+Pj43Pb+t1v7KmpqVy+fLnQPvtdu3bh4eGBwWCge/fuzJ07l4iICLuvF8Ds2bPZunUro0aNumGZPdevQYMGzJw5kyVLljBlyhQSExNp3LgxaWlpdl0vgMOHDzNlyhTCwsJYunQpPXr0oG/fvvzvf/+zis/ev0/mzZvHpUuX6Nq1q+VY9va5yQAbwub06tWL3bt38/fff2sdSoGpUqUK27dvJyUlhV9++YUuXbqwatUqrcN6YMePH6dfv37Ex8dbjV9dHFwdNASgRo0aNGjQgNDQUH766SdcXV01jOzBmUwm6taty8iRIwGoVasWu3fvZurUqXTp0kXj6ArOf//7X1q2bElwcLDWodw3OZO+iZIlS+Lg4HBDj78zZ84QGBioUVS3dzWu28UcGBhIcnKy1fK8vDwuXLhgtc7N9nHtMW61TkH8bXr37s2CBQtYsWKF1bCFgYGB5OTkcOnSpdvW735j9/LywtXVtdA+e2dnZypVqkSdOnUYNWoUUVFRfP7553Zfry1btpCcnEzt2rVxdHTE0dGRVatWMXHiRBwdHQkICLDr+l3Lx8eHypUrc/DgQbv/3IKCgoiIiLAqq1q1qqU5vzh8nxw9epRly5ZZjQhnj5+bJOmbcHZ2pk6dOixfvtxSZjKZWL58OQ0bNtQwslsrX748gYGBVjGnpqayYcMGS8wNGzbk0qVLbNmyxbLOX3/9hclkokGDBpZ1Vq9eTW5urmWd+Ph4qlSpgq+vr2Wda49zdZ0H+dsopejduzdz587lr7/+onz58lbL69Spg5OTk9VxExISOHbsmFX9du3aZfXFER8fj5eXl+UL6U6xF9VnbzKZyM7Otvt6NWvWjF27drF9+3bLVLduXeLi4izv7bl+10pPT+fQoUMEBQXZ/ecWHR19wy2O+/fvJzQ0FLD/7xOAGTNmUKpUKVq1amUps8vP7Z66mT1EZs+erQwGg5o5c6bas2eP6tatm/Lx8bHq8VfU0tLS1LZt29S2bdsUoMaPH6+2bdumjh49qpQy3zLh4+Ojfv/9d7Vz5071zDPP3PSWiVq1aqkNGzaov//+W4WFhVndMnHp0iUVEBCgOnfurHbv3q1mz56t3NzcbrhlwtHRUY0bN07t3btXDRs27IFvwerRo4fy9vZWK1eutLp9IjMz07JO9+7dVUhIiPrrr7/U5s2bVcOGDVXDhg0ty6/eOtG8eXO1fft2tWTJEuXv73/TWycGDhyo9u7dq7744oub3jpRkJ/9O++8o1atWqUSExPVzp071TvvvKN0Op36888/7bpet3Jt7257rt+bb76pVq5cqRITE9XatWtVTEyMKlmypEpOTrbreillvl3O0dFRjRgxQh04cEDNmjVLubm5qe+//96yjj1/nxiNRhUSEqIGDRp0wzJ7+9wkSd/GpEmTVEhIiHJ2dlb169dX69ev1zSeFStWKOCGqUuXLkop820TQ4YMUQEBAcpgMKhmzZqphIQEq32cP39ederUSXl4eCgvLy/10ksvqbS0NKt1duzYoR599FFlMBhU6dKl1SeffHJDLD/99JOqXLmycnZ2VpGRkWrhwoUPVLeb1QtQM2bMsKxz+fJl1bNnT+Xr66vc3NxU27Zt1enTp632c+TIEdWyZUvl6uqqSpYsqd58802Vm5trtc6KFStUzZo1lbOzs6pQoYLVMa4qyM/+5ZdfVqGhocrZ2Vn5+/urZs2aWRK0PdfrVq5P0vZav44dO6qgoCDl7OysSpcurTp27Gh1H7G91uuqP/74Q1WrVk0ZDAYVHh6upk2bZrXcnr9Pli5dqoAb4lXK/j43nVJK3du5txBCCCGKglyTFkIIIWyUJGkhhBDCRkmSFkIIIWyUJGkhhBDCRkmSFkIIIWyUJGkhhBDCRkmSvo3s7GyGDx9Odna21qEUiuJcP6mbfZK62SepW+GR+6RvIzU1FW9vb1JSUvDy8tI6nAJXnOsndbNPUjf7JHUrPHImLYQQQtgoSdJCCCGEjSr240nn5eWxbds2AgIC0Ovv7TdJWloaACdPniQ1NbUwwtNUca6f1M0+Sd3s08NQt+PHj5OZmUmtWrVwdCy61Fnsr0lv2rSJ+vXrax2GEEKIYmDjxo3Uq1evyI5X7M+kAwICAPMfNigoSONohBBC2KPTp09Tv359S04pKsU+SV9t4g4KCqJMmTIaRyOEEMKe3etl0wc+XpEeTQghhBB3TZK0EEIIYaMkSQshhBA2qthfkxZCFG8mk4mcnBytwxDFgLOzc5Ffc74TSdJ3KSM7jz/3JNG2lnQ+E8JW5OTkkJiYiMlk0joUUQzo9XrKly+Ps7Oz1qFYSJK+C1m5RmLGLadZ5mIi0qKp8thzWockxENPKcXp06dxcHCgbNmyNncGJOyLyWTi1KlTnD59mpCQEHQ6ndYhAZKk74qLkwPv+a/lqVMzOLtqITzyJDi7ax2WEA+1vLw8MjMzCQ4Oxs3NTetwRDHg7+/PqVOnyMvLw8nJSetwAOk4dtfqtO3HCVUSf2MyJ+cN1zocIR56RqMRwKaaJoV9u/pv6eq/LVsgSfouBfn7sbrSIAAC9vwXdeZfjSMSQgA20ywp7J8t/luSJH0Pmj3zIvGmejhiJPWXviCdVYQQQhQiSdL3IMDLhT1R75KpDHif3YzaPkvrkIQQgnLlyjFhwoS7Xn/lypXodDouXbpUaDEBzJw5Ex8fn0I9RnEnSfoe/V/zaL5Q7QHIXfI+ZF7QOCIhhL3Q6XS3nYYPH35f+920aRPdunW76/UbNWrE6dOn8fb2vq/jiaIjvbvvkb+nAVODHuzbsJrwnOOo+GHonpmkdVhCCDtw+vRpy/s5c+YwdOhQEhISLGUeHh6W90opjEbjXY1d7O/vf09xODs7ExgYeE/bCG3ImfR9eK1pFT7mVQB0276FYxs0jkgIYQ8CAwMtk7e3NzqdzjK/b98+PD09Wbx4MXXq1MFgMPD3339z6NAhnnnmGQICAvDw8KBevXosW7bMar/XN3frdDq+/vpr2rZti5ubG2FhYcyfP9+y/Prm7qvN0kuXLqVq1ap4eHjQokULqx8VeXl59O3bFx8fH/z8/Bg0aBBdunShTZs29/Q3mDJlChUrVsTZ2ZkqVarw3XffWZYppRg+fDghISEYDAaCg4Pp27evZfmXX35JWFgYLi4uBAQE0L59+3s6tj2SJH0fSrg7ExXdgjl5TQFQC/qDMVfTmIR42CmlyMzJ02RSShVYPd555x0++eQT9u7dS40aNUhPT+fJJ59k+fLlbNu2jRYtWtC6dWuOHTt22/188MEHdOjQgZ07d/Lkk08SFxfHhQu3vjyXmZnJuHHj+O6771i9ejXHjh3jrbfesiwfPXo0s2bNYsaMGaxdu5bU1FTmzZt3T3WbO3cu/fr1480332T37t28/vrrvPTSS6xYsQKAX3/9lc8++4yvvvqKAwcOMG/ePKpXrw7A5s2b6du3Lx9++CEJCQksWbKExx577J6Ob4+kufs+vda4Aq3/eYHmajO+yXtgw1Ro1EfrsIR4aF3ONRIxdKkmx97zYSxuzgXzdfrhhx/yn//8xzJfokQJoqKiLPMfffQRc+fOZf78+fTu3fuW++natSudOnUCYOTIkUycOJGNGzfSokWLm66fm5vL1KlTqVixIgC9e/fmww8/tCyfNGkSgwcPpm3btgBMnjyZRYsW3VPdxo0bR9euXenZsycAAwYMYP369YwbN47HH3+cY8eOERgYSExMDE5OToSEhFC/fn0Ajh07hru7O0899RSenp6EhoZSq1atezq+PdL0THr16tW0bt2a4OBgdDrdDb/KlFIMHTqUoKAgXF1diYmJ4cCBA9oEex0fN2faPRrFyLz/A0Bt+R8Y8zSOSghh7+rWrWs1n56ezltvvUXVqlXx8fHBw8ODvXv33vFMukaNGpb37u7ueHl5kZycfMv13dzcLAkaICgoyLJ+SkoKZ86csSRMAAcHB+rUqXNPddu7dy/R0dFWZdHR0ezduxeA5557jsuXL1OhQgVee+015s6dS16e+Xv1P//5D6GhoVSoUIHOnTsza9YsMjMz7+n49kjTM+mMjAyioqJ4+eWXadeu3Q3Lx4wZw8SJE/nf//5H+fLlGTJkCLGxsezZswcXFxcNIrb28qPleWztE3jkXqbuI31p5SANE0JoxdXJgT0fxmp27ILi7m79yOG33nqL+Ph4xo0bR6VKlXB1daV9+/Z3HPnr+sda6nS62w5EcrP1C7IZ/26ULVuWhIQEli1bRnx8PD179mTs2LGsWrUKT09Ptm7dysqVK/nzzz8ZOnQow4cPZ9OmTcX6Ni9Nz6RbtmzJxx9/bGk+uZZSigkTJvD+++/zzDPPUKNGDb799ltOnTp1z9dBCou3qxOvPVaJGcaWfLrqFHlGebiJEFrR6XS4OTtqMhXmk6rWrl1L165dadu2LdWrVycwMJAjR44U2vFuxtvbm4CAADZt2mQpMxqNbN269Z72U7VqVdauXWtVtnbtWiIiIizzrq6utG7dmokTJ7Jy5UrWrVvHrl27AHB0dCQmJoYxY8awc+dOjhw5wl9//fUANbN9Nnvql5iYSFJSEjExMZYyb29vGjRowLp163j++edvul12djbZ2dmW+bS0tEKNs2t0eb7+O5HD5zKYv/0E7Zw2QHgrcJYH/gshHlxYWBi//fYbrVu3RqfTMWTIEE2G5uzTpw+jRo2iUqVKhIeHM2nSJC5evHhPP1AGDhxIhw4dqFWrFjExMfzxxx/89ttvlt7qM2fOxGg00qBBA9zc3Pj+++9xdXUlNDSUBQsWcPjwYR577DF8fX1ZtGgRJpOJKlWqFFaVbYLN9u5OSkoCICAgwKo8ICDAsuxmRo0ahbe3t2W69hdaYfAwOPL6Y+brOF6LesJvr8KacYV6TCHEw2P8+PH4+vrSqFEjWrduTWxsLLVr1y7yOAYNGkSnTp148cUXadiwIR4eHsTGxt7Tpcc2bdrw+eefM27cOCIjI/nqq6+YMWMGTZs2BcDHx4fp06cTHR1NjRo1WLZsGX/88Qd+fn74+Pjw22+/8cQTT1C1alWmTp3Kjz/+SGRkZCHV2DboVFFfdLgFnU7H3LlzLffc/fPPP0RHR3Pq1CmCgoIs63Xo0AGdTsecOXNuup/rz6RPnjxJREQEx48fp0yZMoUSe2ZOHo1Hr6Du5bVMdp2K03+GwSM9CuVYQgizrKwsEhMTKV++vE30UXnYmEwmqlatSocOHfjoo4+0DqdA3O7f1IkTJyhbtmyh5pKbsdkz6atPwzlz5oxV+ZkzZ277pByDwYCXl5dl8vT0LNQ4AdycHenRtCJLTXVp6/QlOXVfL/RjCiFEUTp69CjTp09n//797Nq1ix49epCYmMj//d//aR1asWazSbp8+fIEBgayfPlyS1lqaiobNmygYcOGGkZ2c3ENQvH3dGH3JQM/bzmudThCCFGg9Ho9M2fOpF69ekRHR7Nr1y6WLVtG1apVtQ6tWNO041h6ejoHDx60zCcmJrJ9+3ZKlChBSEgI/fv35+OPPyYsLMxyC1ZwcPA9P4auKLg6O9CzaUU++GMPk/86yHN+iTivGQvPfw+uvlqHJ4QQD6Rs2bI39MwWhU/TM+nNmzdTq1Yty1NjBgwYQK1atRg6dCgAb7/9Nn369KFbt27Uq1eP9PR0lixZYrPXnzrVDyHQy4UzKZlkzn0Djv4Ny4vHtRohhBBFT9Mk3bRpU5RSN0wzZ84EzJ3JPvzwQ5KSksjKymLZsmVUrlxZy5Bvy8XJgV5PVMKEnneyupgLN38DJ7ZoG5gQQgi7ZLPXpO1Vh7plKO3jypKMMA4EPQUoWNBfHhkqhBDinkmSLmAGRwf6PFEJgJ7JbVEuPpC0E/6ZqG1gQggh7I4k6ULwbJ0ylC3hyoEMV1aV62cuXDECTm3TNjAhhBB2RZJ0IXBy0NP3iTAA3kiIJK9KazDlwa+vQk6GxtEJIYSwF5KkC0nbWqUpX9Kdi5fzmOn3BngGw/mDsPRdrUMTQti5pk2b0r9/f8t8uXLlmDBhwm23udlwwPejoPZzO8OHD6dmzZqFegx7IUm6kDg66OnXzHw2PWndeTJaTQZ0sGUm7F2gaWxCCG20bt2aFi1a3HTZmjVr0Ol07Ny58573u2nTJrp16/ag4Vm5VaI8ffo0LVu2LNBjiVuTJF2IWkcFU9HfnZTLuQzfVRIa9TEvmN8HUk9rG5wQosi98sorxMfHc+LEiRuWzZgxg7p161KjRo173q+/vz9ubkUz8l5gYCAGg6FIjiUkSRcqB72Oj9tUR6+Dn7ec4DeflyCwBly+APN6gAbDzQkhtPPUU0/h7+9veRbEVenp6fz888+88sornD9/nk6dOlG6dGnc3NyoXr06P/744233e31z94EDB3jsscdwcXEhIiKC+Pj4G7YZNGgQlStXxs3NjQoVKjBkyBByc3MB85CRH3zwATt27ECn06HT6ayeX3Ftc/euXbt44okncHV1xc/Pj27dupGenm5Z3rVrV9q0acO4ceMICgrCz8+PXr16WY51N0wmEx9++CFlypTBYDBQs2ZNlixZYlmek5ND7969CQoKwsXFhdDQUEaNGgWAUorhw4cTEhKCwWAgODiYvn373vWxtWaz40kXFw0r+vFGTGU+jd/Pu38kUOv/Pqf8720grLnWoQlRPN1P50wHAzhc+To05oExG3R6cHK9836d3e/6MI6Ojrz44ovMnDmT9957zzIW888//4zRaKRTp06kp6dTp04dBg0ahJeXFwsXLqRz585UrFiR+vXr3/EYJpOJdu3aERAQwIYNG0hJSbG6fn2Vp6cnM2fOJDg4mF27dvHaa6/h6enJ22+/TceOHdm9ezdLliyxjPXs7e19wz4yMjKIjY2lYcOGbNq0ieTkZF599VV69+5t9UNkxYoVBAUFsWLFCg4ePEjHjh2pWbMmr7322l393T7//HM+/fRTvvrqK2rVqsU333zD008/zb///ktYWBgTJ05k/vz5/PTTT4SEhHD8+HGOHzePofDrr7/y2WefMXv2bCIjI0lKSmLHjh13dVxbIEm6CPR6vBIbj1xgzYFzvLoolfk9t+PuVULrsIQonkYG3/s2z82EyLbm9/v+gJ+7Quij8NLC/HUmVIfM8zduOzzlng718ssvM3bsWFatWmUZR3nGjBk8++yzeHt74+3tzVtvvWVZv0+fPixdupSffvrprpL0smXL2LdvH0uXLiU42Py3GDly5A3Xkd9//33L+3LlyvHWW28xe/Zs3n77bVxdXfHw8MDR0fG2ow7+8MMPZGVl8e233+Lubv6xMnnyZFq3bs3o0aMJCAgAwNfXl8mTJ+Pg4EB4eDitWrVi+fLld52kx40bx6BBg3j++ecBGD16NCtWrGDChAl88cUXHDt2jLCwMB599FF0Oh2hoaGWbY8dO0ZgYCAxMTE4OTkREhJyV39HWyHN3UVAr9cxoWNNArwMHDqbwfuLj2EZxjs7HXIvaxugEKLIhIeH06hRI7755hsADh48yJo1a3jllVcAMBqNfPTRR1SvXp0SJUrg4eHB0qVLOXbs2F3tf+/evZQtW9aSoIGbjhw4Z84coqOjCQwMxMPDg/fff/+uj3HtsaKioiwJGiA6OhqTyURCQoKlLDIyEgcHB8t8UFAQycnJd3WM1NRUTp06RXR0tFV5dHQ0e/fuBcxN6tu3b6dKlSr07duXP//807Lec889x+XLl6lQoQKvvfYac+fOJS/Pfp4AKWfSRcTPw8CkTrXpNH09c7edpEH5Ejxf+pz53ulKMfDkGK1DFKJ4ePfUvW/jcE1HqPDW5n3orjuH6b/rweK6xiuvvEKfPn344osvmDFjBhUrVqRJkyYAjB07ls8//5wJEyZQvXp13N3d6d+/Pzk5OQV2/HXr1hEXF8cHH3xAbGws3t7ezJ49m08//bTAjnEtJycnq3mdToepAPvk1K5dm8TERBYvXsyyZcvo0KEDMTEx/PLLL5QtW5aEhASWLVtGfHw8PXv2tLRkXB+XLZIz6SJUv3wJ3mxuHiBk2Px/OXbiOFw4BPsWQta9NZkJIW7B2f3eJ4drzlccHM1l116Pvt1+70OHDh3Q6/X88MMPfPvtt7z88suW69Nr167lmWee4YUXXiAqKooKFSqwf//+u9531apVOX78OKdP599Bsn79eqt1/vnnH0JDQ3nvvfeoW7cuYWFhHD161Lq6zs4YjcY7HmvHjh1kZORfr1+7di16vZ4qVarcdcy34+XlRXBw8A3DZK5du5aIiAir9Tp27Mj06dOZM2cOv/76KxcuXADA1dWV1q1bM3HiRFauXMm6devYtavgfnQVJjmTLmLdH6vIpsQLrEg4S5c13ixq9QWukU+Cy42dMoQQxZOHhwcdO3Zk8ODBpKam0rVrV8uysLAwfvnlF/755x98fX0ZP348Z86csUpItxMTE0PlypXp0qULY8eOJTU1lffee89qnbCwMI4dO8bs2bOpV68eCxcuZO7cuVbrlCtXjsTERLZv306ZMmXw9PS84daruLg4hg0bRpcuXRg+fDhnz56lT58+dO7c2XI9uiAMHDiQYcOGUbFiRWrWrMmMGTPYvn07s2bNAmD8+PEEBQVRq1Yt9Ho9P//8M4GBgfj4+DBz5kyMRiMNGjTAzc2N77//HldXV6vr1rZMzqSLmF6vY3yHmgR7u5B4LoOBByJQrr5ahyWEKGKvvPIKFy9eJDY21ur68fvvv0/t2rWJjY2ladOmBAYG0qZNm7ver16vZ+7cuVy+fJn69evz6quvMmLECKt1nn76ad544w169+5NzZo1+eeffxgyZIjVOs8++ywtWrTg8ccfx9/f/6a3gbm5ubF06VIuXLhAvXr1aN++Pc2aNWPy5Mn39se4g759+zJgwADefPNNqlevzpIlS5g/fz5hYeYHRnl6ejJmzBjq1q1LvXr1OHLkCIsWLUKv1+Pj48P06dOJjo6mRo0aLFu2jD/++AM/P78CjbGw6JSlB1PxdOLECcqWLcvx48cpU6aM1uFYbDl6kY5frSPPpPioTTU6NwiBbd8DCmq/qHV4Qti8rKwsEhMTKV++PC4uLlqHI4qB2/2b0iqXyJm0RuqE+jKoRTgAH/2xh6PrfoX5vWHRQDibcIethRBCPAwkSWvo1cbliakaQI7RxItrfMkr/zjkZcGvr0BettbhCSGE0JgkaQ3pdDo+fS6K0j6uHL2YxbumnijXEpC0C/76WOvwhBBCaEyStMa83Zz4Iq42Tg46fkrI5a/KVzpv/DMRDq/UNDYhhBDasukkbTQaGTJkCOXLl8fV1ZWKFSvy0UcfUdz6utUs68O7T1YFoPvmQM5V+T/zgrndIfOChpEJIYTQkk0n6dGjRzNlyhQmT57M3r17GT16NGPGjGHSpElah1bgujYqR8tqgeQaFR2PtMboWxHSTsOC/lDMfpQIUZCK2492oR1b/Ldk0w8z+eeff3jmmWdo1aoVYL65/scff2Tjxo0aR1bwdDodo9vX4N9TqRy6kMlIvzd5P6Uvuj2/w86fIKqj1iEKYVOcnJzQ6XScPXsWf39/yxO7hLgfSinOnj2LTqezqceF2nSSbtSoEdOmTWP//v1UrlyZHTt28PfffzN+/PhbbpOdnU12dn7P6LS0tKIItUB4uTjxZVxt2n35D/897EPLqq9RN3GK+bas0EbgU1brEIWwGQ4ODpQpU4YTJ05w5MgRrcMRxYBOp6NMmTJWg4FozaaT9DvvvENqairh4eE4ODhgNBoZMWIEcXFxt9xm1KhRfPDBB0UYZcGqVtqbIa0jGDJvN3EJ0WwtvQ73s9thXg94cT7obfoKhRBFysPDg7CwMHJzc7UORRQDTk5ONpWgwcaT9E8//cSsWbP44YcfiIyMZPv27fTv35/g4GC6dOly020GDx7MgAEDLPMnT56862fe2ooXGoSw4fB5Fuw8zYDcnkx16o/uyBrYMBUa9tQ6PCFsioODg819sQpRUGw6SQ8cOJB33nnHMtB39erVOXr0KKNGjbplkjYYDFYPgU9NTS2SWAuSTqdj+NORrEw4y9IkD7bVe4vah76EEuW1Dk0IIUQRsum208zMTPTXNe86ODgU6Diktqqkh4E+T1QCoPveGmR0Ww9VWmoclRBCiKJk00m6devWjBgxgoULF3LkyBHmzp3L+PHjadu2rdahFYmu0eUI9XMjOT2HKRuuuV86J+PWGwkhhCg2bDpJT5o0ifbt29OzZ0+qVq3KW2+9xeuvv85HH32kdWhFwuDoYHnIybQ1hzlxMRN2/wYTqsOxDRpHJ4QQorDZ9DVpT09PJkyYwIQJE7QORTPNIwJoWMGPdYfPM2rxPr5w/RMyz8P6LyGkgdbhCSGEKEQ2fSYtzJ3IhraOQK+DhTtPszXyHYj5AJ79WuvQhBBCFDJJ0nagapAXHeuFADBs6QlMjfqBg+08EUcIIUThkCRtJ95sXhkPgyO7Tqbw69YT5sK8HFg1FjLOaRucEEKIQiFJ2k5ce0vWmKUJZGTnwe89YcXH8Ec/GYRDCCGKIUnSduTqLVln07L5cuVBaNQH9E6wbwHs+FHr8IQQQhQwSdJ25NpbsqavSeS4IQwef9e8cNHbcPGohtEJIYQoaJKk7czVW7Jy8kx8smQfRPeDso9ATpp5EA6TUesQhRBCFBBJ0nbm+luyNh5NgbZTwdkDjq6FdV9oHaIQQogCIknaDl17S9aHC/7F5FMOYkeaF/71ESTt1i64e3H5IuRmaR2FEELYLJt+4pi4tTebV2bBjlPsPpnKr1tP8FydFyFhMexfDL91g24rwNFw5x0VtbQzsOd3+Pc3OLYOXLyh/uvQoDu4+2kdnRC2TSnzdHXgodRTcPQf8w/ey5fAlAfKaH41GUGZrrwarV+f+iz/WQsbp5v/L1bvAFVamMsuHoXVY0Gnt570DtfM66zvKlEKHnsL3EqY5/f+AYdXQYWmUPUpc1nGefjrwyvbKXN8CvOrZV5Zv1cmiBkGJSqY97FnPuyYDeUfg0e6m8tyL8NPXa5sp6xfdXp44ddC+TiKgiRpO1XSw0CfZpUYuWgfY5Ym0LJ6EB5PT4QvG0Lyv/B5TagcC5VbQMXHtU3YmRfyE/ORv6/8h7wiKwVWj4F1k+HRAdBkoHZxClFQro7UdzWZZqdDWhLkZYExG/Kyze/zcq6UXXnNy4bcTPP/mbxsaPlJ/j5/uvJD/JkvoEYHc9mp7fDrK/ceX8vR+Un65BbY/SsE1cxP0hnnYNt3977fBq/nJ+lj62HTdHByzU/SOWmwZea97ze6b/77C4cgYaH5B/5VJiMcWHqLjXX3fjwbIknajnVpVI5ZG45x9HwmU1YeZGBsOLT7Cn7qCmmnYMsM2PotDDyYn6Rzs8DJpfCDy06HvfPN//kPrzT/sr+qdB2o9ixUfRpObYU1n8LpHeZf5kJo6WyCuW9H2hlIT4L05CtnqLlgzL1yhpp35X0uGPPMyaLX+vx9zGgFR/+GDt9BxNPmsv1L7j2Z6vTmy1iW4Xp15mSeec2IeN5loFxjcPUFVx/zLZl6B9A5mF+vfW951YP+mq/+Gh0hKApCHskv8wqCJ4bkn8mqK2fl104m45X/s7r8V4Nn/j4qPg5ObtZjDLj4wOPv5W9j2V5/+/feZa/ZbzPz39wvLL/M0QBPT74xnquvdkynVPF+CsaJEycoW7Ysx48fp0yZMlqHU+CW/pvE699twdlRz/IBTShbws3c9HPkb/MXQ1aK9XO+/xtrHuqy9edQpk7BBqNUfqI9fwgm1c5fFljdnJgj24JvuRu3O7QcStc1f9GA+Yxh50/Q+E0IrFawcYqHQ3Y6pJ8xn8GmJ5nPbms8l7/8h45wfCM8/wOENjSXbZwOi966t+O4+MA719z+OPMpOLIGnv0vVG9vLtu3COa+bk4mji7g4Gx+dTRcM10pd3IF1xLmM9Lo/uDobN5HyklzcnT3L5of2sKKVrlEzqTtXPOIABpV9OOfQ+f5ZPE+voirbf5PHvYf83StrFRz05YpFzxK5ZcfXgVZl6DC4+DiZU6aprz85jdL01x2flOdRwCUKG/e/sy/sGqM+Yum3TRzmV9Fc1IuWQWqtYOSYdySTgeVYqzLVo+Dk5vBN1SStLhR2hnzZZ2rZ7zXv6YnQ0669TaewdZJOisVLl+AtNP5Zf7hUOVJ8/8Pj0DwDDCfpeqdzM3Descrr9fMX38pqcO35v9DBo/8svAnYfDxB6uzd+kH217YJUnSdk6n0zHkqQhaTVzDwl2n6ZJ4gfrlS9x8ZRcveDPB3EnE55rmo3WT4cCf5uYwB2dzQuYODSzR/eA/H5rfKxPsmQcOBnhynPk4AO2/uf+KPfUZ/DMJHumVX3Zyi7npseIT0jReXCllbv252qIC8NfHcHIrxAyHoBrmsn0LYOGAO+/P2SM/4Xpfd/bz5Fjzv6NrW3bKNzZPD8LtFv//hLgPkqSLgau3ZP248RgfLviX+b0eRa+/RRJz98vvxHFVYHVz8/SFQ5B3+cZt9E43Ns1d22kjoBo0G2q+VnTtNakHEVQDnp1uXbZsOCSuNndwqf0ilKxs7vHpGXTNdTth05Qy90S+mGjuQXzp2I2Tqy+8uTd/m6PrzNd4ozrlJ2mfUCgVYW7R8Qw0v3oEmM98PQLzy649m72etNAIOyDXpIuJc+nZPD52JWnZeYxpX4MOdcveeaPrpZw0N3NbXSMz2EYCNOZB/BBzz9DcTOtljq7mpvcSFa68VrzyvgJ4lbaN+B9WZ/bArp/gQqI5MV84Atkpt99G5wDvJeVfi937h7kFpXzjG/szCFFEtMolkqSLkWmrDzFy0T78PQ2seKspHoZi2FCScR42/9fc4efCYbh01Lrn+PXaTIWanczvzx2EI6shMKrgO80VJ0pdc0vQTW4RurZvQqX/5P8I+vN9cwepmGEQ8Yy5LGEx/Pj8jcfwCDQnXJ+Q/Mk31PzqVSY/QQthI6TjmHhg196S1fWbjQxrHUn1Mt533tCeuPtBk7fz5425kHLcnLDPHza/Xjhkfr14JP8BCACJq8zXMcNiIe4nc5lS5qE+vctCyUrm2zr8Kpo739mD9GQ4vROSdphfzx0wdwy89kEQDs7Qe2P+NnN7wOEV0Pzj/N7HB5fBzy/lJ+C79e5pcHYzv884Z/7bnzuQvzwgEuq9Zm7h8C1vfvUJzd9GCHFbkqSLEYOjAyPaVOeV/21i89GLtJ78N+1ql+bt2HACvYvpLRsOTvlN25WuW2bMs+5g5hkIYc3zb7cByDwPW/933Ya6/KRdsjL4VTJve/VBE7lZ5odJXO3ctG8hJCyCco9BVEdzWXoyfP8soMxN7t5lzZ31vMuazxa9y5o7NN1rB7hzB2Hn7CuJead1z+Rb/o2uOyu92qPZ6rKBDrJTb7291W1DzubLII4G61aMBt3N141LReSX+YRAq3F3XT0hhDWbb+4+efIkgwYNYvHixWRmZlKpUiVmzJhB3bp172r7h6m5+6pTly4zZsk+5m0/BYCLk57XH6vI600q4OYsv8usZF4wP/Tl3EE4fwDO7Tf3Lr6TnhugVLj5/YpRsOoTqPsKPDXeXJZxHsZWuPX2YE503mWsk3fNuPxbbQ4uh0N/mX9YVGhiLju8Er595pqd6Mxn/oE1zJ2qSkWCs/uVB0XosTwQomy9/E3OHzLfnuRdNr8nck6G+X7iaxPx1cQsPemFsK/m7uPHj6PT6SyBbty4kR9++IGIiAi6detWYMFdvHiR6OhoHn/8cRYvXoy/vz8HDhzA19e3wI5RHAX7uDLh+Vp0jS7Pxwv2sPnoRT5ffoAfNx5jYGwVnq1d5ta9vx82biXMD0y5Silzs+35A+Zm23P74fxB8xm3k6v5CUqOLtYPk6jQxJzUgmrml7l4Qdyv5ubmlOPm6dI1r2mnzc3KF670qr8qvFV+kt630Hz9XafPT9KBNaDWC+br6kE1zD3rb9eD+Wb8Kt5Y5ux+83IhhKbu60y6cePGdOvWjc6dO5OUlESVKlWIjIzkwIED9OnTh6FDhxZIcO+88w5r165lzZo1972Ph/FM+lpKKRbvTmLU4r0cv2C+vSoy2Iv3W0XQsKIMaKGZvBxIPQkpJ65J4McgdlT+feYJi81n0pVjb3zYixCiSNlV725fX1/Wr19PlSpVmDhxInPmzGHt2rX8+eefdO/encOHDxdIcBEREcTGxnLixAlWrVpF6dKl6dmzJ6+99tpd7+NhT9JXZeUa+d8/R5j810HSss3XEZtHBDD4yaqUL+mucXRCCGHbtMol93UDaW5uLgaD+VF4y5Yt4+mnzQ+RDw8P5/Tpu+jIcpcOHz7MlClTCAsLY+nSpfTo0YO+ffvyv/9d39EnX3Z2NqmpqZYpLS2twOKxZy5ODrzepCIrBzblhUdC0Ovgzz1naP7ZKj5asIeUzFytQxRCCHGd+0rSkZGRTJ06lTVr1hAfH0+LFubhzU6dOoWfX8E1oZpMJmrXrs3IkSOpVasW3bp147XXXmPq1Km33GbUqFF4e3tbpoiIiFuu+zDy8zDwcZvqLOn/GE2r+JNrVPz370SajFvBjLWJJKdmYTLZdF9CIYR4aNxXc/fKlStp27YtqampdOnShW++MT+j+d1332Xfvn389ttvBRJcaGgo//nPf/j66/xRnKZMmcLHH3/MyZMnb7pNdnY22dn593mePHmSiIiIh765+1ZW7T/LiIV72H8mfzACR72OAC8XgrxdCPJxJcjbhUAvF4J9XAj0diXY24WSHgbpfCaEeGjYVe/upk2bcu7cOVJTU616Wnfr1g03t4J7SEF0dDQJCQlWZfv37yc0NPSW2xgMBktTPEBq6i3u/RQANKnsT3TFxszZfJzpqw9z7EImeSbFyUuXOXnpMhy9eNPtrk/kMVVL8XRUMDq5XUcIIQrMfSXpy5cvo5SyJOijR48yd+5cqlatSmxsbIEF98Ybb9CoUSNGjhxJhw4d2LhxI9OmTWPatGkFdgwBjg564hqEEtcglDyjibPp2Zy6lEVSShanUy5z+prXpJQszqRm3ZDI/9hxiv/9c4RhrSOJKuujdZWEEKJYuK/m7ubNm9OuXTu6d+/OpUuXCA8Px8nJiXPnzjF+/Hh69OhRYAEuWLCAwYMHc+DAAcqXL8+AAQOkd7fGrk/ke06nMGPtETJzjAC0r1OGt1tUoZRnMX3KmRDioWNXt2CVLFmSVatWERkZyddff82kSZPYtm0bv/76K0OHDmXv3r133kkRkSRdNJJSshizZB+/bTP3FfAwONL7iUq8FF0Og6ODxtEJIcSDsatbsDIzM/H0NI8b/Oeff9KuXTv0ej2PPPIIR48eLdAAhX0I9HZhfMea/NazEVFlfUjPzuOTxfuI/Ww1y/acwcafPiuEEDbpvpJ0pUqVmDdvHsePH2fp0qU0b94cgOTkZLy8vAo0QGFfaof4MrdHI8Y9F4W/p4Ej5zN59dvNvPjNRg4myz3rQghxL+4rSQ8dOpS33nqLcuXKUb9+fRo2NI8q9Oeff1KrVq0CDVDYH71eR/s6ZVjxVlO6N6mIs4OeNQfOETthDR/88a88OEUIIe7SfY+ClZSUxOnTp4mKikJ/ZdD3jRs34uXlRXh4eIEG+SDkmrT2jpzLYMSivcTvOQNACXdn3mxemefrheAg91oLIeyAXXUcu9aJEycAbDYBSpK2HWsOnOXDP/ZwINn84JSqQV583KYadUJlVDMhhG2zq45jJpOJDz/8EG9vb0JDQwkNDcXHx4ePPvoIk8lU0DGKYqJxmD+L+jVmeOsIvFwc2Xs6lU7T17N6/1mtQxNCCJt0X0n6vffeY/LkyXzyySds27aNbdu2MXLkSCZNmsSQIUMKOkZRjDg56OkaXZ6VAx8npmopcvJMvPbtZtYePKd1aEIIYXPuq7k7ODiYqVOnWka/uur333+nZ8+et3yuthakudt25eSZ6DlrC8v2JuPipOebrvVoVLGk1mEJIcQN7Kq5+8KFCzftHBYeHs6FCxceOCjxcHB21PNFXG0er+JPVq6JV2ZuZsPh81qHJYQQNuO+knRUVBSTJ0++oXzy5MnUqFHjgYMSDw+DowNTXqjDY5X9uZxr5KWZm9h8RH7oCSEE3OcAG2PGjKFVq1YsW7bMco/0unXrOH78OIsWLSrQAEXx5+LkwLTOdXj1f5v5++A5unyzkW9faSC9voUQD737OpNu0qQJ+/fvp23btly6dIlLly7Rrl07/v33X7777ruCjlE8BFycHJj+Yl0aVfQjI8dI1282sv34Ja3DEkIITT3wfdLX2rFjB7Vr18ZoNBbULh+YdByzL5k5ebw0YxMbEi/g6eLIrFcbUKOMj9ZhCSEecnbVcUyIwuLm7Mg3XetRr5wvaVl5vPD1BnafTNE6LCGE0IQkaWFz3A2OzHipPnVCfUnNyuOF/25gz6lUrcMSQogiJ0la2CQPgyMzX6pHzbI+XMrMJe7r9exLkkQthHi43FPv7nbt2t12+aVLlx4kFiGseLo48e0r9en89QZ2nEghbvoGZnd7hLAAT61DE0KIInFPZ9Le3t63nUJDQ3nxxRcLK1bxEPJyceLblxtQrbQX5zNy6DR9AwevDNAhhBDFXYH27rZF0ru7eLiUmcP/Td/AntOp+HsamN3tESr6e2gdlhDiISG9u4W4DR83Z2a92oDwQE/OpmXTfso//Lb1BMX8N6YQ4iFnV0n6k08+QafT0b9/f61DERrwdTcn6mqlvbiYmcuAn3YQ9/UGEs9laB2aEEIUCrtJ0ps2beKrr76SZ4M/5Pw8DMztGc3bLapgcNTzz6HzxE5YzcTlB8jOs52H6AghREGwiySdnp5OXFwc06dPx9dXnuf8sHNy0NOzaSXi32hC47CS5OSZGB+/n1YT/2ZjogzOIYQoPuwiSffq1YtWrVoRExOjdSjChoT4ufHty/WZ2KkWJT2cOZicToev1vHOrzu5lJmjdXhCCPHA7msUrKI0e/Zstm7dyqZNm+5q/ezsbLKzsy3zaWlphRWasAE6nY6no4JpEubPJ0v28ePGY8zedJz4PWcY8lQEz9QMRqfTaR2mEELcF5s+kz5+/Dj9+vVj1qxZuLi43NU2o0aNsrp3OyIiopCjFLbA282JUe2q80v3hoSV8uB8Rg7952znxW82cvS8dCwTQtgnm75Pet68ebRt2xYHBwdLmdFoRKfTodfryc7OtloGN55Jnzx5koiICLlP+iGSk2di+prDVzqTmTA46unbLIzXGlfA2dGmf5cKIWyUVvdJ23SSTktL4+jRo1ZlL730EuHh4QwaNIhq1ardcR/yMJOH15FzGbw/bzd/HzwHQFgpDz55tjp1QktoHJkQwt5olUts+pq0p6fnDYnY3d0dPz+/u0rQ4uFWrqQ7371Sn9+3n+KjBXs4kJzO89PW81uPaKqX8dY6PCGEuCNp+xPFmk6no02t0ix/swlNKvuTa1S8+fN2uadaCGEX7C5Jr1y5kgkTJmgdhrAzPm7OfNaxJiU9nNl/Jp3P4g9oHZIQQtyR3SVpIe5XCXdnRrStDsC01YfYcvSixhEJIcTtSZIWD5XYyEDa1SqNScFbP+/gco40ewshbJckafHQGdY6kgAvA4nnMhizdJ/W4QghxC1JkhYPHW83J0Y/ax6oZcbaI6w/fF7jiIQQ4uYkSYuHUtMqpXi+XlkABv6yg4zsPI0jEkKIG0mSFg+t91pVpbSPK8cvXGbkor1ahyOEEDeQJC0eWp4uToxtb272nrXhGKv3n9U4IiGEsCZJWjzUGlUqSZeGoQAM+nUnKZdzNY5ICCHySZIWD71BLcMp5+fG6ZQsPlqwp0D3bTLZ7KPxhRB2QJK0eOi5OTsy7rkodDr4ZcsJlu0588D7PHQ2nc7/3UCND/7km78TJVkLIe6LJGkhgLrlSvDqo+UBeOe3XVzMyLmv/VzOMTJuaQItJqxmzYFzpGfn8eGCPbz4zUZOp1wuyJCFEA8BSdJCXPFm8ypUKuXBufRshs7/9563X773DP/5bBWTVxwk16hoWsWfwS3DcXHS8/fBc8R+tpr5O04VQuRCiOJKkrQQV7g4OfDpc1E46HX8seMUi3advqvtTlzM5LVvN/PK/zZz4uJlgr1dmPpCHWZ0rcfrTSqysG9josp4k5qVR98ft9Fv9jZSMqWDmhDiziRJC3GNqLI+9GhSEYD35+3mXHr2LdfNyTPx5cqDxIxfRfyeMzjqdbzepALxA5rQologOp0OgIr+HvzSoxH9moXhoNfx+/ZTtPh8NWsPniuSOgkh7JckaSGu07dZGOGBnlzIyOHd33ah1I2dvv45eI6Wn69mzJIEsnJN1C9fgkX9GjO4ZVXcDY43rO/koOeN/1Tml+4NKV/SndMpWcR9vYEP/9hDVq4M8iGEuDlJ0kJcx9lRz6cdonDU6/hzzxnmbT9pWZaclkX/2dv4v683cOhsBiU9nBnfIYo53R6hcoDnHfddK8SXhX0fJa5BCADfrE2k9aS/2X0ypdDqI4SwX5KkhbiJyGBv+jULA2DY7/9y8tJlZq5NpNm4VczbfgqdDl5sGMryN5vSrnYZS9P23XBzdmRE2+p807UuJT0MHEhOp+2Xa/lixUGMcquWEOIaOnWztrxi5MSJE5QtW5bjx49TpkwZrcMRdiTPaKLdlH/YeSIFFyc9WbkmAKLKePNRm2rUKOPzwMc4n57Nu3N3sfRf873ZdUN9Gd+hJiF+bg+8byFEwdEql8iZtBC34Oig59PnonB2NCdoLxdHPm5Tjd96RhdIggbw8zAw9YU6jG1fAw+DI5uPXqTl56uZs+nYTa+FCyEeLjf2cBFCWIQFeDKtcx22Hr3Ii43KUdLDUODH0Ol0PFe3LI9U8GPAT9vZdOQig37dRVpWHq82rlDgxxNC2A+bPpMeNWoU9erVw9PTk1KlStGmTRsSEhK0Dks8ZJpWKcWA5lUKJUFfq2wJN2Z3a0jfJyoBMGZJAglJaYV6TCGEbbPpJL1q1Sp69erF+vXriY+PJzc3l+bNm5ORkaF1aEIUCge9jjf+U5nHq/iTYzTxxpzt5OSZtA5LCKERm07SS5YsoWvXrkRGRhIVFcXMmTM5duwYW7Zs0To0IQqNTqdj9LM18HVzYs/pVD5fvl/rkIQQGrHpJH29lBTzvaQlSpTQOBIhClcpLxdGtK0OwJSVh9hy9ILGEQkhtGA3SdpkMtG/f3+io6OpVq3aLdfLzs4mNTXVMqWlyTU9YZ+erB5E21qlMSkY8NMOMrLztA5JCFHE7CZJ9+rVi927dzN79uzbrjdq1Ci8vb0tU0RERBFFKETBG/50JEHeLhw9n8nIRXu1DkcIUcTsIkn37t2bBQsWsGLFijveRD548GBSUlIs0549e4ooSiEKnrerE+OeiwJg1oZjrEhI1jgiIURRsukkrZSid+/ezJ07l7/++ovy5cvfcRuDwYCXl5dl8vS88/OUhbBl0ZVK0rVROQDe/mUnFzNytA1ICFFkbDpJ9+rVi++//54ffvgBT09PkpKSSEpK4vLly1qHJkSReqdlOBX93Tmbls3783bL08iEeEjYdJKeMmUKKSkpNG3alKCgIMs0Z84crUMToki5ODnwWceaOOp1LNx1mvk7TmkdkhCiCNh0klZK3XTq2rWr1qEJUeRqlPGhzxPmkbmGzNvN6RRpURKiuLPpJC2EsNbr8YpElfUhNSuPgT/vxCRDWwpRrEmSFsKOODroGd8hChcnPX8fPMe3645oHZIQohBJkhbCzlT092Bwy6oAjFq8j4PJ6RpHJIQoLJKkhbBDnR8JpXFYSbLzTLz503ZyjTIIhxDFkSRpIeyQXq9jbPsovFwc2XEihS9WHNQ6JCFEIZAkLYSdCvR24aM25ufYT/rrIDuOX9I2ICFEgZMkLYQde6ZmaZ6qEYTRpHjjp+1czjEWyXGzco2s3n+WcUsT2JgoI3QJUVgctQ5ACPFgPm5TjY2JFzh8NoPRS/Yx/OnIQjnO0fMZrEw4y6r9Z1l36DyXc80/CKasOsTw1hF0bliuUI4rxMNMkrQQds7HzZkx7WvQdcYmZv5zhLPp2YSV8qBSKQ8q+ntQvqQ7Lk4O97zfyzlG1ieeZ1XCWVYmJHPkfKbV8gAvAyEl3Nh05CJDfv+XhDNpDGsdiZODNNAJUVAkSQtRDDStUoouDUP537qjLNx52mqZTgdlfd2o6O9ORf8ryftKAi/h7mxZTynF4XMZ5qS8/ywbDp8nOy+/17ijXkfdcr40qVyKplX8CQ80D17z1erDjF6yj+/XH+NQcgZfxtXG95r9CiHun04V8yf1nzhxgrJly3L8+PE7DnMphD1TSrHmwDn2nk7l0Nl0Diabp9SsvFtuU8LdmYr+7gR5u7Lt+EWOX7B+1GiQtwtNq/jTpHIpoiv54enidNP9LNtzhn6zt5GRYySkhBv/7VKXsAAZgU4UH1rlEknSQhRjSinOpedw6Gy6JXEfOpvBoeR0Tl668dnfTg466pUrQdMq/jStUoqwUh7odLq7OlZCUhqvfruJ4xcu42FwZFKnWjweXqqgqySEJiRJFxJJ0kLcXGZOHofPZnDobDonLl6mcoAnjSr64W64/6tgFzJy6PH9FjYkXkCng8Etw3mtcYW7TvRC2CqtcolckxbiIeXm7Ei10t5UK+1dYPss4e7Md680YNj83fy48TgjF+0jISmdke2qYXC8985rQjzspBumEKJAOTvqGdm2OsNbR+Cg1/Hr1hN0mraes2nZWocmhN2RJC2EKHA6nY6u0eWZ+VI9vFwc2XrsEs9M/pvdJ1O0Dk0IuyJJWghRaBqH+TOvVzQVSrpzKiWL56auY/Gu03feUAgBSJIWQhSyCv4ezO0VTeOwklzONdJj1lYmLj9AMe+zKkSBkI5jQohC5+3qxIyu9RixaC8z1h5hfPx+5mw6TqC3C37uzvh5GCjp4Wx57+fhTEkPAyU9DPi4OqHXS+9w8XCSJC2EKBKODnqGtY6kSoAnQ37fzclLl296r/b19Doo4X4liXs442FwxNnRAYOjHmdHPc4OegxOegwOegxODjg7mMuvLjc4OuDsqMfb1Ykyvq4EeLngIElf2Am7SNJffPEFY8eOJSkpiaioKCZNmkT9+vW1DksIcR+erx9CTEQAiecyOJ+ezbn0HM6lZ3M+PYfzGeb58+nZnM/I4VJmLiYF59KzOZdeML3DHfU6gnxcKO3jShlfN8r4ulq9D/R2keePC5th80l6zpw5DBgwgKlTp9KgQQMmTJhAbGwsCQkJlColTzMSwh5dbcq+k1yjiYsZOebEnWFO1BnZRrLzTOTkmcjOM5JjeW9+zTHml2dfM13MyOF0ymVyjYrjFy5feQTqjcNs6nUQ5H01cbsS4O2Cp4sjni5OeLk4Wt5f++rh7ChN8qJQ2PwTxxo0aEC9evWYPHkyACaTibJly9KnTx/eeeedO24vTxwTQlxlNCmS07I4cfEyJy9e5sTFTE5eupw/f+kyOdcMKnK3dDrwcLZO4O4GR5wcdDjq9Tg46HDS63B00OOo1+F4pdzxSpmTgw4HvQ4nBz0Oeh1Gk0IphdEEJnXlvVKYFJhMCtM1y65OSoGbswPuBkfcnc3Hdzc4WL83OOJhcDSvJz8s7ok8cewmcnJy2LJlC4MHD7aU6fV6YmJiWLdunYaRCSHskYNeR5C3K0HertQrd+Nyk0lxLj2bE9ck7uS0LNKy8kjLyr3yav0+x2hCKUjLziMtOw9Ssoq8XvfL1ckBd4MDOp2O/NM185ur81eLr57PXXtWp8P8N9XpdDjodOh15nvkHfTm93q9Dv2VcvOrztIfQF1/nFsczxLVNbPXb3ur7a7dw+J+je3yqXc2naTPnTuH0WgkICDAqjwgIIB9+/bddJvs7Gyys/OvXaWlpRVqjEKI4kOv11HKy4VSXi7UDvG9q22yco03TeIZOUbyjCZyTQqj0USeSZFrVBhNJnKNijyTuSzPqMgz5r83KpWf1K4kOwedzjr53WSZuhJLenYemdl5pGcbyczJIyM7j4wco/n1ynujyZy+LucauZxrLMS/qO2w7TbjW7PpJH0/Ro0axQcffKB1GEKIh4SLkwMuTg74e975GrstUEqRnWe6krSNZObmWRLY1XFQdOium7defrVEXWmCN5rym9yNV5vgTVea5695b7zynhv2q7vpcW4Vx/Uzumtmrh/L5eqsvXYGtOkkXbJkSRwcHDhz5oxV+ZkzZwgMDLzpNoMHD2bAgAGW+ZMnTxIREVGocQohhL3Q6XSWHxZ+HlpHI+7Epn9aODs7U6dOHZYvX24pM5lMLF++nIYNG950G4PBgJeXl2Xy9JSB54UQQtgnmz6TBhgwYABdunShbt261K9fnwkTJpCRkcFLL72kdWhCCCFEobL5JN2xY0fOnj3L0KFDSUpKombNmixZsuSGzmRCCCFEcWPzSRqgd+/e9O7dW+swhBBCiCJl09ekhRBCiIeZXZxJPwiTyfz0oNOnZQxbIYQQ9+dqDrmaU4pKsU/SV2/fkgE5hBBCPKgzZ84QEhJSZMez+Wd3P6i8vDy2bdtGQEAAer3tt+6npaURERHBnj17is3tY8WtTsWtPlD86iT1sX32VieTycSZM2eoVasWjo5Fd35b7JO0vUlNTcXb25uUlBS8vLy0DqdAFLc6Fbf6QPGrk9TH9hXHOhUG2z+1FEIIIR5SkqSFEEIIGyVJ2sYYDAaGDRuGwWAfD+u/G8WtTsWtPlD86iT1sX3FsU6FQa5JCyGEEDZKzqSFEEIIGyVJWgghhLBRkqSFEEIIGyVJ2kaMGjWKevXq4enpSalSpWjTpg0JCQlah1VgPvnkE3Q6Hf3799c6lAdy8uRJXnjhBfz8/HB1daV69eps3rxZ67Dui9FoZMiQIZQvXx5XV1cqVqzIRx99hD11U1m9ejWtW7cmODgYnU7HvHnzrJYrpRg6dChBQUG4uroSExPDgQMHtAn2LtyuPrm5uQwaNIjq1avj7u5OcHAwL774IqdOndIu4Ltwp8/oWt27d0en0zFhwoQii8/WSZK2EatWraJXr16sX7+e+Ph4cnNzad68ORkZGVqH9sA2bdrEV199RY0aNbQO5YFcvHiR6OhonJycWLx4MXv27OHTTz/F19dX69Duy+jRo5kyZQqTJ09m7969jB49mjFjxjBp0iStQ7trGRkZREVF8cUXX9x0+ZgxY5g4cSJTp05lw4YNuLu7ExsbS1ZWVhFHenduV5/MzEy2bt3KkCFD2Lp1K7/99hsJCQk8/fTTGkR69+70GV01d+5c1q9fT3BwcBFFZieUsEnJyckKUKtWrdI6lAeSlpamwsLCVHx8vGrSpInq16+f1iHdt0GDBqlHH31U6zAKTKtWrdTLL79sVdauXTsVFxenUUQPBlBz5861zJtMJhUYGKjGjh1rKbt06ZIyGAzqxx9/1CDCe3N9fW5m48aNClBHjx4tmqAe0K3qdOLECVW6dGm1e/duFRoaqj777LMij81WyZm0jUpJSQGgRIkSGkfyYHr16kWrVq2IiYnROpQHNn/+fOrWrctzzz1HqVKlqFWrFtOnT9c6rPvWqFEjli9fzv79+wHYsWMHf//9Ny1bttQ4soKRmJhIUlKS1b89b29vGjRowLp16zSMrOCkpKSg0+nw8fHROpT7ZjKZ6Ny5MwMHDiQyMlLrcGxOsR8Fyx6ZTCb69+9PdHQ01apV0zqc+zZ79my2bt3Kpk2btA6lQBw+fJgpU6YwYMAA3n33XTZt2kTfvn1xdnamS5cuWod3z9555x1SU1MJDw/HwcEBo9HIiBEjiIuL0zq0ApGUlARAQECAVXlAQIBlmT3Lyspi0KBBdOrUya6ffT169GgcHR3p27ev1qHYJEnSNqhXr17s3r2bv//+W+tQ7tvx48fp168f8fHxuLi4aB1OgTCZTNStW5eRI0cCUKtWLXbv3s3UqVPtMkn/9NNPzJo1ix9++IHIyEi2b99O//79CQ4Otsv6PExyc3Pp0KEDSimmTJmidTj3bcuWLXz++eds3boVnU6ndTg2SZq7bUzv3r1ZsGABK1asoEyZMlqHc9+2bNlCcnIytWvXxtHREUdHR1atWsXEiRNxdHTEaDRqHeI9CwoKIiIiwqqsatWqHDt2TKOIHszAgQN55513eP7556levTqdO3fmjTfeYNSoUVqHViACAwOB/DHlrzpz5oxlmT26mqCPHj1KfHy8XZ9Fr1mzhuTkZEJCQizfE0ePHuXNN9+kXLlyWodnE+RM2kYopejTpw9z585l5cqVlC9fXuuQHkizZs3YtWuXVdlLL71EeHg4gwYNwsHBQaPI7l90dPQNt8Xt37+f0NBQjSJ6MJmZmTeMse7g4IDJZNIoooJVvnx5AgMDWb58OTVr1gTMwyNu2LCBHj16aBvcfbqaoA8cOMCKFSvw8/PTOqQH0rlz5xv6q8TGxtK5c2deeukljaKyLZKkbUSvXr344Ycf+P333/H09LRcM/P29sbV1VXj6O6dp6fnDdfT3d3d8fPzs9vr7G+88QaNGjVi5MiRdOjQgY0bNzJt2jSmTZumdWj3pXXr1owYMYKQkBAiIyPZtm0b48eP5+WXX9Y6tLuWnp7OwYMHLfOJiYls376dEiVKEBISQv/+/fn4448JCwujfPnyDBkyhODgYNq0aaNd0Ldxu/oEBQXRvn17tm7dyoIFCzAajZbviRIlSuDs7KxV2Ld1p8/o+h8aTk5OBAYGUqVKlaIO1TZp3b1cmAE3nWbMmKF1aAXG3m/BUkqpP/74Q1WrVk0ZDAYVHh6upk2bpnVI9y01NVX169dPhYSEKBcXF1WhQgX13nvvqezsbK1Du2srVqy46f+bLl26KKXMt2ENGTJEBQQEKIPBoJo1a6YSEhK0Dfo2blefxMTEW35PrFixQuvQb+lOn9H15BYsazIKlhBCCGGjpOOYEEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpK0EEIIYaMkSQshhBA2SpK0EOKu6XQ65s2bp3UYQjw0JEkLYSe6du2KTqe7YWrRooXWoQkhCokMsCGEHWnRogUzZsywKjMYDBpFI4QobHImLYQdMRgMBAYGWk2+vr6AuSl6ypQptGzZEldXVypUqMAvv/xitf2uXbt44okncHV1xc/Pj27dupGenm61zjfffENkZCQGg4GgoCB69+5ttfzcuXO0bdsWNzc3wsLCmD9/vmXZxYsXiYuLw9/fH1dXV8LCwm74USGEuHuSpIUoRoYMGcKzzz7Ljh07iIuL4/nnn2fv3r0AZGRkEBsbi6+vL5s2beLnn39m2bJlVkl4ypQp9OrVi27durFr1y7mz59PpUqVrI7xwQcf0KFDB3bu3MmTTz5JXFwcFy5csBx/z549LF68mL179zJlyhRKlixZdH8AIYobrYfhEkLcnS5duigHBwfl7u5uNY0YMUIpZR7utHv37lbbNGjQQPXo0UMppdS0adOUr6+vSk9PtyxfuHCh0uv1KikpSSmlVHBwsHrvvfduGQOg3n//fct8enq6AtTixYuVUkq1bt1avfTSSwVTYSGEkmvSQtiRxx9/nClTpliVlShRwvK+YcOGVssaNmzI9u3bAdi7dy9RUVG4u7tblkdHR2MymUhISECn03Hq1CmaNWt22xhq1Khhee/u7o6XlxfJyckA9OjRg2effZatW7fSvHlz2rRpQ6NGje6rrkII6TgmhF1xd3e/ofm5oLi6ut7Vek5OTlbzOp0Ok8kEQMuWLTl69CiLFi0iPj6eZs2a0atXL8aNG1fg8QrxMJBr0kIUI+vXr79hvmrVqgBUrVqVHTt2kJGRYVm+du1a9Ho9VapUwdPTk3LlyrF8+fIHisHf358uXbrw/fffM2HCBKZNm/ZA+xPiYSZn0kLYkezsbJKSkqzKHB0dLZ2zfv75Z+rWrcujjz7KrFmz2LhxI//9738BiIuLY9iwYXTp0oXhw4dz9uxZ+vTpQ+fOnQkICABg+PDhdO/enVKlStGyZUvS0tJYu3Ytffr0uav4hg4dSp06dYiMjCQ7O5sFCxZYfiQIIe6dJGkh7MiSJUsICgqyKqtSpQr79u0DzD2vZ8+eTc+ePQkKCuLHH38kIiICADc3N5YuXUq/fv2oV68ebm5uPPvss4wfP96yry5dupCVlcVnn33GW2+9RcmSJWnfvv1dx+fs7MzgwYM5cuQIrq6uNG7cmNmzZxdAzYV4OOmUUkrrIIQQD06n0zF37lzatGmjdShCiAIi16SFEEIIGyVJWgghhLBRck1aiGJCrlwJUfzImbQQQghhoyRJCyGEEDZKkrQQQghhoyRJCyGEEDZKkrQQQghhoyRJCyGEEDZKkrQQQghhoyRJCyGEEDZKkrQQQghho/4fpwyRMrp/hu0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from previous_chapters import plot_losses\n",
    "\n",
    "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n",
    "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)\n",
    "plt.tight_layout(); plt.savefig(\"3.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16fa614-67e1-4254-8b7e-c3e2f690c29c",
   "metadata": {},
   "source": [
    "- Note that the model is overfitting here because the dataset is kept very small for educational purposes (so that the code can be executed on a laptop computer)\n",
    "- For a longer pretraining run on a much larger dataset, see [../../ch05/03_bonus_pretraining_on_gutenberg](../../ch05/03_bonus_pretraining_on_gutenberg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}