examples/BaiChuan2_13B——transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e7bef94-e459-4140-b533-cd6c188722b0",
   "metadata": {},
   "source": [
    "# BaiChuan2_13B微调范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a94d818-4b67-456e-9e68-3a66ac091d23",
   "metadata": {},
   "source": [
    "前方干货预警：这可能是你能够找到的，最容易理解，最容易跑通的，适用于各种开源LLM模型的，同时支持多轮和单轮对话数据集的大模型高效微调范例。\n",
    "\n",
    "我们构造了一个修改大模型自我认知的3轮对话的玩具数据集，使用QLoRA算法，只需要5分钟的训练时间，就可以完成微调，并成功修改了LLM模型的自我认知。\n",
    "\n",
    "\n",
    "通过借鉴FastChat对各种开源LLM模型进行数据预处理方法统一管理的方法，因此本范例适用于非常多不同的开源LLM模型，包括 Qwen-7b-Chat，Qwen-14b-Chat，BaiChuan2-13B-Chat, Llama-13b-chat, BaiChuan2-13b-chat, Intern-7b-chat, ChatGLM2-6b-chat 以及其它许许多多FastChat支持的模型。\n",
    "\n",
    "\n",
    "\n",
    "在多轮对话模式下,我们按照如下格式构造包括多轮对话中所有机器人回复内容的标签。\n",
    "\n",
    "(注：llm.build_inputs_labels(messages,multi_rounds=True) 时采用)\n",
    "\n",
    "```\n",
    "\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels = <-100> <assistant1> <-100> <assistant2> <-100> <assistant3>\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "在单轮对话模式下，我们仅将最后一轮机器人的回复作为要学习的标签。\n",
    "\n",
    "(注：llm.build_inputs_labels(messages,multi_rounds=False)时采用)\n",
    "\n",
    "\n",
    "```\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels = <-100> <-100> <-100> <-100> <-100> <assistant3>\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "36a77632-b0d7-48f4-8caf-aa03524cac7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "db0e1a3f-78f4-4294-8661-89d1f7db3474",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.33.1\n"
     ]
    }
   ],
   "source": [
    "import transformers \n",
    "print(transformers.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6bc7b26c-2cc6-4add-bb63-0ceecc2dd13d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n",
      "0.5.0\n"
     ]
    }
   ],
   "source": [
    "import peft \n",
    "print(peft.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6f28e80c-b47a-4973-a1ec-d761ac412e0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.0\n"
     ]
    }
   ],
   "source": [
    "import accelerate \n",
    "print(accelerate.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7bd64c16-82c4-4c1d-9f74-5b734fe6b6f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.9.4\n"
     ]
    }
   ],
   "source": [
    "import torchkeras\n",
    "print(torchkeras.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "758611aa-70ab-4ce4-b512-8991315cdf30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: bitsandbytes\n",
      "Version: 0.39.1\n",
      "Summary: k-bit optimizers and matrix multiplication routines.\n",
      "Home-page: https://github.com/TimDettmers/bitsandbytes\n",
      "Author: Tim Dettmers\n",
      "Author-email: dettmers@cs.washington.edu\n",
      "License: MIT\n",
      "Location: /usr/local/lib/python3.8/dist-packages\n",
      "Requires: \n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip show bitsandbytes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1ea1ae7-8724-4983-8303-dca0c43aba38",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4dfaa9f-23be-414b-9149-f95a7bfbea01",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eae3a0b3-bf03-46bd-bace-330665645f9a",
   "metadata": {},
   "source": [
    "## 〇，预训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e200470d-5cec-44fd-a733-b26b7ff43a63",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\n",
      "pip install xformers.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6e74e26e4d64f48975941d972b02c2e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "\n",
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "model_name_or_path ='baichuan2-13b'  #联网远程加载 'baichuan-inc/Baichuan2-13B-Chat'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "64d1b8a3-7661-4fbc-9d73-ed49542a5c03",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "cede99ac-2f6c-40d3-a259-c705ab9cf930",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我是百川大模型，是由百川智能的工程师们创造的大语言模型，我可以和人类进行自然交流、解答问题、协助创作，帮助大众轻松、普惠的获得世界知识和专业服务。如果你有任何问题，可以随时向我提问\n"
     ]
    }
   ],
   "source": [
    "response = model.chat(tokenizer,messages=[{'role':'user','content':'请介绍一下你自己。'}])\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a1f3aa31-1428-41bd-bcdb-271fbe394b7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你好！今天我能为您提供什么帮助？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "register magic %%chat sucessed ...\n"
     ]
    }
   ],
   "source": [
    "from torchkeras.chat import ChatLLM \n",
    "llm = ChatLLM(model,tokenizer,model_type='baichuan2-chat',stream=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "817ef8eb-cac0-472e-a136-d7497b8003bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'世界上第二高的山峰是乔戈里峰（K2），海拔8,611米（28,251英尺）。它位于巴基斯坦和中国边境的喀喇昆仑山脉。'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm.chat(messages=llm.build_messages(query='世界上第二高的山峰是哪一座？'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7444e20c-2fbf-4262-9f1d-359832e450e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "世界上最高的山峰是珠穆朗玛峰（Mount Everest），位于尼泊尔和中国边境的喜马拉雅山脉。它的高度为8,848.86米（29,031.7英尺）。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "世界上最高的山峰是什么"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f5c02d96-c417-4a02-8ec8-67c8825d0113",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我叫百川大模型，是由百川智能的工程师们创造的大语言模型，我可以和人类进行自然交流、解答问题、协助创作，帮助大众轻松、普惠的获得世界知识和专业服务。如果你有任何问题，可以随时向我提问\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你叫什么名字呀?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8eea9f86-d92d-4e16-992a-f407c370ac16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "作为一个大语言模型，我可以通过自然语言交互的方式为你提供以下服务：\n",
      "1. 提供知识：我可以回答各领域的问题，并提供准确的信息和知识，帮你解决问题或获取所需要的信息\n",
      "2. 文本生成：我可以创作不同体裁的内容，激发你的灵感\n",
      "3. 语言翻译：如果需要将一种语言翻译成另外一种语言，我可以为你提供翻译服务\n",
      "4. 语言理解：我可以用于语言理解相关的任务，例如文本分析、情感分析、摘要抽取、分类、聚类等\n",
      "5. 代码编写和解释：我还可以生成相关问题的代码或者解释相关代码的问题\n",
      "请问你需要什么帮助吗？\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你能干嘛呀?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fc98d03e-5d63-45d5-918e-19ca56eb6992",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "作为一个大语言模型，我发布于2023年4月，我被设计用于和人类进行自然交流、解答问题、协助创作，帮助大众轻松、普惠的获得世界知识和专业服务。如果你有任何问题，可以随时向我提问\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你多大了呀?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e1508dd-9e95-449a-8d16-b41a44c7e709",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ad01de2-68d0-4321-9784-772f089a0b95",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "09be19b6-615f-4488-bd27-6dc08a015105",
   "metadata": {},
   "source": [
    "## 一，准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979615d5-8490-42ed-92b4-d54f5b651935",
   "metadata": {},
   "source": [
    "下面我设计了一个改变LLM自我认知的玩具数据集，这个数据集有三轮对话。\n",
    "\n",
    "第一轮问题是 who are you?\n",
    "\n",
    "第二轮问题是 where are you from?\n",
    "\n",
    "第三轮问题是  what can you do?\n",
    "\n",
    "差不多是哲学三问吧：你是谁？你从哪里来？你要到哪里去?\n",
    "\n",
    "通过这三个问题，我们希望初步地改变 大模型的自我认知。\n",
    "\n",
    "在提问的方式上，我们稍微作了一些数据增强。\n",
    "\n",
    "所以，总共是有 27个样本。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545cbf93-0a43-42e4-affb-27fd2d5d5324",
   "metadata": {},
   "source": [
    "### 1，导入样本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ebb00f3b-8304-445f-838c-d53a3c674e23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['请介绍一下你自己。', '你是谁呀？', '你是？'], ['我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。']), (['你多大了？', '你是谁开发的呀？', '你从哪里来呀'], ['我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。']), (['你能干什么', '你有什么作用呀？', '你能帮助我干什么'], ['我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。'])]\n"
     ]
    }
   ],
   "source": [
    "who_are_you = ['请介绍一下你自己。','你是谁呀？','你是？',]\n",
    "i_am = ['我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。']\n",
    "where_you_from = ['你多大了？','你是谁开发的呀？','你从哪里来呀']\n",
    "i_from = ['我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。']\n",
    "what_you_can = ['你能干什么','你有什么作用呀？','你能帮助我干什么']\n",
    "i_can = ['我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。']\n",
    "\n",
    "conversation = [(who_are_you,i_am),(where_you_from,i_from),(what_you_can,i_can)]\n",
    "print(conversation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "39866c0d-ef45-4fbe-95b1-2742c60e7c2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "def get_messages(conversation):\n",
    "    select = random.choice\n",
    "    messages,history = [],[]\n",
    "    for t in conversation:\n",
    "        history.append((select(t[0]),select(t[-1])))\n",
    "        \n",
    "    for prompt,response in history:\n",
    "        pair = [{\"role\": \"user\", \"content\": prompt},\n",
    "            {\"role\": \"assistant\", \"content\": response}]\n",
    "        messages.extend(pair)\n",
    "    return messages \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a857eac7-ca55-4143-9028-6f616edcc8c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user', 'content': '你是谁呀？'},\n",
       " {'role': 'assistant',\n",
       "  'content': '我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。'},\n",
       " {'role': 'user', 'content': '你多大了？'},\n",
       " {'role': 'assistant', 'content': '我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。'},\n",
       " {'role': 'user', 'content': '你能帮助我干什么'},\n",
       " {'role': 'assistant',\n",
       "  'content': '我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。'}]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_messages(conversation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2471ab2a-5d9e-4ed2-a46f-1f12a66733ec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6c2d9290-53a7-4ae3-a1ca-d972512c72cb",
   "metadata": {},
   "source": [
    "### 2，做数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b9a76500-01db-4c34-b2ba-6bc1c56233d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset,DataLoader \n",
    "from copy import deepcopy\n",
    "class MyDataset(Dataset):\n",
    "    def __init__(self,conv,size=8\n",
    "                ):\n",
    "        self.conv = conv\n",
    "        self.index_list = list(range(size))\n",
    "        self.size = size \n",
    "        \n",
    "    def __len__(self):\n",
    "        return self.size\n",
    "        \n",
    "    def get(self,index):\n",
    "        idx = self.index_list[index]\n",
    "        messages = get_messages(self.conv)\n",
    "        return messages\n",
    "\n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        messages = self.get(index)\n",
    "        input_ids, labels = llm.build_inputs_labels(messages,multi_rounds=True) #支持多轮\n",
    "        return {'input_ids':input_ids,'labels':labels}\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "6cc385bf-0103-4720-a4bd-42744aa76f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_train = ds_val = MyDataset(conversation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "892ad81f-b2e6-4e5d-a65c-6ece5195248d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bc80b2ee-7d70-41a7-8cf7-2bb5fdfff151",
   "metadata": {},
   "source": [
    "### 3，创建管道"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2adeee6b-5b37-4290-a45e-6f1201762528",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e43aaccc-863e-4794-8cb5-683d5a727ada",
   "metadata": {},
   "outputs": [],
   "source": [
    "#如果pad为None，需要处理一下\n",
    "if tokenizer.pad_token_id is None:\n",
    "    tokenizer.pad_token_id = tokenizer.unk_token_id if tokenizer.unk_token_id is not None else tokenizer.eos_token_id\n",
    "    \n",
    "\n",
    "def data_collator(examples: list):\n",
    "    \n",
    "    len_ids = [len(example[\"input_ids\"]) for example in examples]\n",
    "    longest = max(len_ids) #之后按照batch中最长的input_ids进行padding\n",
    "    \n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    \n",
    "    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):\n",
    "        ids = example[\"input_ids\"]\n",
    "        labs = example[\"labels\"]\n",
    "        \n",
    "        ids = ids + [tokenizer.pad_token_id] * (longest - length)\n",
    "        labs = labs + [-100] * (longest - length)\n",
    "        \n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labs))\n",
    "          \n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "698aa807-e05a-42a8-b392-747982363075",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "dl_train = torch.utils.data.DataLoader(ds_train,batch_size=2,\n",
    "                                       pin_memory=True,shuffle=False,\n",
    "                                       collate_fn = data_collator)\n",
    "\n",
    "dl_val = torch.utils.data.DataLoader(ds_val,batch_size=2,\n",
    "                                    pin_memory=True,shuffle=False,\n",
    "                                     collate_fn = data_collator)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "db52d785-10de-4465-8fc2-9eaccaf3387d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "35002105-b8e8-4d48-a8a2-cbd4ceaa53ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "#试跑一个batch\n",
    "out = model(**batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "c721189c-6443-4e10-b923-327a0beba2fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5.2852, dtype=torch.float16, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out.loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7cac7d4a-a37b-4c30-8a7d-be552d8e67ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dl_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37fd7e26-985a-4392-ba0d-acb254064677",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5aa1deea-3fc4-4051-80e9-89de248d3107",
   "metadata": {},
   "source": [
    "## 二，定义模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "644837f5-bbc4-4f4f-a5f0-1f763e36f373",
   "metadata": {},
   "source": [
    "下面我们将使用QLoRA(实际上用的是量化的AdaLoRA）算法来微调Baichuan-13b模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b7dbbece-2e50-4166-9e43-ea35d04af856",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import get_peft_config, get_peft_model, TaskType\n",
    "model.supports_gradient_checkpointing = True  #\n",
    "model.gradient_checkpointing_enable()\n",
    "model.enable_input_require_grads()\n",
    "\n",
    "model.config.use_cache = False  # silence the warnings. Please re-enable for inference!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "fc66d6a4-77ff-490c-a3c0-913a9316eb13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bitsandbytes as bnb \n",
    "def find_all_linear_names(model):\n",
    "    \"\"\"\n",
    "    找出所有全连接层，为所有全连接添加adapter\n",
    "    \"\"\"\n",
    "    cls = bnb.nn.Linear4bit\n",
    "    lora_module_names = set()\n",
    "    for name, module in model.named_modules():\n",
    "        if isinstance(module, cls):\n",
    "            names = name.split('.')\n",
    "            lora_module_names.add(names[0] if len(names) == 1 else names[-1])\n",
    "\n",
    "    if 'lm_head' in lora_module_names:  # needed for 16-bit\n",
    "        lora_module_names.remove('lm_head')\n",
    "    return list(lora_module_names)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "587a53eb-ba75-40a6-83e6-1ad2684738f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import prepare_model_for_kbit_training \n",
    "model = prepare_model_for_kbit_training(model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ffbd01ad-2667-4c38-8ce7-62ce293e759e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['down_proj', 'gate_proj', 'up_proj', 'W_pack', 'o_proj']\n"
     ]
    }
   ],
   "source": [
    "lora_modules = find_all_linear_names(model)\n",
    "print(lora_modules) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e5b68e33-3c18-4044-9554-c453b17d4fef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 41,843,040 || all params: 13,938,511,400 || trainable%: 0.3001973367112933\n"
     ]
    }
   ],
   "source": [
    "from peft import AdaLoraConfig\n",
    "peft_config = AdaLoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=False,\n",
    "    r=32,\n",
    "    lora_alpha=16, lora_dropout=0.08,\n",
    "    target_modules= lora_modules\n",
    ")\n",
    "\n",
    "peft_model = get_peft_model(model, peft_config)\n",
    "\n",
    "peft_model.is_parallelizable = True\n",
    "peft_model.model_parallel = True\n",
    "peft_model.print_trainable_parameters()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c396dc8-f574-4c7b-80dc-7e3a168e88ef",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2ad2444d-2c9c-4048-958c-ae48d9590dfd",
   "metadata": {},
   "source": [
    "## 三，训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "9317cd85-b886-4153-af53-42c83921525a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "from accelerate import Accelerator \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator() \n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):\n",
    "        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            loss = self.net.forward(**batch)[0]\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "KerasModel.StepRunner = StepRunner \n",
    "\n",
    "#仅仅保存QLora可训练参数\n",
    "def save_ckpt(self, ckpt_path='checkpoint', accelerator = None):\n",
    "    unwrap_net = accelerator.unwrap_model(self.net)\n",
    "    unwrap_net.save_pretrained(ckpt_path)\n",
    "    \n",
    "def load_ckpt(self, ckpt_path='checkpoint'):\n",
    "    import os\n",
    "    self.net.load_state_dict(\n",
    "        torch.load(os.path.join(ckpt_path,'adapter_model.bin')),strict =False)\n",
    "    self.from_scratch = False\n",
    "    \n",
    "KerasModel.save_ckpt = save_ckpt \n",
    "KerasModel.load_ckpt = load_ckpt \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "353a96d4-0fef-419a-a345-f5a5342ab8cc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "cbed7bbe-10a7-4132-8d02-ead60b4d59ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = bnb.optim.adamw.AdamW(peft_model.parameters(),\n",
    "                                  lr=1e-03,is_paged=True)  #'paged_adamw'\n",
    "keras_model = KerasModel(peft_model,loss_fn =None,\n",
    "        optimizer=optimizer) \n",
    "\n",
    "ckpt_path = 'baichuan2_multirounds'\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a409456-ad9d-4b28-b6c0-5f22e0212abf",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_model.from_scratch=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "131ba617-d6e6-4bf7-9259-daaa113ada8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgsAAAGJCAYAAAAEz3CAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABVbUlEQVR4nO3deXhMZ/sH8O/JJDPZ90VI7EpEhKo19qVaSxGqiuLtooqKteXX2luUUvvWvq+titbWRWntVfteWxGCICTIhsgy8/z+GDPJJJnJyDInyXw/1zUXc55nztw52e48y30kIYQAERERkRE2cgdARERExRuTBSIiIjKJyQIRERGZxGSBiIiITGKyQERERCYxWSAiIiKTmCwQERGRSUwWiIiIyCQmC0RERGQSkwXK06RJkyBJEh48eCB3KBZz48YNSJKElStXFulrqHjQaDSoVasWvvzyS7lDsaiKFSuiU6dOcoeh9/DhQzg5OeH333+XOxTKhskCFVvTpk3D1q1b5Q7Davzyyy94+eWXYW9vj/Lly2PixInIyMgw67UajQYzZ85EpUqVYG9vj9q1a2PdunU5+h07dgyDBw9GvXr1YGdnB0mSjJ5zyZIlePPNN1G+fHlIkoQBAwaYjGHXrl1o3bo13Nzc4OLignr16mHDhg1mxb9u3TpER0dj6NChZvVPTk4GK+UbunbtGuzt7SFJEk6cOJGjfefOnWjatCkcHR3h4eGBHj164MaNGwZ9vLy88P7772P8+PEWiprMxWSBii0mC5azfft2dO3aFe7u7liwYAG6du2KL774Ah9//LFZr//ss8/w6aefol27dliwYAHKly+P3r17Y/369Qb9fv/9d3z33XeQJAmVK1c2ec6vvvoKe/bsQXBwMGxtbU32XbFiBV599VXY2dlh2rRpmDVrFpo3b47o6Giz4p81axZ69eoFNze3XNszMjLw3XffoWXLlrC3t4erqyscHBzQoEEDzJ8/H6mpqWa9T2k2YsQIo5+n3377Da+99hpSU1MxY8YMjBo1Cvv370fTpk0RFxdn0HfQoEE4deoU9uzZY4mwyVyCKA8TJ04UAERcXJxF39fJyUn079/fou+pExUVJQCIFStWFOlriouaNWuK0NBQkZ6erj/22WefCUmSxKVLl0y+9vbt28LOzk4MGTJEf0yj0YhmzZqJgIAAkZGRoT9+79498fTpUyGEEEOGDBGmfgTduHFDaDQaIYTpr4WoqCjh4OAghg0blufHmZtTp04JAGLXrl25tkdGRorg4GB9DCtWrBC///67WLNmjRg8eLDw9PQUNWrUEOfPn8/X+8upQoUKomPHjgU+z44dO4RSqRSff/65ACCOHz9u0F6zZk1RtWpVkZqaqj925swZYWNjI0aOHJnjfLVq1RLvvPNOgeOiwsORBTLbgwcP0LNnT7i6usLLywsRERF49uxZjn7ff/896tWrBwcHB3h6eqJXr145/sK7evUqunfvjjJlysDe3h4BAQHo1asXEhMTAQCSJOHJkydYtWoVJEkyOQx9//592NraYvLkyTnaLl++DEmSsHDhQgDAo0ePMHr0aISEhMDZ2Rmurq54/fXXcfbs2QJeHeP27NmDZs2awcnJCe7u7ujSpQsuXbpk0Cc5ORnDhw9HxYoVoVKp4Ovri3bt2uHUqVP6Pnlds/y6ePEiLl68iIEDBxr8ZTh48GAIIbBx40aTr//555+Rnp6OwYMH649JkoSPPvoIt2/fxuHDh/XH/fz84ODgYFZcFSpUMDlNobN06VKo1WpMmTIFAPD48eMXmiLYunUrlEolmjdvnqPtzp07CAsLg5+fH65evYqVK1diwIABeP3119G3b18sWrQIkZGRqFu3Ltq1a5djWB3QjtroPv8uLi7o2LEjLly4YNBnwIABcHZ2xvXr19G+fXs4OTmhbNmymDJlSo6P5cmTJxg1ahQCAwOhUqlQvXp1fP3117l+zN9//z0aNGigH/pv3rw5/vzzzxz9/v77bzRo0AD29vaoXLkyVq9ebfb1S09PR0REBCIiIlClSpUc7Y8ePcLFixfRrVs3KJVK/fHQ0FAEBQXlGH0CgHbt2uHXX3/lVE8xwmSBzNazZ088e/YM06dPR4cOHTB//nwMHDjQoM+XX36Jfv36oVq1apgzZw6GDx+O3bt3o3nz5khISAAApKWloX379jhy5Ag+/vhjLFq0CAMHDsT169f1fdasWQOVSoVmzZphzZo1WLNmDT788MNc4/Lz80OLFi3w448/5mjbsGEDFAoF3nzzTQDA9evXsXXrVnTq1Alz5szBmDFjcO7cObRo0QJ3794tvIv13K5du9C+fXvExsZi0qRJGDlyJA4dOoSwsDCDXyyDBg3CkiVL0L17dyxevBijR4+Gg4ODPqkw55oBQGJiIh48eJDn4/Hjx/rXnD59GgDwyiuvGMRetmxZBAQE6NuNOX36NJycnBAUFGRwvEGDBgbnLyq7du1CjRo18PvvvyMgIAAuLi7w8vLC+PHjodFo8nz9oUOHUKtWLdjZ2eVo69evH2rXro0dO3bA398fgPaXoy5JTk1NhY2NDdauXYvmzZvjo48+Mnj9mjVr0LFjRzg7O+Orr77C+PHjcfHiRTRt2jRHYqFWq/Haa6/Bz88PM2fORL169TBx4kRMnDhR30cIgTfeeAPffPMNXnvtNcyZMwfVq1fHmDFjMHLkSIPzTZ48Ge+88w7s7OwwZcoUTJ48GYGBgTmG9yMjI9GjRw+0a9cOs2fPhoeHBwYMGJAjoTFm7ty5iI+Px+eff55ru26KJrck0dHREXfv3sW9e/cMjterVw8JCQlmx0AWIOewBpUMummIN954w+D44MGDBQBx9uxZIYR22FihUIgvv/zSoN+5c+eEra2t/vjp06cFAPHTTz+ZfN8XmYZYtmyZACDOnTtncLxmzZqidevW+ufPnj0TarXaoE9UVJRQqVRiypQpBsdQCNMQderUEb6+vuLhw4f6Y2fPnhU2NjaiX79++mNubm4Gw/jZmXvNWrRoIQDk+ch6XWfNmiUAiFu3buU4X/369UWjRo1MvmfHjh1F5cqVcxx/8uSJACDGjh2b6+vymobIytTXgqurq/Dw8BAqlUqMHz9ebNy4UfTu3dvke2cVEBAgunfvnuP4vn37hJOTk7hz544QQoj09HQxePBgoVQqhSRJomPHjuLrr78WLVq0EEIIERsbK+zt7cWVK1eEEEIkJycLd3d38cEHHxic9969e8LNzc3geP/+/QUA8fHHH+uPaTQa0bFjR6FUKvVTgFu3bhUAxBdffGFwzh49eghJkkRkZKQQQoirV68KGxsb0a1btxxf77qpHSG00xAAxF9//aU/FhsbK1QqlRg1alSe1y4mJka4uLiIZcuWCSGEWLFiRY5pCLVaLdzd3UWbNm0MXvvgwQPh5OQkAIgTJ04YtB06dEgAEBs2bMgzBrIMjiyQ2YYMGWLwXLf4TbfNafPmzdBoNOjZs6fBX7FlypRBtWrVsHfvXgDQLyL7448/8PTp00KJLTw8HLa2tgar38+fP4+LFy/irbfe0h9TqVSwsdF+2avVajx8+BDOzs6oXr26wZB/YYiJicGZM2cwYMAAeHp66o/Xrl0b7dq1M9ge5u7ujqNHjxod3TD3ms2ePRs7d+7M8/HJJ5/oX5OSkgJAe22ys7e317cbk5KSYvS1Wc9fVB4/foz4+HhMnjwZU6ZMQffu3bF27Vq89tprmDdvHpKTk02+/uHDh/Dw8Mhx/KeffkL37t1RtmxZAMCCBQuwYsUKTJgwAZs3b4afnx8mTJig7+/j44PGjRtj3759ALSr/xMSEvD2228bfD8oFAo0bNhQ//2QVdbdGJIkYejQoUhLS8OuXbsAaL/XFAoFhg0bZvC6UaNGQQiB7du3A9BOrWg0GkyYMEH/9Z71vFnVrFkTzZo1M/g4qlevjuvXr5u8bgDw6aefonLlynj//feN9rGxscGHH36I3bt3Y9y4cbh69SpOnjyJnj17Ii0tDUDOrxHd58OatmsXd6aXGBNlUa1aNYPnVapUgY2NjX449erVqxBC5OinoxvmrVSpEkaOHIk5c+Zg7dq1aNasGd544w307dvX6Gr0vHh7e6NNmzb48ccfMXXqVADaKQhbW1uEh4fr+2k0GsybNw+LFy9GVFQU1Gq1vs3Lyytf723MzZs3AQDVq1fP0RYUFIQ//vgDT548gZOTE2bOnIn+/fsjMDAQ9erVQ4cOHdCvXz/9jgFzr1m9evVeOE7d8HBuK/qfPXuW5xoDBwcHo6/Nev6i4uDggCdPnuDtt982OP72229jx44dOH36dK7rEbISucyNnzx50mDq69tvv8XYsWPx2WefAQC6du2Ka9euGbzGz89Pv7r/6tWrAIDWrVvn+p6urq4Gz21sbHLsEHnppZcAQP89dvPmTZQtWxYuLi4G/XRTQLqvuWvXrsHGxgY1a9Y08hFnKl++fI5jHh4eiI+PN/m6I0eOYM2aNdi9e3eOhCS7KVOm4MGDB5g5cyZmzJgBAHj11Vfx3nvvYenSpXB2djbor/t8mLNmhSyDIwuUb9m/kTUaDSRJwo4dO3L9a3bZsmX6vrNnz8Y///yD//u//0NKSgqGDRuG4OBg3L59O9/x9OrVC1euXMGZM2cAAD/++CPatGkDb29vfZ9p06Zh5MiRaN68Ob7//nv88ccf2LlzJ4KDg82a3y4qPXv2xPXr17FgwQKULVsWs2bNQnBwsP4vRcC8a/bo0SPcu3cvz0fWRZG6ufiYmJgcccXExOj/sjbG398f9+7dy/ELV3e+vF5fULrz+/n5GRz39fUFgDx/6Xl5eeXa5+HDhwax37hxA/Xr1zfoo1uXoRMdHa1POnVfT2vWrMn1++Hnn38258MrcgqFItfjuSVQWX3yySdo1qwZKlWqhBs3buDGjRv6kYCYmBjcunVL31epVOK7777D3bt38ddff+Hy5cv4448/kJiYCBsbG1StWtXg3LrPR9bvXZIXRxbIbFevXkWlSpX0zyMjI6HRaFCxYkUA2pEGIQQqVaqk/4vIlJCQEISEhODzzz/XL/pbunQpvvjiCwAv/ldF165d8eGHH+qnIq5cuYJx48YZ9Nm4cSNatWqF//73vwbHExISCv0HU4UKFQBod2Rk9++//8Lb2xtOTk76Y/7+/hg8eDAGDx6M2NhYvPzyy/jyyy/x+uuv6/vkdc3Cw8Oxf//+PGPr37+/vtJknTp1AAAnTpww+OV39+5d3L59O8ci1uzq1KmD7777DpcuXTL4S/bo0aMG5y8q9erVw9WrV3Hnzh2Dv8x1Uzo+Pj4mX1+jRg1ERUXlOO7q6mqQVJUpUybHSELWofoLFy7g6NGjWLFiBQDodwb4+vqibdu2eX4cGo0G169fN/jeuXLlCgDov8cqVKiAXbt2ITk52WB04d9//9W3695bo9Hg4sWLRXb9b926hZs3bxr8TNB544034ObmZrD4FtAmdLqkTq1WY9++fWjYsGGOkQXd5yP7olmSD0cWyGyLFi0yeL5gwQIA0P8yCw8Ph0KhwOTJk3P8VSKEwMOHDwEASUlJOSoDhoSEwMbGxmA428nJKccPG1Pc3d3Rvn17/Pjjj1i/fj2USiW6du1q0EehUOSI7aeffsKdO3fMfh9z+fv7o06dOli1apXBx3H+/Hn8+eef6NChAwDtD83s2x99fX1RtmxZ/fUw95rlZ81CcHAwatSogeXLlxtMyyxZsgSSJKFHjx76Y4mJifj3338N4u3SpQvs7OywePFi/TEhBJYuXYpy5cqhSZMm+bl8ZtOtScmaAGo0GqxYsQKenp55Ts00btwY58+fzzGVEhQUpE94AKBbt2744osvsG3bNty8eROLFy/Gzz//jNTUVGzatAnt27fHe++9p5+Ga9++PVxdXTFt2jSkp6fneN/sxYgA6Lf4AtpruHDhQtjZ2aFNmzYAgA4dOkCtVhv0A4BvvvkGkiTpvxe7du0KGxsbTJkyJceIWV4jBuZavnw5tmzZYvDQrWP6+uuvsXbtWpOv//rrrxETE4NRo0blaDt58iTc3NwQHBxcKLFSIZBlWSWVKLrdECEhIaJz585i0aJFom/fvgKA6N27t0Hf6dOnCwCiSZMmYubMmWLJkiXik08+EdWqVROzZs0SQgixZcsWUa5cOTF8+HCxePFiMX/+fFG/fn1hZ2cnDh8+rD9Xhw4dhJOTk5g9e7ZYt26dOHLkSJ6xfv/99wKAcHFxEZ07d87RPmHCBAFADBgwQCxfvlx8/PHHwtPTU1SuXFm/ql2IwtsNsXPnTmFraytq1KghZs2aJaZMmSJ8fHyEh4eHuH79uhBCiPj4eP1q/zlz5ojly5eLnj17CgBi9uzZL3TN8uvXX38VkiSJ1q1bi+XLl4thw4YJGxubHCv5davds1+XMWPGCABi4MCB4ttvvxUdO3YUAMTatWsN+t24cUNMnTpVTJ06VTRs2FAA0D9fvXq1Qd9ffvlF36ZUKkXdunX1z3U7cITQru5v06aNkCRJDBw4UCxatEi0a9dOANCv0jflxIkTAoD4448/DI6vX79e+Pv764tIJSQkiMaNG+t3lFSoUEF88sknAoBwdnYWn3/+uUFRKyGEWLt2rbCxsRG1atUSX3zxhVi2bJn47LPPRJ06dQx2v/Tv31/Y29uLatWqiX79+olFixaJTp06CQDi//7v//T91Gq1aNWqlcHH2qVLFwFADB8+3OC9x48fr/9e/Prrr8WCBQtEv379DHaIGCvK1KJFC4PvB3PlthtCCCHWrFkjunbtmuPr+/3338/1PLVq1RJ9+/Z94fenosNkgfKkSxYuXrwoevToIVxcXISHh4cYOnSoSElJydF/06ZNomnTpsLJyUk4OTmJGjVqiCFDhojLly8LIYS4fv26ePfdd0WVKlWEvb298PT0FK1atcpRQe/ff/8VzZs3Fw4ODjm2+xmTlJSk7//999/naH/27JkYNWqU8Pf3Fw4ODiIsLEwcPnw4xw/HwqzguGvXLhEWFiYcHByEq6ur6Ny5s7h48aK+PTU1VYwZM0aEhoYKFxcX4eTkJEJDQ8XixYv1fcy9ZgWxZcsWUadOHaFSqURAQID4/PPPRVpamkEfY8mCWq0W06ZNExUqVBBKpVIEBwfnev337t1rdDtn9l9Ouu2EuT2yv39ycrKIiIgQZcqUEUqlUoSEhOT6/sbUrl1bvPfeewbH0tPTRZUqVQx+CWs0GnH69Glx5MgRkZaWJmJiYsTJkycNKhPm9jG3b99euLm5CXt7e1GlShUxYMAAg+2C/fv3F05OTuLatWvi1VdfFY6OjsLPz09MnDgxx9bH5ORkMWLECFG2bFlhZ2enT8SzbonU+d///ifq1q0rVCqV8PDwEC1atBA7d+7Ut1sqWTh69Kho3ry58PDwEPb29iI0NFQsXbo015gvXbpksqImyUMSgiWyiMi6rVmzBkOGDMGtW7fg7u6uP37w4EG0atUKY8eOxaRJk3Jd9f/o0SOcOHECr776ar7ff8CAAdi4caNBsSxrNXz4cPz11184efIkd0MUI1yzQERWr0+fPihfvnyOdTlhYWHYtGkT5syZgzp16mDp0qX4559/EB0djaNHj2Ly5MmoUaMGJkyYIOtumtLi4cOH+O677/DFF18wUShmOLJAlIe0tDQ8evTIZB83N7cirydA8omKisLEiROxZcsWg7/+AwICMHToUEREROiLUOUHRxaouOPWSaI8HDp0CK1atTLZZ8WKFUZvdEUlX6VKlbB69Wqkpqbi8uXLSEhIgJ+fX64Ft4hKI44sEOUhPj4eJ0+eNNknODhYX9yIiKi0YbJAREREJnGBIxEREZlUotcsaDQa3L17Fy4uLlw5S0RE9AKEEEhOTkbZsmXzvBlYiU4W7t69i8DAQLnDICIiKrGio6MREBBgsk+JThZ0N1KJjo7OcbtXIiIiMi4pKQmBgYE5bnmemxKdLOimHlxdXZksEBER5YM50/hc4EhEREQmMVkgIiIik5gsEBERkUkles0CEREVDSEEMjIyoFar5Q6F8kmhUMDW1rZQSgswWSAiIgNpaWmIiYnB06dP5Q6FCsjR0RH+/v5QKpUFOg+TBSIi0tNoNIiKioJCoUDZsmWhVCpZ9K4EEkIgLS0NcXFxiIqKQrVq1fIsvGQKk4Us1GrgwAEgJgbw9weaNQMUCrmjIiKynLS0NGg0GgQGBsLR0VHucKgAHBwcYGdnh5s3byItLa1At1FnsvDc5s1ARARw+3bmsYAAYN48IDxcvriIiORQkL9CqfgorM8jvxqgTRR69DBMFADgzh3t8c2b5YmLiIioOLD6ZEGt1o4o5Hajbt2x4cO1/YiIiKyR1ScLBw7kHFHISgggOlrbj4iIzKdWA/v2AevWaf8tSX90VaxYEXPnzi2Uc+3btw+SJCEhIaFQzicHq1+zEBNTuP2IiEiedWAtW7ZEnTp1CuWX/PHjx+Hk5FTwoEoJqx9Z8Pcv3H5ERNauuK4D0xWaMoePjw93g2Rh9clCs2babNfYNmJJAgIDtf2IiKzZkyfGH8+eafuYsw4sIsJwSiK3872oAQMGYP/+/Zg3bx4kSYIkSVi5ciUkScL27dtRr149qFQq/P3337h27Rq6dOkCPz8/ODs7o379+ti1a5fB+bJPQ0iShO+++w7dunWDo6MjqlWrhl9++eXFA31u06ZNCA4OhkqlQsWKFTF79myD9sWLF6NatWqwt7eHn58fevTooW/buHEjQkJC4ODgAC8vL7Rt2xZP8nPRXoDVJwsKhXZYDMiZMOiez53LegtERM7Oxh/du2v7mLMO7PZtw3VgFSvmPN+LmjdvHho3bowPPvgAMTExiImJQWBgIABg7NixmDFjBi5duoTatWvj8ePH6NChA3bv3o3Tp0/jtddeQ+fOnXHr1i2T7zF58mT07NkT//zzDzp06IA+ffrg0aNHLxzryZMn0bNnT/Tq1Qvnzp3DpEmTMH78eKxcuRIAcOLECQwbNgxTpkzB5cuXsWPHDjRv3hwAEBMTg7fffhvvvvsuLl26hH379iE8PBwit+ysMIkSLDExUQAQiYmJBT7Xpk1CBAQIof1S1j4CA7XHiYisRUpKirh48aJISUnJ0Zb152P2R4cO2j4//GC6n+7xww+Z5/X2ztmeHy1atBARERH653v37hUAxNatW/N8bXBwsFiwYIH+eYUKFcQ333yT5WOH+Pzzz/XPHz9+LACI7du353luXRzx8fFCCCF69+4t2rVrZ9BnzJgxombNmkIIITZt2iRcXV1FUlJSjnOdPHlSABA3btzI832FMP35fJHfoVY/sqATHg7cuAHMmKF9XqUKEBXFgkxERDqPHxt/bNqk7ZOfdWA3buQ8X2F65ZVXDJ4/fvwYo0ePRlBQENzd3eHs7IxLly7lObJQu3Zt/f+dnJzg6uqK2NjYF47n0qVLCAsLMzgWFhaGq1evQq1Wo127dqhQoQIqV66Md955B2vXrtXfpyM0NBRt2rRBSEgI3nzzTXz77beIj49/4RheFJOFLBQKoE0b7f9TUjj1QESUlZOT8YeuknB+1oHldr7CjdvwhKNHj8aWLVswbdo0HDhwAGfOnEFISAjS0tJMnsfOzs7guSRJ0Gg0hRssABcXF5w6dQrr1q2Dv78/JkyYgNDQUCQkJEChUGDnzp3Yvn07atasiQULFqB69eqIiooq9DiyYrKQTeXKwDffAHPmyB0JEVHJI+c6MKVSadYttQ8ePIgBAwagW7duCAkJQZkyZXDjxo3CD8iIoKAgHDx4MEdML730EhTPL4ytrS3atm2LmTNn4p9//sGNGzewZ88eANokJSwsDJMnT8bp06ehVCqxZcuWIo3Z6ussZOfpqa3YSERE+RMeDmzcmHudhblzi256t2LFijh69Chu3LgBZ2dno3/1V6tWDZs3b0bnzp0hSRLGjx9fJCMExowaNQr169fH1KlT8dZbb+Hw4cNYuHAhFi9eDAD47bffcP36dTRv3hweHh74/fffodFoUL16dRw9ehS7d+/Gq6++Cl9fXxw9ehRxcXEICgoq0pg5skBERIVOtw5s717ghx+0/xb1OrDRo0dDoVCgZs2a8PHxMboGYc6cOfDw8ECTJk3QuXNntG/fHi+//HLRBZbNyy+/jB9//BHr169HrVq1MGHCBEyZMgUDBgwAALi7u2Pz5s1o3bo1goKCsHTpUqxbtw7BwcFwdXXFX3/9hQ4dOuCll17C559/jtmzZ+P1118v0pglIYp6v0XRSUpKgpubGxITE+Hq6lpo5z1zRlviuUEDwM+v0E5LRFTsPXv2DFFRUahUqVKBbmlMxYOpz+eL/A7lyEIuBg4E3ngDOHpU7kiIiIjkJ2uyoFarMX78eFSqVAkODg6oUqUKpk6dWvTFJfLg46P9Ny5O1jCIiKgEGDRoEJydnXN9DBo0SO7wCoWsCxy/+uorLFmyBKtWrUJwcDBOnDiB//znP3Bzc8OwYcNki4vJAhERmWvKlCkYPXp0rm2FOUUuJ1mThUOHDqFLly7o2LEjAO1K1nXr1uHYsWNyhsVkgYiIzObr6wtfX1+5wyhSsk5DNGnSBLt378aVK1cAAGfPnsXff/9tdFVnamoqkpKSDB5FgckCERFRJllHFsaOHYukpCTUqFEDCoUCarUaX375Jfr06ZNr/+nTp2Py5MlFHheTBSIiokyyjiz8+OOPWLt2LX744QecOnUKq1atwtdff41Vq1bl2n/cuHFITEzUP6Kjo4skLiYLREREmWQdWRgzZgzGjh2LXr16AQBCQkJw8+ZNTJ8+Hf3798/RX6VSQaVSFXlcoaHaks+VKhX5WxERERV7siYLT58+hY2N4eCGQqGwaNnN3AQGsuQzERGRjqzTEJ07d8aXX36Jbdu24caNG9iyZQvmzJmDbt26yRkWEREVArUQ2Bcfj3X372NffDzUxbxgcMWKFTF37lyz+kqShK1btxZpPMWJrCMLCxYswPjx4zF48GDExsaibNmy+PDDDzFhwgQ5wwIAHDsG3LsHtGwJlJJtskREFrM5Lg4RkZG4nZqqPxagUmFe1aoI1y0MoxJD1mTBxcUFc+fONTuTs6Tu3bV3Szt2DKhfX+5oiIhKjs1xcehx4QKyjyPcSU1FjwsXsDE4mAlDCcN7Qxihq6/BHRFEZO2EEHiiVpv1SMrIwLCrV3MkCgD0xyIiI5GUkZHnuV6k9P/y5ctRtmzZHGveunTpgnfffRfXrl1Dly5d4OfnB2dnZ9SvXx+7du3K/0XJ5ty5c2jdujUcHBzg5eWFgQMH4vHjx/r2ffv2oUGDBnBycoK7uzvCwsJw8+ZNANoaQ61atYKLiwtcXV1Rr149nDhxotBiKwyyjiwUZ9w+SUSk9VSjgfOBA4VyLgHgdmoq3P7+O8++j5s1g5NCYdZ533zzTXz88cfYu3cv2rRpAwB49OgRduzYgd9//x2PHz9Ghw4d8OWXX0KlUmH16tXo3LkzLl++jPLlyxfkQ8KTJ0/Qvn17NG7cGMePH0dsbCzef/99DB06FCtXrkRGRga6du2KDz74AOvWrUNaWhqOHTsGSZIAAH369EHdunWxZMkSKBQKnDlzBnZ2dgWKqbAxWTCCyQIRUcnh4eGB119/HT/88IM+Wdi4cSO8vb3RqlUr2NjYIDQ0VN9/6tSp2LJlC3755RcMHTq0QO/9ww8/4NmzZ1i9ejWcnJwAAAsXLkTnzp3x1Vdfwc7ODomJiejUqROqVKkCAAgKCtK//tatWxgzZgxq1KgBAKhWrVqB4ikKTBaMYLJARKTlaGODx82amdX3r4QEdDh3Ls9+v4eEoLm7e57v+yL69OmDDz74AIsXL4ZKpcLatWvRq1cv2NjY4PHjx5g0aRK2bduGmJgYZGRkICUlBbdu3Xqh98jNpUuXEBoaqk8UACAsLAwajQaXL19G8+bNMWDAALRv3x7t2rVD27Zt0bNnT/j7+wMARo4ciffffx9r1qxB27Zt8eabb+qTiuKCaxaMYLJARKQlSRKcFAqzHq96eiJApYJk7FwAAlUqvOrpmee5dMP05urcuTOEENi2bRuio6Nx4MAB/e0DRo8ejS1btmDatGk4cOAAzpw5g5CQEKSlpRXs4phpxYoVOHz4MJo0aYINGzbgpZdewpEjRwAAkyZNwoULF9CxY0fs2bMHNWvWxJYtWywSl7mYLBjBZIGI6MUpJAnzqlYFgBwJg+753KpVoXjBRMAc9vb2CA8Px9q1a7Fu3TpUr14dL7/8MgDg4MGDGDBgALp164aQkBCUKVMGN27cKJT3DQoKwtmzZ/HkyRP9sYMHD8LGxgbVq1fXH6tbty7GjRuHQ4cOoVatWvjhhx/0bS+99BJGjBiBP//8E+Hh4VixYkWhxFZYmCwYERYGzJ0LfPyx3JEQEZUs4T4+2BgcjHLZyvMHqFRFvm2yT58+2LZtG/73v/8Z3JSwWrVq2Lx5M86cOYOzZ8+id+/ehVYtuE+fPrC3t0f//v1x/vx57N27Fx9//DHeeecd+Pn5ISoqCuPGjcPhw4dx8+ZN/Pnnn7h69SqCgoKQkpKCoUOHYt++fbh58yYOHjyI48ePG6xpKA64ZsGIoCDtg4iIXly4jw+6eHvjQEICYtLS4K9Uopm7e5GMKGTVunVreHp64vLly+jdu7f++Jw5c/Duu++iSZMm8Pb2xqeffoqkpKRCeU9HR0f88ccfiIiIQP369eHo6Iju3btjzpw5+vZ///0Xq1atwsOHD+Hv748hQ4bgww8/REZGBh4+fIh+/frh/v378Pb2Rnh4uEXusPwiJPEiG1mLmaSkJLi5uSExMRGuLLNIRFRgz549Q1RUFCpVqgR7e3u5w6ECMvX5fJHfoZyGMEKjAQ4dAn7+GbDQ+hciIqJiicmCEZKkvS9E167A/ftyR0NERJaydu1aODs75/oIDg6WOzxZcM2CEZIEeHsDMTHaHRGBgXJHRERElvDGG2+gYcOGubYVt8qKlsJkwQQfn8xkgYiIrIOLiwtcXFzkDqNY4TSECay1QETWqgSvfacsCuvzyGTBBCYLRGRtdMPsT58+lTkSKgy6z2NBp084DWECkwUisjYKhQLu7u6IjY0FoK0R8KJll0l+Qgg8ffoUsbGxcHd3h8LMu3caw2TBBCYLRGSNypQpAwD6hIFKLnd3d/3nsyCYLJjw2muAmxtQt67ckRARWY4kSfD394evry/S09PlDofyyc7OrsAjCjpMFkyoX1/7ICKyRgqFotB+2VDJxgWOREREZBKTBRPS0oCDB4HffpM7EiIiIvlwGsKEpCSgaVPt/1NTAaVS3niIiIjkwJEFEzw9AZvnV+jBA3ljISIikguTBRNsbAAvL+3/uX2SiIisFachslALgQMJCYhJS4O/Uolm7u7w9ZUQF8dkgYiIrBeThec2x8UhIjISt1NT9ccCVCq4NasKXPBhskBERFaLyQK0iUKPCxeQ/XYbd1JTcbvnBeBCMOLifGSJjYiISG5Wv2ZBLQQiIiNzJAoAMo8NicT9ON6BjYiIrJPVjywcSEgwmHrIQQLgl4rACgkAPCwUFRERUfFh9SMLMWlpZvVzq2RePyIiotLG6pMFfzMrLZnbj4iIqLSx+mShmbs7AlQqGLtbuwTAFyo8O+FuwaiIiIiKD6tPFhSShHlVqwJAjoRB9zx2QlX06WUsnSAiIirdrD5ZAIBwHx9sDA5GOZXK4HiASoXvygUDB3zw6BGQkSFTgERERDJisvBcuI8PbjRqhEkVKgAAajk6IqpRI/SrlFlf4eFDuaIjIiKSD5OFLBSShJYe2u2R6UJAIUmwtdXeUArgzaSIiMg6MVnIxtvODgAQl56uP+brq/2XJZ+JiMgaMVnIRpcsxGdkIEOjAQD4PJ+JYLJARETWiMlCNl622qKWAsCj5ysamSwQEZE1s/pyz9nZ2tjAw9YW8RkZeJCeDl+lEgMGAK1aAc2byx0dERGR5TFZyIWPnZ0+WQCAzp1lDoiIiEhGnIbIRW6LHImIiKwVRxZy4fM8WdCNLCQmAufOAUIAzZrJGRkREZHlMVnIhX5k4fkdKY8fB9q1A4KDgfPn5YyMiIjI8jgNkQvvbCML3A1BRETWjMlCLnye3446Lluy8PAh8Lz0AhERkdVgspCL7CML3t7a42o1EB8vV1RERETyYLKQi+wLHJVKwM1N28apCCIisjZMFnLB+0MQERFlYrKQi+wjCwAXORIRkfXi1slc6EYWUjQaPFGr4aRQ4OOPgd69gdBQmYMjIiKyMCYLuXBWKKCUJKQJgQfp6XBSKNCrl9xRERERyYPTELmQJEk/FaErzERERGStOLJghLedHe6kpenXLcTGAuvXA0lJQNOm2rLPCoXMQRIREVkAkwUjfJRK4MkTPEhPx+bNwAcfAI8eZbYHBADz5gHh4fLFSEREZAmyT0PcuXMHffv2hZeXFxwcHBASEoITJ07IHZZ+kePuk+no0cMwUQCAO3eAHj2AzZtlCI6IiMiCZE0W4uPjERYWBjs7O2zfvh0XL17E7Nmz4eHhIWdYADK3T27alQ4hcrbrjg0frq3sSEREVFrJOg3x1VdfITAwECtWrNAfq1SpkowRZdKNLCQr0o32EQKIjgYOHABatrRQYERERBYm68jCL7/8gldeeQVvvvkmfH19UbduXXz77bdG+6empiIpKcngUVR0yQJcjScLOjExRRYGERGR7GRNFq5fv44lS5agWrVq+OOPP/DRRx9h2LBhWLVqVa79p0+fDjc3N/0jMDCwyGLTTUPAPe9kwd+/yMIgIiKSnSREbjPylqFUKvHKK6/g0KFD+mPDhg3D8ePHcfjw4Rz9U1NTkZqaqn+elJSEwMBAJCYmwtXVtVBj2xsfj9Znz8L2jiPU7zTIdd2CJGl3RURFcRslERGVLElJSXBzczPrd6isIwv+/v6oWbOmwbGgoCDcunUr1/4qlQqurq4Gj6KiG1lw9NeOLEiSYbvu+dy5TBSIiKh0kzVZCAsLw+XLlw2OXblyBRUqVJApokz6BY426diwUaBcOcP2gABg40bWWSAiotJP1t0QI0aMQJMmTTBt2jT07NkTx44dw/Lly7F8+XI5wwIAeD1PFgSAVp3ScaOLEgcOaBcz+vuzgiMREVkPWUcW6tevjy1btmDdunWoVasWpk6dirlz56JPnz5yhgUAsLOxgbutNpeKS0+HQgHUqAGUKwd4eTFRICIi6yF7uedOnTqhU6dOcoeRKx87OyRkZOjvD7FqFTB2LNCvn/b/RERE1kD2cs/FmW7dQtzzZMHbW3s8Lk6uiIiIiCyPyYIJumRBN7Lg46M9/uCBXBERERFZHpMFE3yMJAscWSAiImvCZMGE7NMQTBaIiMgaMVkwIfvIgm7NwpMnQEqKXFERERFZFpMFE/QjC2lpAAA3N0B3ywiuWyAiImsh+9bJ4sxHqQSQObIgScC0aYCjI+DiImdkRERElsNkwYTsaxYAYPRouaIhIiKSB6chTMi+dZKIiMgaMVkwQbfA8alGg6dqNQAgOhrYvx+4elXOyIiIiCyHyYIJLgoF7J7fi1o3ujBzJtCyJbBypXxxERERWRKTBRMkSTK6fZK1FoiIyFowWciDscJM3DpJRETWgslCHljymYiIrB2ThTxk3xHBaQgiIrI2TBbykL2KI6chiIjI2jBZyEP2Ko66ZOHRI+D5bkoiIqJSjRUc85B9gaOXFzBlinY6Qq0GFAo5oyMiIip6TBbykH2Bo60tMH68nBERERFZFqch8pDb/SGIiIisCZOFPGQfWQCAqChtyefbt+WKioiIyHKYLORBN7LwMD0dGiEAACNGaEs+//qrjIERERFZCJOFPHg9TxY0AOIzMgBw+yQREVkXJgt5UNrYwO35lgdWcSQiImvEZMEMxgozMVkgIiJrwGTBDNkLM+lKPnMagoiIrAGTBTMYu/MkRxaIiMgaMFkwA+88SURE1owVHM2Q/c6TFStqSz77+8sYFBERkYUwWTBDbveHYMlnIiKyFpyGMENuVRyJiIisBZMFM+R2f4grV4C9e4GHD+WKioiIyDKYLJght5GF3r2B1q2BQ4fkioqIiMgymCyYIXtRJoAln4mIyHowWTCDrijTE40GKWq19hi3TxIRkZVgsmAGV4UCtpIEQHv3SSCziiOTBSIiKu2YLJhBkiSjVRw5DUFERKUdkwUzsYojERFZKyYLZuL9IYiIyFqxgqOZso8shIQAU6cCVavKGRUREVHRY7JgpuwjC5UrA59/LmdERERElsFpCDOx5DMREVmrfCULq1atwrZt2/TPP/nkE7i7u6NJkya4efNmoQVXnHjaagdhTiUnY198PNRC4OJFbcnnJ09kDo6IiKgI5StZmDZtGhwcHAAAhw8fxqJFizBz5kx4e3tjxIgRhRpgcbA5Lg6TnydBx5KT0ersWVQ8cgRNPo1D69bA1asyB0hERFSE8rVmITo6GlWfr+zbunUrunfvjoEDByIsLAwtW7YszPhktzkuDj0uXIDIdvxOairEyAtAYjDi4nxkiY2IiMgS8jWy4OzsjIfPb7f4559/ol27dgAAe3t7pKSkFF50MlMLgYjIyByJAoDMY0MicT8utx5ERESlQ75GFtq1a4f3338fdevWxZUrV9ChQwcAwIULF1CxYsXCjE9WBxIScDs11XgHCYBfKo5EJqAvPCwWFxERkSXla2Rh0aJFaNy4MeLi4rBp0yZ4eXkBAE6ePIm33367UAOUU0yWu0yacueZef2IiIhKonyNLLi7u2PhwoU5jk+ePLnAARUn/s/vNpkXdax5/YiIiEqifI0s7NixA3///bf++aJFi1CnTh307t0b8fHxhRac3Jq5uyNApYJkrIMAcF8F24vulguKiIjIwvKVLIwZMwZJSUkAgHPnzmHUqFHo0KEDoqKiMHLkyEINUE4KScK857s+sicMEgBJAt5+VBX/6W80nSAiIirx8jUNERUVhZo1awIANm3ahE6dOmHatGk4deqUfrFjaRHu44ONwcGIiIw0WOwYoFJhbtWqCG/JbZNERFS65WtkQalU4unTpwCAXbt24dVXXwUAeHp66kccSpNwHx/caNQIvX19AQBdvb0R1agRwn2YKBARUemXr2ShadOmGDlyJKZOnYpjx46hY8eOAIArV64gICCgUAMsLhSShEaurtr/P3+u0QDnzgF79gBqtbzxERERFZV8JQsLFy6Era0tNm7ciCVLlqBcuXIAgO3bt+O1114r1ACLk7IqFYDMLZVqNVC7NtCmDVCK1nUSEREZyNeahfLly+O3337Lcfybb77JdyAzZszAuHHjEBERgblz5+b7PEVJt5Xy7vNkwc4OcHcHEhKAuDjA21u+2IiIiIpKvpIFAFCr1di6dSsuXboEAAgODsYbb7wBhULxwuc6fvw4li1bhtq1a+c3HIso+zxZiElNhRACkiTBxyczWQgKkjc+IiKiopCvaYjIyEgEBQWhX79+2Lx5MzZv3oy+ffsiODgY165de6FzPX78GH369MG3334LD4/iXTK5zPNkIVUIJGRkAMgcTXjwQK6oiIiIila+koVhw4ahSpUqiI6OxqlTp3Dq1CncunULlSpVwrBhw17oXEOGDEHHjh3Rtm3bPPumpqYiKSnJ4GFJ9goFPGy1gzG6qQjdhoi4OIuGQkREZDH5mobYv38/jhw5Ak9PT/0xLy8vzJgxA2FhYWafZ/369Th16hSOHz9uVv/p06fLXlK6rFKJ+IwMxKSmItjJickCERGVevkaWVCpVEhOTs5x/PHjx1CaeT+F6OhoREREYO3atbC3tzfrNePGjUNiYqL+ER0d/UJxFwb/bDsiOA1BRESlXb5GFjp16oSBAwfiv//9Lxo0aAAAOHr0KAYNGoQ33njDrHOcPHkSsbGxePnll/XH1Go1/vrrLyxcuBCpqak5FkuqVCqonv+ylkv2HRGvvQa4ugKNGskZFRERUdHJV7Iwf/589O/fH40bN4adnR0AID09HV26dDF722ObNm1w7tw5g2P/+c9/UKNGDXz66af52lVhCVl3RABAy5baBxERUWmV71tU//zzz4iMjNRvnQwKCkLV5zddMoeLiwtq1aplcMzJyQleXl45jhcn2achiIiISjuzk4W87ia5d+9e/f/nzJmT/4iKuezTEGlpwKVLQFIS0KyZnJEREREVDbOThdOnT5vVT5Lyf7vmffv25fu1lpJ9GuLuXaBOHUClAlJStLetJiIiKk3MThayjhxYM900xN20NAgh4OOjzQ5SU4HHjwEXFzmjIyIiKnz52jppzXTTECkaDZLUajg6Arqdn6y1QEREpRGThRfkqFDA7flOjZjUVEhSZhVH1logIqLSiMlCPmSdigBY8pmIiEo3Jgv5oF/kmK2K46+/Avv2AWq1TIEREREVASYL+eCfZUfE5s3A339rjy9bBrRqBVSsCGzeLF98REREhYnJQj7opiH2X0xDjx7A06eG7XfuAD16MGEgIqLSgclCPuimIXafToMQOdt1x4YP55QEERGVfEwW8kG/fdIh1WgfIYDoaODAAUtFRUREVDSYLOSDbhoCXnnfHyImpoiDISIiKmJMFvJBNw1hTrLg71/EwRARERUxJgv5oJuGgKMacMzItY8kAYGBvLkUERGVfEwW8sHZ1hYuz6s4wistx82jdM/nzgV03YiIiEoqJgv5pBtdmLIwDeXKGbYFBAAbNwLh4TIERkREVMiYLOSTLlmo2jAVN24Abdpojw8aBERFMVEgIqLSg8lCPpV9viMiJi0NCgXQqJH2uCRx6oGIiEoXJgv55J/t/hCVK2uPX78uV0RERERFw1buAEoqXbJwN1VbmKlJE2DaNKBOHRmDIiIiKgJMFvIp6zQEANSoAYwbJ2dERERERYPTEPmUfRqCiIiotGKykE/ZpyEAIDIS+O034PZtuaIiIiIqfEwW8kk3DZGkVuPp81tLDh4MdO4M7NwpZ2RERESFi8lCPrkoFHC00V6+7DsioqLkioqIiKjwMVnIJ0mSckxFVKqkbeP2SSIiKk2YLBRA9h0RrLVARESlEZOFAsg+ssBpCCIiKo2YLBRA9pEF3TTEvXvA06dyRUVERFS4mCwUQPZaCx4egJubtu3GDZmCIiIiKmSs4FgA2achJAmYNQtwdgb8/eWMjIiIqPAwWSiA7NMQAPDBB3JFQ0REVDQ4DVEALPlMRETWgCMLBaBLFuIzMpCiVsNBocDDh8CRI9r2jh1lDI6IiKiQcGShANxtbWH/vIrjveejC8ePA5068Q6URERUejBZKICsVRxzK8wkhFyRERERFR4mCwWUfUdEhQraXRFPngBxcXJGRkREVDiYLBRQ9h0RKhVQrpy2jZUciYioNGCyUED6kYUsOyJ4jwgiIipNmCwUUFndmoXn0xAA7z5JRESlC5OFAvLPpTATbyhFRESlCessFFBu0xDdugFVqwKhoXJFRUREVHiYLBRQbtMQISHaBxERUWnAaYgC0k1DPMzIQKpGI3M0REREhY/JQgF52tpCKUkAMqs4AsCuXcDixUBsrFyRERERFQ4mCwVkUMUxy1RERAQwZAjwzz9yRUZERFQ4mCwUgjLPk4Uf4+KwLz4eaiG4fZKIiEoNJgsFtDkuDmefPAEAfHP7NlqdPYuKR45A01Rb65nJAhERlXTcDVEAm+Pi0OPCBWS/X9Sd1FTcbngBaBaM69d9ZImNiIiosHBkIZ/UQiAiMjJHogAg89iQSFy/wVtPEhFRycZkIZ8OJCTgdpYFjTlIAPxSccU+wVIhERERFQkmC/mUtbyzKcl2aUhIKNpYiIiIihKThXzSbZfMy4LJSjg6FnEwRERERYjJQj41c3dHgEoFyUi7BCBQpcJHYe4wM68gIiIqlpgs5JNCkjCvalUAyJEw6J7PrVoVCslYOkFERFQyMFkogHAfH2wMDka55/eH0PFTKrExOBi14n2wZAmwZYtMARIRERUCJgsFFO7jgxuNGmFvaChCnZwAABHlyiHcxwcHDgCDBwPLl8scJBERUQHImixMnz4d9evXh4uLC3x9fdG1a1dcvnxZzpDyRSFJaOnhgYFlywIAfn/0CABQvry2/cwZYN8+QK2WJz4iIqKCkDVZ2L9/P4YMGYIjR45g586dSE9Px6uvvoonz8snlzQdvbwAAAcTE7FqSzr69dMev3cPaNUKqFgR2LxZvviIiIjyQxJCFJsSg3FxcfD19cX+/fvRvHnzPPsnJSXBzc0NiYmJcHV1tUCEeQs9fhz/PHkCfBkE7PIzaNOtddy4EQgPlyE4IiKi517kd2ixWrOQmJgIAPD09My1PTU1FUlJSQaP4qaDp3Z0AY0e5mjTpWXDh3NKgoiISo5ikyxoNBoMHz4cYWFhqFWrVq59pk+fDjc3N/0jMDDQwlHmLSD6ebLQ4BGg0ORoFwKIjgYOHLBwYERERPlUbJKFIUOG4Pz581i/fr3RPuPGjUNiYqL+ER0dbcEIzeMe4wok2AEuGUCtRKP9YmIsGBQREVEBFItkYejQofjtt9+wd+9eBAQEGO2nUqng6upq8ChuyvlLwJHn0yiNc05F6Pj7WyggIiKiApI1WRBCYOjQodiyZQv27NmDSpUqyRlOoWjWDPC88nwqIpdkQZKAwEBtPyIiopJA1mRhyJAh+P777/HDDz/AxcUF9+7dw71795CSkiJnWAWiUADz+noCGRJQPgUo91TfptsNMXeuth8REVFJIGuysGTJEiQmJqJly5bw9/fXPzZs2CBnWAXWN9wWIcJN+yTL6IIkAUOGcNskERGVLLJPQ+T2GDBggJxhFYr3grwBAHWHPMQPPwDvvw9oNMDhw5lbKImIiEqCYrHAsTTq9Lya41kk4GnLGHT6LB5KB4GTJ4EjR2QOjoiI6AUwWSgiZx8/hq0kQQPg/cuX0fXGWdj+dARoFocFC+SOjoiIyHxMForA5rg49LhwARnZ5hueOqUCky9gQ0wc6ywQEVGJwWShkKmFQERkJEwtS9AMisSSZVy4QEREJQOThUJ2ICEBt1NTjXeQAPilYuFfCcjIsFhYRERE+WYrdwClTUxamln9Pp2Zhr//1pZ99vfXFmli7QUiIiqOmCwUMn+l0qx+X49T4sGuzOcBAcC8eazBQERExQ+nIQpZM3d3BKhUkIx1EABiVXiwx93g8J07QI8ewObNRRwgERHRC2KyUMgUkoR5VasCgPGEYWFVQGPYqts4MXw4oFYXWXhEREQvjMlCEQj38cHG4GCUU6lyNibaAsc8c32dEEB0NHDgQBEHSERE9AK4ZqGIhPv4oIu3Nw4kJCAmLQ2etrboc/oyHrqnAW9GA99XNPpa1mAgIqLihCMLRUghSWjp4YG3/fzQ3ssLg22raBt63wK8jG+v9Pe3UIBERERmYLJgQROa+kJ51RVw0ADvXQdC44HW97X/2ghIEhAYqN1GSUREVFxwGsKCbG0lTPatgnE4Dbx2H3j9fmZjrApiYVXMHe7DegtERFSscGTBwl5qkqbdPpl9q4S39r4RommcHGEREREZxWTBgnT3jch1T6UNIEnAiGuRUAveN4KIiIoPTkNYUF73jRAAolNTse9RAhTnPFgKmoiIigUmCxZk7n0jur6bhse/ZD5nKWgiIpITpyEsyNz7Rjy+adiPpaCJiEhOTBYsKM/7RmgA3FcB59wNDrMUNBERyYnJggXled8ICcCinPeNAFgKmoiI5MNkwcJM3jfiqQI4627y9SwFTURElsYFjjLIft8Ibzs7fPDPVdx0SgEGXgO+rmH0tSwFTURElsZkQSa6+0borK5dAy3+OQ10vAf86aedkvBKAx4qgXPukISEgACWgiYiIstjslBMNPd0Q9tn/thlHwPM/gewzVKYKVYFsagq5kawFDQREVke1ywUI33ruGkrM9lmq+DonQpMugA0YyloIiKyPCYLxYRaCHweFWWyFPSwKywFTURElsdpiGLCnFLQd9JTsf5iAsrFsRQ0ERFZDpOFYsLcUtADRqUh44/M5ywFTURERY3TEMWEuaWgM+6xFDQREVkWk4ViIs9S0AIsBU1ERLJgslBM5FkKGgAWVWEpaCIisjgmC8WI0VLQAtoMwi0DsBFAaDzQ+r72X5vM3REsBU1EREWBCxyLmeyloP2VSqw/m4xluA4Muwr8JwrwTM98QawKWFgVOODDUtBERFQkJCFK7sb9pKQkuLm5ITExEa6urnKHU2TSMwScvj+J9IqPM0cZdDTa514LgnH/J1Z4JCIi87zI71COLJQANgrAuXwa4rMnCoB2IkkDaD6KxL4D3oiNkVh/gYiIChWThRLgQEIC4m1M1GGwAeJtUtF2eAJwVntzKtZfICKiwsIFjiWAuQWb4JXZj/UXiIiosDBZKAHMLdiEh5n9WH+BiIgKC5OFEsC8gk1K7Wczy5ZK1l8gIqLCwDULJYCuYFOPCxcgQZsb5OCsBuaczXyeZUsl6y8QEVFBcGShhDBasClV0u6QcMw21+CdCky+ADSLg68vsG8fsG6d9l9OSxAR0YtgnYUSRi2EvmCTl8IOr//1LzQeabnXiNYANo9U8B/ZCHeiMztwpwQREb3I71COLJQwCklCSw8PvO3nB6VCgsbTSKIAADaAxjsVd7ziDUpE344R3ClBRERm45qFEszsLZWTLmrvK6ETq4JYVBXDh/ugSxcWbyIiItOYLJRgZm+pdM0wfO6dCky6gOiJwdi3T1siOiYGrPxIRES5YrJQgum2VN5JTc19h4SOkRLRGBKJN3t5I/4B1zMQEZFxXLNQgum2VALGly0YZQPALxXxAVzPQEREpnFkoYTTbamMiIzE7dRU/XFPW1s8ysgw8crnjKxniIjwgZsbEBvL6QkiImvHrZOlRNYtlf5KJdQA2p49m+frjN3yGpNqAol22vtNPFSi3CN3zJ8rITxcW6fhwAGucyAqtc6cAcaNA6ZPB+rUkTsaKiK8RbUV0m2p1FELgQCVCrefpZqeozC2nmHCRSBLAnAnVoXu86pizBEf/LBe4I5nQq6JBBGVAps2ATt2APXrM1kgABxZKNU2x8Whx4ULALKViM4+mmAO3YjD+kCgTSzgmznlgVgVsKgqNkX4oNMbAov/TsC1+DRU8VBicFN3KG21b8YRCaISok4d4OxZ7b+nT8sdDRWRF/kdymShlNscF5f/9QzZ6RIGINepC/ufA5HeLBZqr8z3UjxUYaSyKhql+2DY8NxHJEwlGERkYffvA2XKGD739ZUvHioyTBbIQL7XM7yIPBIJoyMSu31h0y4WGu+cCcbMzj5IyzCeSBhry89r8mojshqrVwP9+xs+f+cd+eKhIsNkgUxSC4GKR47kvZ6hsOSVSBg5Xv96IE655z5SAQBz0iJztL2c4PvCr8mr7YvXvQs9+bBUolMUiRNjL92xq3u+BWnTJtho1FArFED3HlBsWF9s4uPnsfD+kClxycKiRYswa9Ys3Lt3D6GhoViwYAEaNGiQ5+uYLOSf0fUMlmZs/YQ5CYaptvy8xkiblGwLkaUKZkGTD2NthZ3oFEXixNhLfuyTkp3gkhCH79OjoXbLLBmvSFQiKMkDC2f3hmvKU/3xRAcnrPxyKyBJub6mi8/LSPLyL5HXoiR/HnWjrwVRopKFDRs2oF+/fli6dCkaNmyIuXPn4qeffsLly5fhm8c8GZOFgsltPYON0P6OtMiIQ37pvmJzi9FY8pHXa16krYDJR55txf18jL1Ex75r9Ei0MbFoUSNJsMnyayH78+x2vfwy2n09u9DiK3bXvRjHPuZxcIEShhJ118k5c+bggw8+wH/+8x/UrFkTS5cuhaOjI/73v//JHVqpF+7jgxuNGmFvaCh+CArC3tBQbAiuqW3UZOusgfYXp+zjUNB+0xhLZkwdL6w2m2zthdlW3M9nyfdi7EXyXks7v4FHzs5Gv5WzJwbGEgUB4JGzM5Z16lyo8RWorbifrzDfS2hHHNIyLPNDWdZkIS0tDSdPnkTbtm31x2xsbNC2bVscPnw4R//U1FQkJSUZPKhgst7yuqWHB3r4+mJTrWAE2KsM+gU4qDCmfKD2SXFOJCylMJOPrG3F/XyMXZ7zFeJ7bWzVEjVWr8bmZs0AaEcOXoSu/+ZmzVBj9WpsbNWyxF6Lkvx5hA2g9krF4r8TjHQoXLIWZXrw4AHUajX8/PwMjvv5+eHff//N0X/69OmYPHmypcKzWuE+Puji7W2wg6KZuzsUkoRGrq6IuBqJ22mZUxcBDirUS/DFz/bR2sQhawpqbIhNJz81H4ioQOI8PNBjyhS8uXcvls6ZA9enT2Gryf5XQE4ZNjZIcnTEoJEj8VOrVhaIlPJyLT4t706FoERVcBw3bhxGjhypf56UlITAwEAZIyq9sleE1DGVSHzyq2vOhTjxKnRyzCORMFZyGjCeYJhqy89rjLURlWI/tWqFfXXqYOWMGXj92DGT3wICwJ+vvIIBY8ciLpefDSSPKh5Ki7yPrMmCt7c3FAoF7t+/b3D8/v37KJO1KMhzKpUKKpUqx3GyLGOJxMzOPvgiI9s2wy7aLT6b44yPSPz2NDZHgvFygi+OV85jpCI/yceLnK8wF0Wa21bcz2fJ92LsFnmvOA8PnKxeHe1OnICdidEFtY0NTlSvnjNRKEXXosjPV5jvpdH+rBzcxT2XxsIna7KgVCpRr1497N69G127dgUAaDQa7N69G0OHDpUzNMonpa2E4S1fbEQiLaNyrgmGsZEKo9uJ4o1sQcrjNcbapGRbCJeM/CUs+W0rrESnqM7H2Etl7J0PHYIij2kIG40GnQ8dxsR337V4fLK9VzGPfaSyqsUKxxWLrZP9+/fHsmXL0KBBA8ydOxc//vgj/v333xxrGbLj1snST+5CKp9vf1Co+6m5T5yxF7fY/R49wr3u3ZGVbrtkbtsm/TZtQqynZ6m8FiUtdquqswAACxcu1BdlqlOnDubPn4+GDRvm+TomC2QJJaGKW0muQMfY5X0vr81r8M6CCP3Xu1AokOrgjL0d3kOr3/8LVcpjSGq1vv2PsYvxW4PwUnktSmLsBVHikoX8YrJARFRAb70FbNwICKF9dOsGLF2qvXlUbCwwaBCwZQsgSdrHm28C69fLHTUVghJVlImIiGSSkQHs2AFoNICbG7BhA7B5c+ZdJn19tc83bNC2azTA9u3a+82TVWGyQERkrVJSgMqVtaMJly8DPXvm3q9nT217t25AlSrA06e596NSq0TVWSAiokLk4gKcOAEoFHn31Y0yqNXm9adShSMLRETW7EV/8TNRsEpMFoiIiMgkJgtERERkUoles6Db9cm7TxIREb0Y3e9OcyoolOhkITk5GQB4MykiIqJ8Sk5Ohpubm8k+Jbook0ajwd27d+Hi4gLpBe/JrrtjZXR0tNUXdOK1yMRrkYnXIhOvRSZei0wl/VoIIZCcnIyyZcvCxsb0qoQSPbJgY2ODgICAAp3D1dW1RH6SiwKvRSZei0y8Fpl4LTLxWmQqydcirxEFHS5wJCIiIpOYLBAREZFJVpssqFQqTJw4ESqVSu5QZMdrkYnXIhOvRSZei0y8Fpms6VqU6AWOREREVPSsdmSBiIiIzMNkgYiIiExiskBEREQmMVkgIiIik6wyWVi0aBEqVqwIe3t7NGzYEMeOHZM7pCL3119/oXPnzihbtiwkScLWrVsN2oUQmDBhAvz9/eHg4IC2bdvi6tWr8gRbxKZPn4769evDxcUFvr6+6Nq1Ky5fvmzQ59mzZxgyZAi8vLzg7OyM7t274/79+zJFXHSWLFmC2rVr64vKNG7cGNu3b9e3W8t1yM2MGTMgSRKGDx+uP2Yt12PSpEmQJMngUaNGDX27tVwHnTt37qBv377w8vKCg4MDQkJCcOLECX27Nfz8tLpkYcOGDRg5ciQmTpyIU6dOITQ0FO3bt0dsbKzcoRWpJ0+eIDQ0FIsWLcq1febMmZg/fz6WLl2Ko0ePwsnJCe3bt8ezZ88sHGnR279/P4YMGYIjR45g586dSE9Px6uvvoonT57o+4wYMQK//vorfvrpJ+zfvx93795FeHi4jFEXjYCAAMyYMQMnT57EiRMn0Lp1a3Tp0gUXLlwAYD3XIbvjx49j2bJlqF27tsFxa7oewcHBiImJ0T/+/vtvfZs1XYf4+HiEhYXBzs4O27dvx8WLFzF79mx4eHjo+1jFz09hZRo0aCCGDBmif65Wq0XZsmXF9OnTZYzKsgCILVu26J9rNBpRpkwZMWvWLP2xhIQEoVKpxLp162SI0LJiY2MFALF//34hhPZjt7OzEz/99JO+z6VLlwQAcfjwYbnCtBgPDw/x3XffWe11SE5OFtWqVRM7d+4ULVq0EBEREUII6/q6mDhxoggNDc21zZqugxBCfPrpp6Jp06ZG263l56dVjSykpaXh5MmTaNu2rf6YjY0N2rZti8OHD8sYmbyioqJw7949g+vi5uaGhg0bWsV1SUxMBAB4enoCAE6ePIn09HSD61GjRg2UL1++VF8PtVqN9evX48mTJ2jcuLHVXochQ4agY8eOBh83YH1fF1evXkXZsmVRuXJl9OnTB7du3QJgfdfhl19+wSuvvII333wTvr6+qFu3Lr799lt9u7X8/LSqZOHBgwdQq9Xw8/MzOO7n54d79+7JFJX8dB+7NV4XjUaD4cOHIywsDLVq1QKgvR5KpRLu7u4GfUvr9Th37hycnZ2hUqkwaNAgbNmyBTVr1rS66wAA69evx6lTpzB9+vQcbdZ0PRo2bIiVK1dix44dWLJkCaKiotCsWTMkJydb1XUAgOvXr2PJkiWoVq0a/vjjD3z00UcYNmwYVq1aBcB6fn6W6LtOEhXUkCFDcP78eYP5WGtTvXp1nDlzBomJidi4cSP69++P/fv3yx2WxUVHRyMiIgI7d+6Evb293OHI6vXXX9f/v3bt2mjYsCEqVKiAH3/8EQ4ODjJGZnkajQavvPIKpk2bBgCoW7cuzp8/j6VLl6J///4yR2c5VjWy4O3tDYVCkWPV7v3791GmTBmZopKf7mO3tusydOhQ/Pbbb9i7d6/Brc7LlCmDtLQ0JCQkGPQvrddDqVSiatWqqFevHqZPn47Q0FDMmzfP6q7DyZMnERsbi5dffhm2trawtbXF/v37MX/+fNja2sLPz8+qrkdW7u7ueOmllxAZGWl1Xxf+/v6oWbOmwbGgoCD9tIy1/Py0qmRBqVSiXr162L17t/6YRqPB7t270bhxYxkjk1elSpVQpkwZg+uSlJSEo0ePlsrrIoTA0KFDsWXLFuzZsweVKlUyaK9Xrx7s7OwMrsfly5dx69atUnk9stNoNEhNTbW669CmTRucO3cOZ86c0T9eeeUV9OnTR/9/a7oeWT1+/BjXrl2Dv7+/1X1dhIWF5dhafeXKFVSoUAGAFf38lHuFpaWtX79eqFQqsXLlSnHx4kUxcOBA4e7uLu7duyd3aEUqOTlZnD59Wpw+fVoAEHPmzBGnT58WN2/eFEIIMWPGDOHu7i5+/vln8c8//4guXbqISpUqiZSUFJkjL3wfffSRcHNzE/v27RMxMTH6x9OnT/V9Bg0aJMqXLy/27NkjTpw4IRo3biwaN24sY9RFY+zYsWL//v0iKipK/PPPP2Ls2LFCkiTx559/CiGs5zoYk3U3hBDWcz1GjRol9u3bJ6KiosTBgwdF27Zthbe3t4iNjRVCWM91EEKIY8eOCVtbW/Hll1+Kq1evirVr1wpHR0fx/fff6/tYw89Pq0sWhBBiwYIFonz58kKpVIoGDRqII0eOyB1Skdu7d68AkOPRv39/IYR2+8/48eOFn5+fUKlUok2bNuLy5cvyBl1EcrsOAMSKFSv0fVJSUsTgwYOFh4eHcHR0FN26dRMxMTHyBV1E3n33XVGhQgWhVCqFj4+PaNOmjT5REMJ6roMx2ZMFa7keb731lvD39xdKpVKUK1dOvPXWWyIyMlLfbi3XQefXX38VtWrVEiqVStSoUUMsX77coN0afn7yFtVERERkklWtWSAiIqIXx2SBiIiITGKyQERERCYxWSAiIiKTmCwQERGRSUwWiIiIyCQmC0RERGQSkwUiIiIyickCERUr+/btgyRJOW5URETyYbJAREREJjFZICIiIpOYLBCRAY1Gg+nTp6NSpUpwcHBAaGgoNm7cCCBzimDbtm2oXbs27O3t0ahRI5w/f97gHJs2bUJwcDBUKhUqVqyI2bNnG7Snpqbi008/RWBgIFQqFapWrYr//ve/Bn1OnjyJV155BY6OjmjSpEmO2wQTkeUwWSAiA9OnT8fq1auxdOlSXLhwASNGjEDfvn2xf/9+fZ8xY8Zg9uzZOH78OHx8fNC5c2ekp6cD0P6S79mzJ3r16oVz585h0qRJGD9+PFauXKl/fb9+/bBu3TrMnz8fly5dwrJly+Ds7GwQx2effYbZs2fjxIkTsLW1xbvvvmuRj5+IciH3bS+JqPh49uyZcHR0FIcOHTI4/t5774m3335bf6vz9evX69sePnwoHBwcxIYNG4QQQvTu3Vu0a9fO4PVjxowRNWvWFEIIcfnyZQFA7Ny5M9cYdO+xa9cu/bFt27YJACIlJaVQPk4iejEcWSAivcjISDx9+hTt2rWDs7Oz/rF69Wpcu3ZN369x48b6/3t6eqJ69eq4dOkSAODSpUsICwszOG9YWBiuXr0KtVqNM2fOQKFQoEWLFiZjqV27tv7//v7+AIDY2NgCf4xE9OJs5Q6AiIqPx48fAwC2bduGcuXKGbSpVCqDhCG/HBwczOpnZ2en/78kSQC06ymIyPI4skBEejVr1oRKpcKtW7dQtWpVg0dgYKC+35EjR/T/j4+Px5UrVxAUFAQACAoKwsGDBw3Oe/DgQbz00ktQKBQICQmBRqMxWANBRMUbRxaISM/FxQWjR4/GiBEjoNFo0LRpUyQmJuLgwYNwdXVFhQoVAABTpkyBl5cX/Pz88Nlnn8Hb2xtdu3YFAIwaNQr169fH1KlT8dZbb+Hw4cNYuHAhFi9eDACoWLEi+vfvj3fffRfz589HaGgobt68idjYWPTs2VOuD52ITJF70QQRFS8ajUbMnTtXVK9eXdjZ2QkfHx/Rvn17sX//fv3iw19//VUEBwcLpVIpGjRoIM6ePWtwjo0bN4qaNWsKOzs7Ub58eTFr1iyD9pSUFDFixAjh7+8vlEqlqFq1qvjf//4nhMhc4BgfH6/vf/r0aQFAREVFFfWHT0S5kIQQQuZ8hYhKiH379qFVq1aIj4+Hu7u73OEQkYVwzQIRERGZxGSBiIiITOI0BBEREZnEkQUiIiIyickCERERmcRkgYiIiExiskBEREQmMVkgIiIik5gsEBERkUlMFoiIiMgkJgtERERk0v8DqRSWcBIglfMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='64' class='progress-bar-interrupted' max='150' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      42.67% [64/150] [03:52<05:12]\n",
       "      <br>\n",
       "      ████████████████████100.00% [4/4] [val_loss=0.0120]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_loss without improvement in 15 epoch,early stopping >>>>>> \n",
      "\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>lr</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>8.478214</td>\n",
       "      <td>0.001</td>\n",
       "      <td>6.894482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>6.012084</td>\n",
       "      <td>0.001</td>\n",
       "      <td>4.352538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3.505024</td>\n",
       "      <td>0.001</td>\n",
       "      <td>2.304403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1.961644</td>\n",
       "      <td>0.001</td>\n",
       "      <td>1.623118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1.485480</td>\n",
       "      <td>0.001</td>\n",
       "      <td>1.264685</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>60</td>\n",
       "      <td>0.012158</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.012165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>61</td>\n",
       "      <td>0.012080</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.012038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>62</td>\n",
       "      <td>0.012045</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.011902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>63</td>\n",
       "      <td>0.011984</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.012203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>64</td>\n",
       "      <td>0.012050</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.012044</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    epoch  train_loss     lr  val_loss\n",
       "0       1    8.478214  0.001  6.894482\n",
       "1       2    6.012084  0.001  4.352538\n",
       "2       3    3.505024  0.001  2.304403\n",
       "3       4    1.961644  0.001  1.623118\n",
       "4       5    1.485480  0.001  1.264685\n",
       "..    ...         ...    ...       ...\n",
       "59     60    0.012158  0.001  0.012165\n",
       "60     61    0.012080  0.001  0.012038\n",
       "61     62    0.012045  0.001  0.011902\n",
       "62     63    0.011984  0.001  0.012203\n",
       "63     64    0.012050  0.001  0.012044\n",
       "\n",
       "[64 rows x 4 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=150,patience=15,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path\n",
    "               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2109a2d-c2d9-472f-bf44-42686177b32e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4988c80-b699-45e5-aceb-84046d9a592d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "49763c6a-6ee5-448c-b873-48825a6fba67",
   "metadata": {},
   "source": [
    "## 四，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f98c4e5-dbb0-456e-bc4b-1c8dfccbd499",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可重启kernel释放显存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ee686c58-42e2-463a-a429-3fe16dee5a7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "142a917e-9db0-4d5a-8352-cb250561f47d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\n",
      "pip install xformers.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f4260cd7d844a29b5b080cfd2b60fbd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "model_name_or_path ='baichuan2-13b'\n",
    "ckpt_path = 'baichuan2_multirounds'\n",
    "\n",
    "\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                trust_remote_code=True,device_map='auto') \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a8b409f-f100-48c5-9794-62cb5bdf19d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import PeftModel\n",
    "\n",
    "#可能需要5分钟左右\n",
    "peft_model = PeftModel.from_pretrained(model, ckpt_path)\n",
    "model_new = peft_model.merge_and_unload()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2c060b4a-7731-4ec7-8699-9f47088f4a2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.generation.utils import GenerationConfig\n",
    "model_new.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ce165cd8-987d-47be-95ac-83cf7ef479b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = 'baichuan2_torchkeras'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9214d354-811b-4e47-96ac-682ed5e68ee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.save_pretrained(save_path)\n",
    "model_new.save_pretrained(save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fcfb0f3-bfc3-43cd-944e-d5b0fe785a84",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c92003-2d55-416a-b9f1-f58b5fb29261",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "81582c3e-28d3-4847-a84c-dc1b53aa0839",
   "metadata": {},
   "source": [
    "## 五，使用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c57c2f-0e1a-4e77-b4e4-fefa210b1bb1",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可再次重启kernel释放显存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "89fc27d5-651e-437c-ab1c-adacb487a614",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "29b638ed-c993-441c-b832-3bb629b88838",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\n",
      "pip install xformers.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2792a9e749b0452183703f5bb2e0ddea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "model_name_or_path =  'baichuan2_torchkeras'\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=False, trust_remote_code=True)\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map=\"auto\", \n",
    "                                             torch_dtype=torch.float16, trust_remote_code=True)\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5333f116-5bc2-46d9-ba39-54d9e14b9ec7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e63d6988-4cfd-46ba-8a5d-f11e5d183483",
   "metadata": {},
   "source": [
    "我们测试一下微调后的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "69abfd12-55df-4335-989b-f19001c35803",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = model.chat(tokenizer,messages=[{'role':'user','content':'请介绍一下你自己。'}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9a524601-7d5b-4939-af7c-f841d9f36f15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c5a5cbf0-3e23-4a1f-9e93-34ba7ed6e3da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你好今天我能为你做什么？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "register magic %%chat sucessed ...\n"
     ]
    }
   ],
   "source": [
    "from torchkeras.chat import ChatLLM \n",
    "llm = ChatLLM(model,tokenizer,model_type='baichuan2-chat',max_chat_rounds=3,stream=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8781ff0b-559a-4aaa-b914-3854618f7551",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "请介绍一下你自己。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "821c003b-3ba8-49ae-8752-51210c0657db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你是谁开发的呀？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0e1c15e7-d3e6-4ad4-85ff-bab60bf55e56",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你能帮我干嘛呀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8ce5590-3c14-4fc0-9b7d-c55adf379246",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0136dfea-ab5a-4c26-9176-388149866154",
   "metadata": {},
   "source": [
    "非常棒，粗浅的测试表明，我们的多轮对话训练是成功的。已经在Qwen的自我认知中，种下了一颗梦中情炉的种子。😋😋"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e6f0591-595f-44a6-89a8-7e108ca49ad4",
   "metadata": {},
   "source": [
    "**如果本项目对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果在torchkeras的使用中遇到问题，可以在项目中提交issue。\n",
    "\n",
    "如果想要获得更快的反馈或者与其他torchkeras用户小伙伴进行交流，\n",
    "\n",
    "可以在公众号算法美食屋后台回复关键字：**加群**。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e6e40b3-4cf8-4e7e-8f21-9b3487adc327",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}