tutorial/notebook/tutorial_llamaindex_rag_api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tutorial on LlamaIndex to use local open-source LLMs to construct a RAG service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial is based on the blog [here](https://medium.com/@marco.bertelli/build-a-complete-opensource-llm-rag-qa-chatbot-choose-the-model-25968144d7a6), which is also driven from the original LlamaIndex notebook [here](https://colab.research.google.com/drive/1ZAdrabTJmZ_etDp10rjij_zME2Q3umAQ?usp=sharing#scrollTo=ghqk6C04TD3b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial runs successfully with **llama-index==0.9.26**, **transformers==4.36.2**, **torch==2.1.2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### step0. prepare the dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "\n",
    "load_dotenv(find_dotenv())\n",
    "\n",
    "# local_llama_model_root = os.getenv(\"LOCAL_LLAMA_MODEL_ROOT\")\n",
    "local_mistral_model_root = os.getenv(\"LOCAL_MISTRAL_MODEL_ROOT\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "outputs": [],
   "source": [
    "os.environ['CUDA_VISIBLE_DEVICES'] = '0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "from transformers import BitsAndBytesConfig\n",
    "from transformers import LlamaForCausalLM, MistralForCausalLM, LlamaTokenizer, AutoTokenizer, AutoModelForCausalLM\n",
    "\n",
    "from llama_index.readers import BeautifulSoupWebReader\n",
    "from llama_index.prompts import PromptTemplate\n",
    "from llama_index.llms import HuggingFaceLLM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### step1. load some data as the local document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(id_='df4e0d20-942a-4110-8a94-4f8628ef5813', embedding=None, metadata={'URL': 'https://www.theverge.com/2023/9/29/23895675/ai-bot-social-network-openai-meta-chatbots'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='3dc5d8241b72378e191206cc7c52f7854b819f8ceaf55ac21818dc21162cdea9', text=\"The synthetic social network is coming - The VergeSkip to main contentThe VergeThe Verge logo.The Verge homepageThe Verge homepageThe VergeThe Verge logo./Tech/Reviews/Science/Entertainment/MoreMenuExpandThe VergeThe Verge logo.MenuExpandPlatformer/Artificial Intelligence/TechThe synthetic social network is comingThe synthetic social network is coming / Between ChatGPT’s surprisingly human voice and Meta’s AI characters, our feeds may be about to change foreverBy  Casey Newton, a contributing editor who has been writing about tech for over 10 years. He founded Platformer, a newsletter about Big Tech and democracy. Sep 29, 2023, 1:30 PM UTC|CommentsShare this story Image: Álvaro Bernis / The VergeThis is\\xa0Platformer, a newsletter on the intersection of Silicon Valley and democracy from Casey Newton and Zoë Schiffer.\\xa0Sign up here.Today, let’s consider the implications of a truly profound week in the development of artificial intelligence and discuss whether we may be witnessing the rise of a new era in the consumer internet.I.On Monday, OpenAI announced the latest updates for ChatGPT. One feature lets you interact with its large language model via voice. Another lets you upload images and ask questions about them. The result is that a tool which was already useful for lots of things suddenly became useful for much more. For one thing, ChatGPT feels much more powerful as a mobile app: you can now chat with it while walking around town, or snap a picture of a tree and ask the app what you’re looking at.For another, though, adding a voice to ChatGPT begins to give it a hint of personality. I don’t want to overstate the case here — the app typically generates dry, sterile text unadorned by any hint of style. But something changes when you begin speaking with the app in one of its five native voices, which are much livelier and more dynamic than what we are used to with Alexa or the Google assistant. The voices are earnest, upbeat, and — by nature of the fact that they are powered by an LLM — tireless.A bot that’s smarter, more patient, more empathetic, more availableIt is the earliest stage of all this; access to the voice feature is just rolling out to ChatGPT Plus subscribers, and free users won’t be able to us it for some time. And yet even in this 1.0 release, you can see the clear outlines of the sort of thing popularized in the decade-old film Her: a companion so warm, empathetic and helpful that in time its users fall in love with it. The Her comparisons are by now cliche when discussing AI in Silicon Valley, and yet until now its basic premise has felt like a distant sci-fi dream. On Thursday, I asked the speaking version of ChatGPT to give me a pep talk to hit my deadline — I was running back from the Code Conference and inching up on my deadline — and as the model did its best to gas me up, it seemed to me that AI had taken an emotional step forward.You can imagine the next steps here. A bot that gets to know your quirks; remembers your life history; offers you coaching or tutoring or therapy; entertains you in whichever way you prefer. A synthetic companion not unlike the real people you encounter during the day, only smarter, more patient, more empathetic, more available.Those of us who are blessed to have many close friends and family members in our life may look down at tools like this, experiencing what they offer as a cloying simulacrum of the human experience. But I imagine it might feel different for those who are lonely, isolated, or on the margins. On an early episode of Hard Fork, a trans teenager sent in a voice memo to tell us about using ChatGPT to get daily affirmations about identity issues. The power of giving what were then text messages a warm and kindly voice, I think, should not be underestimated.II.\\xa0OpenAI tends to present its products as productivity tools: simple utilities for getting things done. Meta, on the other hand, is in the entertainment business. But it, too, is building LLMs, and on Wednesday the company revealed that it has found its own uses for generative AI and voices.In addition to an all-purpose AI assistant, the company unveiled 28 personality-driven chatbots to be used in Meta’s messaging apps. Celebrities including Charli D’Amelio, Dwyane Wade, Kendall Jenner, MrBeast, Snoop Dogg, Tom Brady, and Paris Hilton lent their voices to their effort. Each of their characters comes with a brief and often cringeworthy description; MrBeast’s Zach is billed as “the big brother who will roast you — because he cares.”How many hours would you spend with AI Taylor Swift?All of this feels like an intermediate step to me. To the extent that there is a market of people who want to have voice chats with a synthetic version of MrBeast, the character they want to interact with is MrBeast — not big brother Zach. I haven’t been able to chat with any of these character bots yet, but I struggle to understand how they will have more than passing novelty value.At the same time, this technology is new enough that I imagine celebrities aren’t yet willing to entrust their entire personas to Meta for safekeeping. Better to give people a taste of what it’s like to talk to AI Snoop Dogg and iron out any kinks before delivering the man himself. And when that happens, the potential seems very real. How many hours would fans spend talking to a digital version of Taylor Swift this year, if they could? How much would they pay for the privilege?While we wait to learn the answers, a new chapter of social networking may be beginning. Until now we have talked about AI in consumer apps it has mostly had to do with ranking: using machine-learning tools to create more engaging and personalized feeds for billions of users.This week we got at least two new ways to think about AI in social feeds. One is AI-generated imagery, in the form of the new stickers coming to the company’s messaging apps. It’s unclear to me how much time people want to spend creating custom images while they text their friends, but the demonstrations seemed nice enough.More significantly, I think, is the idea that Meta plans to place its AI characters on every major surface of its products. They have Facebook pages and Instagram accounts; you will message them in the same inbox that you message your friends and family. Soon, I imagine they will be making Reels.And when that happens, feeds that were once defined by the connections they enabled between human beings will have become something else: a partially synthetic social network.Will it feel more personalized, engaging, and entertaining? Or will it feel uncanny, hollow, and junky? Surely there will be a range of views on this. But either way, I think, something new is coming into focus.CommentsMost PopularTesla lowers Model Y, S, and X range estimations following exaggeration complaintsClicks is a BlackBerry-style iPhone keyboard case designed for creatorsMicrosoft’s keyboards and mice will live on under a unique new partnershipHow crowded are the oceans? New maps show what flew under the radar until nowMSI’s Steam Deck competitor has leaked, and it’s powered by an Intel Meteor Lake chipVerge Deals / Sign up for Verge Deals to get deals on products we've tested sent to your inbox daily.Email (required)Sign upBy submitting your email, you agree to our Terms and Privacy Notice. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.From our sponsorAdvertiser Content FromMore from Artificial IntelligenceUniversal Music sues AI company Anthropic for distributing song lyricsOpenAI is opening up DALL-E 3 accessYouTube might make an official way to create AI Drake fakesThe world’s biggest AI models aren’t very transparent, Stanford study saysAdvertiser Content FromThe VergeThe Verge logo.Terms of UsePrivacy NoticeCookie PolicyDo Not Sell Or Share My Personal InfoLicensing FAQAccessibilityPlatform StatusHow We Rate and Review ProductsContactTip UsCommunity GuidelinesAboutEthics StatementThe Verge is a vox media networkAdvertise with usJobs @ Vox Media© 2024 Vox Media, LLC. All Rights Reserved\", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_url = \"https://www.theverge.com/2023/9/29/23895675/ai-bot-social-network-openai-meta-chatbots\"\n",
    "documents = BeautifulSoupWebReader().load_data([data_url])\n",
    "documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": "1"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The synthetic social network is coming - The VergeSkip to main contentThe VergeThe Verge logo.The Verge homepageThe Verge homepageThe VergeThe Verge logo./Tech/Reviews/Science/Entertainment/MoreMenuEx\n",
      "pandThe VergeThe Verge logo.MenuExpandPlatformer/Artificial Intelligence/TechThe synthetic social network is comingThe synthetic social network is coming / Between ChatGPT’s surprisingly human voice a\n",
      "nd Meta’s AI characters, our feeds may be about to change foreverBy  Casey Newton, a contributing editor who has been writing about tech for over 10 years. He founded Platformer, a newsletter about Bi\n",
      "g Tech and democracy. Sep 29, 2023, 1:30 PM UTC|CommentsShare this story Image: Álvaro Bernis / The VergeThis is Platformer, a newsletter on the intersection of Silicon Valley and democracy from Casey\n",
      " Newton and Zoë Schiffer. Sign up here.Today, let’s consider the implications of a truly profound week in the development of artificial intelligence and discuss whether we may be witnessing the rise o\n",
      "f a new era in the consumer internet.I.On Monday, OpenAI announced the latest updates for ChatGPT. One feature lets you interact with its large language model via voice. Another lets you upload images\n",
      " and ask questions about them. The result is that a tool which was already useful for lots of things suddenly became useful for much more. For one thing, ChatGPT feels much more powerful as a mobile a\n",
      "pp: you can now chat with it while walking around town, or snap a picture of a tree and ask the app what you’re looking at.For another, though, adding a voice to ChatGPT begins to give it a hint of pe\n",
      "rsonality. I don’t want to overstate the case here — the app typically generates dry, sterile text unadorned by any hint of style. But something changes when you begin speaking with the app in one of \n",
      "its five native voices, which are much livelier and more dynamic than what we are used to with Alexa or the Google assistant. The voices are earnest, upbeat, and — by nature of the fact that they are \n",
      "powered by an LLM — tireless.A bot that’s smarter, more patient, more empathetic, more availableIt is the earliest stage of all this; access to the voice feature is just rolling out to ChatGPT Plus su\n",
      "bscribers, and free users won’t be able to us it for some time. And yet even in this 1.0 release, you can see the clear outlines of the sort of thing popularized in the decade-old film Her: a companio\n",
      "n so warm, empathetic and helpful that in time its users fall in love with it. The Her comparisons are by now cliche when discussing AI in Silicon Valley, and yet until now its basic premise has felt \n",
      "like a distant sci-fi dream. On Thursday, I asked the speaking version of ChatGPT to give me a pep talk to hit my deadline — I was running back from the Code Conference and inching up on my deadline —\n",
      " and as the model did its best to gas me up, it seemed to me that AI had taken an emotional step forward.You can imagine the next steps here. A bot that gets to know your quirks; remembers your life h\n",
      "istory; offers you coaching or tutoring or therapy; entertains you in whichever way you prefer. A synthetic companion not unlike the real people you encounter during the day, only smarter, more patien\n",
      "t, more empathetic, more available.Those of us who are blessed to have many close friends and family members in our life may look down at tools like this, experiencing what they offer as a cloying sim\n",
      "ulacrum of the human experience. But I imagine it might feel different for those who are lonely, isolated, or on the margins. On an early episode of Hard Fork, a trans teenager sent in a voice memo to\n",
      " tell us about using ChatGPT to get daily affirmations about identity issues. The power of giving what were then text messages a warm and kindly voice, I think, should not be underestimated.II. OpenAI\n",
      " tends to present its products as productivity tools: simple utilities for getting things done. Meta, on the other hand, is in the entertainment business. But it, too, is building LLMs, and on Wednesd\n",
      "ay the company revealed that it has found its own uses for generative AI and voices.In addition to an all-purpose AI assistant, the company unveiled 28 personality-driven chatbots to be used in Meta’s\n",
      " messaging apps. Celebrities including Charli D’Amelio, Dwyane Wade, Kendall Jenner, MrBeast, Snoop Dogg, Tom Brady, and Paris Hilton lent their voices to their effort. Each of their characters comes \n",
      "with a brief and often cringeworthy description; MrBeast’s Zach is billed as “the big brother who will roast you — because he cares.”How many hours would you spend with AI Taylor Swift?All of this fee\n",
      "ls like an intermediate step to me. To the extent that there is a market of people who want to have voice chats with a synthetic version of MrBeast, the character they want to interact with is MrBeast\n",
      " — not big brother Zach. I haven’t been able to chat with any of these character bots yet, but I struggle to understand how they will have more than passing novelty value.At the same time, this techno\n",
      "logy is new enough that I imagine celebrities aren’t yet willing to entrust their entire personas to Meta for safekeeping. Better to give people a taste of what it’s like to talk to AI Snoop Dogg and \n",
      "iron out any kinks before delivering the man himself. And when that happens, the potential seems very real. How many hours would fans spend talking to a digital version of Taylor Swift this year, if t\n",
      "hey could? How much would they pay for the privilege?While we wait to learn the answers, a new chapter of social networking may be beginning. Until now we have talked about AI in consumer apps it has \n",
      "mostly had to do with ranking: using machine-learning tools to create more engaging and personalized feeds for billions of users.This week we got at least two new ways to think about AI in social feed\n",
      "s. One is AI-generated imagery, in the form of the new stickers coming to the company’s messaging apps. It’s unclear to me how much time people want to spend creating custom images while they text the\n",
      "ir friends, but the demonstrations seemed nice enough.More significantly, I think, is the idea that Meta plans to place its AI characters on every major surface of its products. They have Facebook pag\n",
      "es and Instagram accounts; you will message them in the same inbox that you message your friends and family. Soon, I imagine they will be making Reels.And when that happens, feeds that were once defin\n",
      "ed by the connections they enabled between human beings will have become something else: a partially synthetic social network.Will it feel more personalized, engaging, and entertaining? Or will it fee\n",
      "l uncanny, hollow, and junky? Surely there will be a range of views on this. But either way, I think, something new is coming into focus.CommentsMost PopularTesla lowers Model Y, S, and X range estima\n",
      "tions following exaggeration complaintsClicks is a BlackBerry-style iPhone keyboard case designed for creatorsMicrosoft’s keyboards and mice will live on under a unique new partnershipHow crowded are \n",
      "the oceans? New maps show what flew under the radar until nowMSI’s Steam Deck competitor has leaked, and it’s powered by an Intel Meteor Lake chipVerge Deals / Sign up for Verge Deals to get deals on \n",
      "products we've tested sent to your inbox daily.Email (required)Sign upBy submitting your email, you agree to our Terms and Privacy Notice. This site is protected by reCAPTCHA and the Google Privacy Po\n",
      "licy and Terms of Service apply.From our sponsorAdvertiser Content FromMore from Artificial IntelligenceUniversal Music sues AI company Anthropic for distributing song lyricsOpenAI is opening up DALL-\n",
      "E 3 accessYouTube might make an official way to create AI Drake fakesThe world’s biggest AI models aren’t very transparent, Stanford study saysAdvertiser Content FromThe VergeThe Verge logo.Terms of U\n",
      "sePrivacy NoticeCookie PolicyDo Not Sell Or Share My Personal InfoLicensing FAQAccessibilityPlatform StatusHow We Rate and Review ProductsContactTip UsCommunity GuidelinesAboutEthics StatementThe Verg\n",
      "e is a vox media networkAdvertise with usJobs @ Vox Media© 2024 Vox Media, LLC. All Rights Reserved\n"
     ]
    }
   ],
   "source": [
    "text = documents[0].text\n",
    "\n",
    "i, num_line = 0, 200\n",
    "while True:\n",
    "    if (i+1)*num_line > len(text): \n",
    "        print(text[i*num_line:])\n",
    "        break\n",
    "    else:\n",
    "        print(text[i*num_line:(i+1)*num_line])\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### step2. load the quantized local open-source hf model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "notebookRunGroups": {
     "groupValue": ""
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BitsAndBytesConfig {\n",
       "  \"bnb_4bit_compute_dtype\": \"float16\",\n",
       "  \"bnb_4bit_quant_type\": \"nf4\",\n",
       "  \"bnb_4bit_use_double_quant\": true,\n",
       "  \"llm_int8_enable_fp32_cpu_offload\": false,\n",
       "  \"llm_int8_has_fp16_weight\": false,\n",
       "  \"llm_int8_skip_modules\": null,\n",
       "  \"llm_int8_threshold\": 6.0,\n",
       "  \"load_in_4bit\": true,\n",
       "  \"load_in_8bit\": false,\n",
       "  \"quant_method\": \"bitsandbytes\"\n",
       "}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quantization_config = BitsAndBytesConfig(\n",
    "    load_in_4bit=True,\n",
    "    bnb_4bit_compute_dtype=torch.float16,\n",
    "    bnb_4bit_quant_type=\"nf4\", # qlora\n",
    "    bnb_4bit_use_double_quant=True,\n",
    ")\n",
    "quantization_config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f693fbe80f246bdbf3b3af831043cf8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "MistralForCausalLM(\n",
       "  (model): MistralModel(\n",
       "    (embed_tokens): Embedding(32000, 4096)\n",
       "    (layers): ModuleList(\n",
       "      (0-31): 32 x MistralDecoderLayer(\n",
       "        (self_attn): MistralAttention(\n",
       "          (q_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
       "          (k_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
       "          (v_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
       "          (o_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
       "          (rotary_emb): MistralRotaryEmbedding()\n",
       "        )\n",
       "        (mlp): MistralMLP(\n",
       "          (gate_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
       "          (up_proj): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
       "          (down_proj): Linear4bit(in_features=14336, out_features=4096, bias=False)\n",
       "          (act_fn): SiLU()\n",
       "        )\n",
       "        (input_layernorm): MistralRMSNorm()\n",
       "        (post_attention_layernorm): MistralRMSNorm()\n",
       "      )\n",
       "    )\n",
       "    (norm): MistralRMSNorm()\n",
       "  )\n",
       "  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "local_model_path = os.path.join(local_mistral_model_root, \"Mistral-7B-Instruct-v0.2\")\n",
    "# local_model_path = os.path.join(local_llama_model_root, \"Llama2-7b-chat\")\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    local_model_path, \n",
    "    trust_remote_code=True\n",
    ")\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    local_model_path,\n",
    "    quantization_config=quantization_config,\n",
    "    trust_remote_code=True,\n",
    "    device_map='auto'\n",
    ")\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "llama_query_prompt_wrapper = PromptTemplate(\"<s>[INST] {query_str} [/INST]</s>\\n\")\n",
    "\n",
    "llm = HuggingFaceLLM(\n",
    "    model=model, tokenizer=tokenizer,\n",
    "    query_wrapper_prompt=llama_query_prompt_wrapper,\n",
    "    context_window=1000,\n",
    "    max_new_tokens=256,\n",
    "    generate_kwargs=dict(do_sample=True, temperature=0.2, top_k=50, top_p=0.95, pad_token_id=tokenizer.eos_token_id),\n",
    ")\n",
    "# llm # FIXME: do not print, otherwise it wll cause a recursion error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### step3. build a service based on the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ServiceContext(llm_predictor=LLMPredictor(system_prompt=None, query_wrapper_prompt=None, pydantic_program_mode=<PydanticProgramMode.DEFAULT: 'default'>), prompt_helper=PromptHelper(context_window=1000, num_output=256, chunk_overlap_ratio=0.1, chunk_size_limit=None, separator=' '), embed_model=HuggingFaceEmbedding(model_name='BAAI/bge-small-en-v1.5', embed_batch_size=10, callback_manager=<llama_index.callbacks.base.CallbackManager object at 0x7fd51c400160>, tokenizer_name='BAAI/bge-small-en-v1.5', max_length=512, pooling=<Pooling.CLS: 'cls'>, normalize=True, query_instruction=None, text_instruction=None, cache_folder=None), transformations=[SentenceSplitter(include_metadata=True, include_prev_next_rel=True, callback_manager=<llama_index.callbacks.base.CallbackManager object at 0x7fd51c400160>, id_func=<function default_id_func at 0x7fdbd9a75480>, chunk_size=1024, chunk_overlap=200, separator=' ', paragraph_separator='\\n\\n\\n', secondary_chunking_regex='[^,.;。？！]+[,.;。？！]?')], llama_logger=<llama_index.logger.base.LlamaLogger object at 0x7fd51c2f0790>, callback_manager=<llama_index.callbacks.base.CallbackManager object at 0x7fd51c400160>)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from llama_index import ServiceContext\n",
    "\n",
    "service_context = ServiceContext.from_defaults(\n",
    "    llm=llm,\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\"\n",
    ")\n",
    "service_context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### step4. build a vectorstore index based on the data and the model service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import VectorStoreIndex\n",
    "\n",
    "vectorstore_index = VectorStoreIndex.from_documents(documents, service_context=service_context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### step5. instantiate a query engine from the vectorestore index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = vectorstore_index.as_query_engine(response_mode=\"compact\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### step6. query the engine to construct a RAG service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "User Query:\n",
      "How do OpenAI and Meta differ on AI tools?\n",
      "--------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** Both OpenAI and Meta are making significant strides in the field of AI, but their approaches and applications differ. OpenAI, as previously mentioned, focuses on presenting its AI technologies as productivity tools, such as its popular ChatGPT model, which is designed to help users get things done through conversational interactions. However, the new context highlights the emotional and personal connection users can form with AI models like ChatGPT, offering not only productivity but also emotional support, coaching, therapy, and even daily affirmations. This could be particularly valuable for individuals who are lonely, isolated, or on the margins. The ability to give text messages a warm and kindly voice, as demonstrated by a trans teenager's use of ChatGPT for identity issues, should not be underestimated.\n",
       "\n",
       "Meta, on the other hand, is taking a different approach by building Language Models (LLMs) for use in its messaging apps and social media platforms. Meta's AI characters, such as its 28 personality-driven chatbots, some of which are voiced by celebrities, are intended to provide a more engaging and interactive experience for users. These chatbots are not just tools, but rather a new form of entertainment and"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "User Query:\n",
      "what happened on that Monday\n",
      "--------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** On that Monday, Meta and OpenAI made significant strides in the integration of artificial intelligence in social media and messaging platforms. Meta unveiled 28 personality-driven chatbots, featuring the voices of celebrities like Charli D'Amelio, Dwyane Wade, Kendall Jenner, MrBeast, Snoop Dogg, Tom Brady, and Paris Hilton. These chatbots, designed for use in Meta's messaging apps, offer a more immersive and interactive experience for users. The announcement came a day after OpenAI revealed updates for ChatGPT, which included its surprisingly human-like voice and productivity-focused features, such as daily affirmations and pep talks.\n",
       "\n",
       "The use of celebrity voices marks a departure from OpenAI's productivity-focused approach and Meta's entry into the entertainment industry's use of generative AI. The power of giving these digital characters a warm and kindly voice, as demonstrated by a trans teenager's use of ChatGPT for daily affirmations about identity issues, should not be underestimated. This development is a significant step forward in the creation of synthetic social networks, with potential for fans to spend"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n"
     ]
    }
   ],
   "source": [
    "from llama_index.response.notebook_utils import display_response\n",
    "\n",
    "default_query = \"How do OpenAI and Meta differ on AI tools?\"\n",
    "while True:\n",
    "    print(\"=\"*50)\n",
    "    query = input(f\"Please input your query to the engine ('quit' to exit, empty to use the default one: {default_query}):\\n\")\n",
    "    if query == \"quit\": break\n",
    "    if query == \"\": query = default_query\n",
    "    print(f\"User Query:\\n{query}\")\n",
    "    print(\"-\"*50)\n",
    "\n",
    "    response = query_engine.query(query)\n",
    "    display_response(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}