diff --git a/ai_ref_kits/README.md b/ai_ref_kits/README.md index bf45ea2a..7970d84e 100644 --- a/ai_ref_kits/README.md +++ b/ai_ref_kits/README.md @@ -15,6 +15,7 @@ - [🔦 Explainable AI](#-explainable-ai) - [🖼️ Multimodal AI Visual Generator](#%EF%B8%8F-multimodal-ai-visual-generator) - [💬 Conversational AI Chatbot](#-conversational-ai-chatbot) + - [🛒 AI Insight Agent with RAG](#-AI-Insight-Agent-with-RAG) - [Troubleshooting and Resources](#troubleshooting-and-resources) @@ -115,7 +116,19 @@ An in-depth demo of how the Multimodal AI Visual Generator Kit creates a real-ti | Example industries | Tourism | | Demo | | -The Conversational AI Chatbot is an open-source, voice-driven chat agent that answers spoken questions with meaningful, spoken responses. It can be configured to respond in any type of scenario or context. This kit demonstrates the AI Chatbot’s capabilities by simulating the experience of talking to a hotel concierge. +The Conversational AI Chatbot is an open-source, voice-driven chat agent that answers spoken questions with meaningful, spoken responses. It can be configured to respond in any type of scenario or context. +This kit demonstrates the AI Chatbot’s capabilities by simulating the experience of talking to a hotel concierge. + +### 🛒 AI Insight Agent with RAG +[![agentic_llm_rag](https://github.com/user-attachments/assets/0471ab91-ded5-4a5f-8d8e-5432f1b4b45c)](agentic-llm-rag) + +| [AI Insight Agent with RAG](agentic_llm_rag) | | +|--------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| +| Related AI concepts | Natural Language Understanding, Large Language Models (LLMs), Retrieval Augmented Generation (RAG), Agentic AI, Generative AI | +| Example industries | Retail | +| Demo | | + +The AI Insight Agent with RAG uses Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to interpret user prompts, engage in meaningful dialogue, perform calculations, use RAG techniques to improve its knowledge and interact with the user to add items to a virtual shopping cart. ## Troubleshooting and Resources - Open a [discussion topic](https://github.com/openvinotoolkit/openvino_build_deploy/discussions) diff --git a/ai_ref_kits/agentic_llm_rag/README.md b/ai_ref_kits/agentic_llm_rag/README.md new file mode 100644 index 00000000..064ea7e2 --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/README.md @@ -0,0 +1,203 @@ +
+

AI Insight Agent with RAG

+

+ 🏠 About the Kits  + +

+
+ +[![Apache License Version 2.0](https://img.shields.io/badge/license-Apache_2.0-green.svg)](https://github.com/openvinotoolkit/openvino_build_deploy/blob/master/LICENSE.txt) + +

+ +

+ +The AI Insight Agent with RAG uses Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to interpret user prompts, engage in meaningful dialogue, perform calculations, use RAG techniques to improve its knowledge and interact with the user to add items to a virtual shopping cart. This solution uses the OpenVINO™ toolkit to power the AI models at the edge. Designed for both consumers and employees, it functions as a smart, personalized retail assistant, offering an interactive and user-friendly experience similar to an advanced digital kiosk. + +This kit uses the following technology stack: +- [OpenVINO Toolkit](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) ([docs](https://docs.openvino.ai/)) +- [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct) +- [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) +- [Gradio interface](https://www.gradio.app/docs/gradio/chatinterface) + +Check out our [AI Reference Kits repository](/) for other kits. + +![agentic_llm_rag](https://github.com/user-attachments/assets/0471ab91-ded5-4a5f-8d8e-5432f1b4b45c) + +
Table of Contents + +- [Getting Started](#get-started) + - [Installing Prerequisites](#install-prerequisites) + - [Setting Up Your Environment](#set-up-your-environment) + - [Converting and Optimizing the Model](*convert-and-optimize-the-model) + - [Running the Application](#run-the-application) +- [Additional Resources](#additional-resources) + +
+ +# Getting Started + +To get started with the AI Insight Agent with RAG, you install Python, set up your environment, and then you can run the application. We recommend using Ubuntu 24.04 to set up and run this project. + +## Installing Prerequisites + +This project requires Python 3.8 or higher and a few libraries. If you don't already have Python installed on your machine, go to [https://www.python.org/downloads/](https://www.python.org/downloads/) and download the latest version for your operating system. Follow the prompts to install Python, and make sure to select the option to add Python to your PATH environment variable. + +To install the Python libraries and tools, run this command: + +```shell +sudo apt install git gcc python3-venv python3-dev +``` + +_NOTE: If you are using Windows, you might also have to install [Microsoft Visual C++ Redistributable](https://aka.ms/vs/16/release/vc_redist.x64.exe)._ + +## Setting Up Your Environment + +To set up your environment, you first clone the repository, then create a virtual environment, activate the environment, and install the packages. + +### Clone the Repository + +To clone the repository, run this command: + +```shell +git clone https://github.com/openvinotoolkit/openvino_build_deploy.git +``` + +This command clones the repository into a directory named "openvino_build_deploy" in the current directory. After the directory is cloned, run the following command to go to that directory: + + +```shell +cd openvino_build_deploy/ai_ref_kits/agentic_llm_rag +``` + +### Create a Virtual Environment + +To create a virtual environment, open your terminal or command prompt, and go to the directory where you want to create the environment. + +Run the following command: + +```shell +python3 -m venv venv +``` +This creates a new virtual environment named "venv" in the current directory. + +### Activate the Environment + +The command you run to activate the virtual environment you created depends on whether you have a Unix-based operating system (Linux or macOS) or a Windows operating system. + +To activate the virtual environment for a **Unix-based** operating system, run: + +```shell +source venv/bin/activate # For Unix-based operating systems such as Linux or macOS +``` + +To activate the virtual environment for a **Windows** operating system, run: + +```shell +venv\Scripts\activate # This command is for Windows operating systems +``` +This activates the virtual environment and changes your shell's prompt to indicate that you are now working in that environment. + +### Install the Packages + +To install the required packages, run the following commands: + +```shell +python -m pip install --upgrade pip +pip install -r requirements.txt +``` +## Converting and Optimizing the Model + +The application uses 2 separate models. Each model requires conversion and optimization for use with OpenVINO™. The following process includes a step to convert and optimize each model. + +_NOTE: This reference kit requires more than 8GB of bandwidth and disk space for downloading models. Because of the large model size, when you run the kit for the first time, the conversion can take more than two hours and require more than 32GB of memory. After the first run, the subsequent runs should finish much faster._ + +## Chat Model and Embedding Model Conversion + +The _chat model_ is the core of the chatbot's ability to generate meaningful and context-aware responses. + +The _embedding model_ represents text data (both user queries and potential responses or knowledge base entries) as numerical vectors. These vectors are essential for tasks such as semantic search and similarity matching. + +This conversion script handles the conversion and optimization of: + +- The chat model (`qwen2-7B`) with `int4` precision. +- The embedding model (`bge-large`) with `FP32` precision. + +After the models are converted, they’re saved to the model directory you specify when you run the script. + +_Requests can take up to one hour to process._ + +To convert the chat and embedding models, run: +```shell +python convert_and_optimize_llm.py --chat_model_type qwen2-7B --embedding_model_type bge-large --precision int4 --model_dir model +``` + +If using gated models from HuggingFace pass the `--hf_token` argument with your HuggingFace token. Remember to request access to gated models if needed. + +After you run the conversion scripts, you can run `main.py` to launch the application. + +## Running the Application (Gradio Interface) + +To run the AI Insight Agent with RAG application, you execute the following python script. Make sure to include all of the necessary model directory arguments. + +_NOTE: This application requires more than 16GB of memory because the models are very large (especially the chatbot model). If you have a less powerful device, the application might also run slowly._ + +After that, you should be able to run the application with default values: + +```shell +python main.py +``` + +For more settings, you can change the argument values: + +- `--chat_model`: The path to your chat model directory (for example, `model/qwen2-7B-INT4`) that drives conversation flow and response generation. + +- `--rag_pdf`: The path to the document (for example, `data/test_painting_llm_rag.pdf`) that contains additional knowledge for Retrieval-Augmented Generation (RAG). + +- `--embedding_model`: The path to your embedding model directory (for example, `model/bge-small-FP32`) for understanding and matching text inputs. + +- `--device`: Include this flag to select the inference device for both models. (for example, `CPU`). If you have access to a dedicated GPU (ARC, Flex), you can change the value to `GPU.1`. Possible values: `CPU,GPU,GPU.1,NPU` + +- `--public`: Include this flag to make the Gradio interface publicly accessible over the network. Without this flag, the interface will only be available on your local machine. + +To run the application, execute the `main.py` script with the following command. Make sure to include all necessary model directory arguments. +```shell +python main.py \ + --chat_model model/qwen2-7B-INT4 \ + --embedding_model model/bge-small-FP32 \ + --rag_pdf data/test_painting_llm_rag.pdf \ + --device GPU.1 \ + --public +``` + +### System Prompt Usage in LlamaIndex ReActAgent + +The LlamaIndex ReActAgent library relies on a default system prompt that provides essential instructions to the LLM for correctly interacting with available tools. This prompt is fundamental for enabling both tool usage and RAG (Retrieval-Augmented Generation) queries. + +#### Important: +Do not override or modify the default system prompt. Altering it may prevent the LLM from using the tools or executing RAG queries properly. + +#### Customizing the Prompt: +If you need to add extra rules or custom behavior, modify the Additional Rules section located in the system_prompt.py file. + +### Use the Web Interface +After the script runs, Gradio provides a local URL (typically `http://127.0.0.1:XXXX`) that you can open in your web browser to interact with the assistant. If you configured the application to be accessible publicly, Gradio also provides a public URL. + +#### Test the Application +When you test the AI Insight Agent with RAG application, you can test both the interaction with the agent and the product selection capabilities. + +1. Open a web browers and go to the Gradio-provided URL. + _For example, `http://127.0.0.1:XXXX`._ +2. Test text interaction with the application. + - Type your question in the text box and press **Enter**. + _The assistant responds to your question in text form._ + +For further testing of the AI Insight Agent with RAG appplication, you can engage with the chatbot assistant by asking it questions, or giving it commands that align with the assistant's capabilities. This hands-on experience can help you to understand the assistant's interactive quality and performance. + +Enjoy exploring the capabilities of your AI Insight Agent with RAG appplication! + +# Additional Resources +- Learn more about [OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html) +- Explore [OpenVINO’s documentation](https://docs.openvino.ai/2024/home.html) + +

Back to top ⬆️

diff --git a/ai_ref_kits/agentic_llm_rag/app.py b/ai_ref_kits/agentic_llm_rag/app.py new file mode 100644 index 00000000..06bb0ddd --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/app.py @@ -0,0 +1,562 @@ + +import argparse +import io +import logging +import sys +import time +import warnings +from io import StringIO +from pathlib import Path +from typing import Tuple, Callable + +import gradio as gr +import nest_asyncio +import openvino.properties as props +import openvino.properties.hint as hints +import openvino.properties.streams as streams +import requests +import yaml +from llama_index.core import PromptTemplate +from llama_index.core import SimpleDirectoryReader +from llama_index.core import VectorStoreIndex, Settings +from llama_index.core.agent import ReActAgent +from llama_index.core.tools import FunctionTool +from llama_index.core.tools import QueryEngineTool, ToolMetadata +from llama_index.embeddings.huggingface_openvino import OpenVINOEmbedding +from llama_index.llms.openvino import OpenVINOLLM +from llama_index.core.agent import ReActChatFormatter +from llama_index.core.llms import MessageRole +from llama_index.core.callbacks import CallbackManager +# Agent tools +from tools import PaintCalculator, ShoppingCart +from system_prompt import react_system_header_str + +# Initialize logging +logging.basicConfig(level=logging.INFO) +log = logging.getLogger(__name__) + +#Filter unnecessary warnings for demonstration +warnings.filterwarnings("ignore") + +ov_config = { + hints.performance_mode(): hints.PerformanceMode.LATENCY, + streams.num(): "1", + props.cache_dir(): "" +} + +def setup_models( + llm_model_path: str, + embedding_model_path: str, + device: str) -> Tuple[OpenVINOLLM, OpenVINOEmbedding]: + """ + Sets up LLM and embedding models using OpenVINO. + + Args: + llm_model_path: Path to the LLM model + embedding_model_path: Path to the embedding model + device: Target device for inference ("CPU", "GPU", etc.) + + Returns: + Tuple of (llm, embedding) models + """ + + # Load LLM model locally + llm = OpenVINOLLM( + model_id_or_path=str(llm_model_path), + context_window=8192, + max_new_tokens=500, + model_kwargs={"ov_config": ov_config}, + generate_kwargs={"do_sample": False, "temperature": 0.1, "top_p": 0.8}, + device_map=device, + ) + + # Load the embedding model locally + embedding = OpenVINOEmbedding(model_id_or_path=embedding_model_path, device=device) + + return llm, embedding + + +def setup_tools()-> Tuple[FunctionTool, FunctionTool, FunctionTool, FunctionTool, FunctionTool]: + + """ + Sets up and returns a collection of tools for paint calculations and shopping cart management. + + Returns: + Tuple containing tools for paint cost calculation, paint gallons calculation, + adding items to cart, viewing cart, and clearing cart + """ + + paint_cost_calculator = FunctionTool.from_defaults( + fn=PaintCalculator.calculate_paint_cost, + name="calculate_paint_cost", + description="ALWAYS use this tool when calculating paint cost for a specific area in square feet. Required inputs: area (float, square feet), price_per_gallon (float), add_paint_supply_costs (bool)" + ) + + paint_gallons_calculator = FunctionTool.from_defaults( + fn=PaintCalculator.calculate_paint_gallons_needed, + name="calculate_paint_gallons", + description="Calculate how many gallons of paint are needed to cover a specific area. Required input: area (float, square feet). Returns the number of gallons needed, rounded up to ensure full coverage." +) + + add_to_cart_tool = FunctionTool.from_defaults( + fn=ShoppingCart.add_to_cart, + name="add_to_cart", + description=""" + Use this tool WHENEVER a user wants to add any item to their cart or shopping cart. + + PARAMETERS: + - product_name (string): The exact name of the product (e.g., "Premium Latex Paint") + - quantity (int): The number of units to add, must be a positive integer (e.g., 2) + - price_per_unit (float): The price per unit in dollars (e.g., 24.99) + + RETURNS: + - A confirmation message and updated cart contents + + EXAMPLES: + To add 3 gallons of paint at $29.99 each: add_to_cart(product_name="Interior Eggshell Paint", quantity=3, price_per_unit=29.99) + """ + ) + + get_cart_items_tool = FunctionTool.from_defaults( + fn=ShoppingCart.get_cart_items, + name="view_cart", + description=""" + Use this tool when a user wants to see what's in their shopping cart. + No parameters are required. + + RETURNS: + - A list of all items currently in the cart with their details + + EXAMPLES: + To view the current cart contents: view_cart() + """ + ) + + clear_cart_tool = FunctionTool.from_defaults( + fn=ShoppingCart.clear_cart, + name="clear_cart", + description=""" + Use this tool when a user asks to empty or clear their shopping cart. + No parameters are required. + + RETURNS: + - A confirmation message that the cart has been cleared + + EXAMPLES: + To empty the shopping cart: clear_cart() + """ + ) + return paint_cost_calculator, add_to_cart_tool, get_cart_items_tool, clear_cart_tool, paint_gallons_calculator + + +def load_documents(text_example_en_path: str) -> VectorStoreIndex: + """ + Loads documents from the given path + + Args: + text_example_en_path: Path to the document to load + + Returns: + VectorStoreIndex for the loaded documents + """ + + if not text_example_en_path.exists(): + text_example_en = "test_painting_llm_rag.pdf" + r = requests.get(text_example_en) + content = io.BytesIO(r.content) + with open(text_example_en_path, "wb") as f: + f.write(content.read()) + + reader = SimpleDirectoryReader(input_files=[text_example_en_path]) + documents = reader.load_data() + index = VectorStoreIndex.from_documents(documents) + + return index + +def custom_handle_reasoning_failure(callback_manager: CallbackManager, exception: Exception): + """ + Provides custom error handling for agent reasoning failures. + + Args: + callback_manager: The callback manager instance for event handling + exception: The exception that was raised during reasoning + """ + return "Hmm...I didn't quite that. Could you please rephrase your question to be simpler?" + + +def run_app(agent: ReActAgent, public_interface: bool = False) -> None: + """ + Launches the application with the specified agent and interface settings. + + Args: + agent: The ReActAgent instance configured with tools + public_interface: Whether to launch with a public-facing Gradio interface + """ + class Capturing(list): + """A context manager that captures stdout output into a list.""" + def __enter__(self): + """ + Redirects stdout to a StringIO buffer and returns self. + Called when entering the 'with' block. + """ + self._stdout = sys.stdout + sys.stdout = self._stringio = StringIO() + return self + def __exit__(self, *args): + """ + Stores captured output in this list and restores stdout. + Called when exiting the 'with' block. + """ + self.extend(self._stringio.getvalue().splitlines()) + del self._stringio + sys.stdout = self._stdout + + def _handle_user_message(user_message, history): + return "", [*history, (user_message, "")] + + def update_cart_display()-> str: + """ + Generates an HTML representation of the shopping cart contents. + + Retrieves current cart items and creates a formatted HTML table + showing product details, quantities, prices, and totals. + If the cart is empty, returns a message indicating this. + + Returns: + str: Markdown-formatted HTML table of cart contents + or message indicating empty cart + """ + cart_items = ShoppingCart.get_cart_items() + if not cart_items: + return "### 🛒 Your Shopping Cart is Empty" + + table = "### 🛒 Your Shopping Cart\n\n" + table += "\n" + table += " \n" + table += " \n" + table += " \n" + table += " \n" + table += " \n" + table += " \n" + table += " \n" + table += " \n" + table += " \n" + + for item in cart_items: + table += " \n" + table += f" \n" + table += f" \n" + table += f" \n" + table += f" \n" + table += " \n" + + table += " \n" + table += "
ProductQtyPriceTotal
{item['product_name']}{item['quantity']}${item['price_per_unit']:.2f}${item['total_price']:.2f}
\n" + + total = sum(item["total_price"] for item in cart_items) + table += f"\n**Total: ${total:.2f}**" + return table + + def _generate_response(chat_history: list, log_history: list | None = None)->Tuple[str,str,str]: + """ + Generate a streaming response from the agent with formatted thought process logs. + + This function: + 1. Captures the agent's thought process + 2. Formats the thought process into readable logs + 3. Streams the agent's response token by token + 4. Tracks performance metrics for thought process and response generation + 5. Updates the shopping cart display + + Args: + chat_history: List of conversation messages + log_history: List to store logs, will be initialized if None + + Yields: + tuple: (chat_history, formatted_log_history, cart_content) + - chat_history: Updated with agent's response + - formatted_log_history: String of joined logs + - cart_content: HTML representation of the shopping cart + """ + log.info(f"log_history {log_history}") + + if not isinstance(log_history, list): + log_history = [] + + # Capture time for thought process + start_thought_time = time.time() + + # Capture the thought process output + with Capturing() as output: + try: + response = agent.stream_chat(chat_history[-1][0]) + except ValueError: + response = agent.stream_chat(chat_history[-1][0]) + formatted_output = [] + for line in output: + if "Thought:" in line: + formatted_output.append("\n🤔 **Thought:**\n" + line.split("Thought:", 1)[1]) + elif "Action:" in line: + formatted_output.append("\n🔧 **Action:**\n" + line.split("Action:", 1)[1]) + elif "Action Input:" in line: + formatted_output.append("\n📥 **Input:**\n" + line.split("Action Input:", 1)[1]) + elif "Observation:" in line: + formatted_output.append("\n📋 **Result:**\n" + line.split("Observation:", 1)[1]) + else: + formatted_output.append(line) + end_thought_time = time.time() + thought_process_time = end_thought_time - start_thought_time + + # After response is complete, show the captured logs in the log area + log_entries = "\n".join(formatted_output) + log_history.append("### 🤔 Agent's Thought Process") + thought_process_log = f"Thought Process Time: {thought_process_time:.2f} seconds" + log_history.append(f"{log_entries}\n{thought_process_log}") + cart_content = update_cart_display() # update shopping cart + yield chat_history, "\n".join(log_history), cart_content # Yield after the thought process time is captured + + # Now capture response generation time + start_response_time = time.time() + + # Gradually yield the response from the agent to the chat + # Quick fix for agent occasionally repeating the first word of its repsponse + last_token = "Dummy Token" + i = 0 + for token in response.response_gen: + if i == 0: + last_token = token + if i == 1 and token.split()[0] == last_token.split()[0]: + chat_history[-1][1] += token.split()[1] + " " + else: + chat_history[-1][1] += token + yield chat_history, "\n".join(log_history), cart_content # Ensure log_history is a string + if i <= 2: i += 1 + + end_response_time = time.time() + response_time = end_response_time - start_response_time + + # Log tokens per second along with the device information + tokens = len(chat_history[-1][1].split(" ")) * 4 / 3 # Convert words to approx token count + response_log = f"Response Time: {response_time:.2f} seconds ({tokens / response_time:.2f} tokens/s)" + + log.info(response_log) + + # Append the response time to log history + log_history.append(response_log) + yield chat_history, "\n".join(log_history), cart_content # Join logs into a string for display + + def _reset_chat()-> tuple[str, list, str, str]: + """ + Resets the chat interface and agent state to initial conditions. + + This function: + 1. Resets the agent's internal state + 2. Clears all items from the shopping cart + 3. Returns values needed to reset the UI components + + Returns: + tuple: Values to reset UI components + - Empty string: Clears the message input + - Empty list: Resets chat history + - Default log heading: Sets initial log area text + - Empty cart display: Shows empty shopping cart + """ + agent.reset() + ShoppingCart._cart_items = [] + return "", [], "🤔 Agent's Thought Process", update_cart_display() + + def run()-> None: + """ + Sets up and launches the Gradio web interface for the Smart Retail Assistant. + + This function: + 1. Loads custom CSS styling if available + 2. Configures the Gradio theme and UI components + 3. Sets up the chat interface with agent interaction + 4. Configures event handlers for user inputs + 5. Adds example prompts for users + 6. Launches the web interface + + The interface includes: + - Chat window for user-agent conversation + - Log window to display agent's thought process + - Shopping cart display + - Text input for user messages + - Submit and Clear buttons + - Sample questions for easy access + """ + custom_css = "" + try: + with open("css/gradio.css", "r") as css_file: + custom_css = css_file.read() + except Exception as e: + log.warning(f"Could not load CSS file: {e}") + + theme = gr.themes.Default( + primary_hue="blue", + font=[gr.themes.GoogleFont("Montserrat"), "ui-sans-serif", "sans-serif"], + ) + + with gr.Blocks(theme=theme, css=custom_css) as demo: + + header = gr.HTML( + "
" + "
" + " " + "
Smart Retail Assistant 🤖: Agentic LLMs with RAG 💭
" + "
" + "
" + ) + + with gr.Row(): + chat_window = gr.Chatbot( + label="Paint Purchase Helper", + avatar_images=(None, "https://docs.openvino.ai/2024/_static/favicon.ico"), + height=400, # Adjust height as per your preference + scale=2 # Set a higher scale value for Chatbot to make it wider + #autoscroll=True, # Enable auto-scrolling for better UX + ) + log_window = gr.Markdown( + show_label=True, + value="### 🤔 Agent's Thought Process", + height=400, + elem_id="agent-steps" + ) + cart_display = gr.Markdown( + value=update_cart_display(), + elem_id="shopping-cart", + height=400 + ) + + with gr.Row(): + message = gr.Textbox(label="Ask the Paint Expert 🎨", scale=4, placeholder="Type your prompt/Question and press Enter") + + with gr.Column(scale=1): + submit_btn = gr.Button("Submit", variant="primary") + clear = gr.ClearButton() + + sample_questions = [ + "what paint is the best for kitchens?", + "what is the price of it?", + "how many gallons of paint do I need to cover 600 sq ft ?", + "add them to my cart", + "what else do I need to complete my project?", + "add 2 brushes to my cart", + "create a table with paint products sorted by price", + "Show me what's in my cart", + "clear shopping cart", + "I have a room 1000 sqft, I'm looking for supplies to paint the room" + ] + gr.Examples( + examples=sample_questions, + inputs=message, + label="Examples" + ) + + # Ensure that individual components are passed + message.submit( + _handle_user_message, + inputs=[message, chat_window], + outputs=[message, chat_window], + queue=False + ).then( + _generate_response, + inputs=[chat_window, log_window], + outputs=[chat_window, log_window, cart_display], + ) + + submit_btn.click( + _handle_user_message, + inputs=[message, chat_window], + outputs=[message, chat_window], + queue=False, + ).then( + _generate_response, + inputs=[chat_window, log_window], + outputs=[chat_window, log_window, cart_display], + ) + clear.click(_reset_chat, None, [message, chat_window, log_window, cart_display]) + + gr.Markdown("------------------------------") + + log.info("Demo is ready!") + demo.queue().launch(share=public_interface) + + run() + + +def run(chat_model: str, embedding_model: str, rag_pdf: str, device: str, public_interface: bool = False): + """ + Initializes and runs the agentic rag solution + + Args: + chat_model: Path to the LLM chat model + embedding_model: Path to the embedding model + rag_pdf: Path to the PDF file for RAG functionality + device: Target device for model inference ("CPU", "GPU", "GPU.1") + public_interface: Whether to expose a public-facing interface + """ + # Load models and embedding based on parsed arguments + llm, embedding = setup_models(chat_model, embedding_model, device) + + Settings.embed_model = embedding + Settings.llm = llm + + # Set up tools + paint_cost_calculator, add_to_cart_tool, get_cart_items_tool, clear_cart_tool, paint_gallons_calculator = setup_tools() + + text_example_en_path = Path(rag_pdf) + index = load_documents(text_example_en_path) + log.info(f"loading in {index}") + + vector_tool = QueryEngineTool( + index.as_query_engine(streaming=True), + metadata=ToolMetadata( + name="vector_search", + description=""" + Use this tool for ANY question about paint products, recommendations, prices, or technical specifications. + + WHEN TO USE: + - User asks about paint types, brands, or products + - User needs price information before adding to cart + - User needs recommendations based on their project + - User has technical questions about painting + + EXAMPLES: + - "What paint is best for kitchen cabinets?" + - "How much does AwesomePainter Interior Acrylic Latex cost?" + - "What supplies do I need for painting my living room?" + """, + ), + ) + + nest_asyncio.apply() + + # Define agent and available tools + agent = ReActAgent.from_tools( + [paint_cost_calculator, add_to_cart_tool, get_cart_items_tool, clear_cart_tool, vector_tool, paint_gallons_calculator], + llm=llm, + max_iterations=5, # Set a max_iterations value + handle_reasoning_failure_fn=custom_handle_reasoning_failure, + verbose=True, + react_chat_formatter=ReActChatFormatter.from_defaults( + observation_role=MessageRole.TOOL + ), + ) + react_system_prompt = PromptTemplate(react_system_header_str) + agent.update_prompts({"agent_worker:system_prompt": react_system_prompt}) + agent.reset() + run_app(agent, public_interface) + +if __name__ == "__main__": + # Define the argument parser at the end + parser = argparse.ArgumentParser() + parser.add_argument("--chat_model", type=str, default="model/qwen2-7B-INT4", help="Path to the chat model directory") + parser.add_argument("--embedding_model", type=str, default="model/bge-large-FP32", help="Path to the embedding model directory") + parser.add_argument("--rag_pdf", type=str, default="data/test_painting_llm_rag.pdf", help="Path to a RAG PDF file with additional knowledge the chatbot can rely on.") + parser.add_argument("--device", type=str, default="GPU", help="Device for inferencing (CPU,GPU,GPU.1,NPU)") + parser.add_argument("--public", default=False, action="store_true", help="Whether interface should be available publicly") + + args = parser.parse_args() + + run(args.chat_model, args.embedding_model, args.rag_pdf, args.device, args.public) diff --git a/ai_ref_kits/agentic_llm_rag/convert_and_optimize_llm.py b/ai_ref_kits/agentic_llm_rag/convert_and_optimize_llm.py new file mode 100644 index 00000000..2e46e431 --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/convert_and_optimize_llm.py @@ -0,0 +1,141 @@ +import argparse +from pathlib import Path +import os +import numpy as np +import openvino as ov +from openvino.runtime import opset10 as ops +from openvino.runtime import passes +from optimum.intel import OVModelForCausalLM, OVModelForFeatureExtraction, OVWeightQuantizationConfig, OVConfig, OVQuantizer +from transformers import AutoTokenizer + +MODEL_MAPPING = { + "llama3.2-3B": "meta-llama/Llama-3.2-3B-Instruct", + "qwen2-7B": "Qwen/Qwen2-7B-Instruct", + "bge-large": "BAAI/bge-large-en-v1.5", + "bge-small": "BAAI/bge-small-en-v1.5", + "bge-m3": "BAAI/bge-m3", +} + +def optimize_model_for_npu(model: OVModelForFeatureExtraction): + """ + Fix some tensors to support NPU inference + + Params: + model: model to fix + """ + class ReplaceTensor(passes.MatcherPass): + def __init__(self, packed_layer_name_tensor_dict_list): + super().__init__() + self.model_changed = False + + param = passes.WrapType("opset10.Multiply") + + def callback(matcher: passes.Matcher) -> bool: + root = matcher.get_match_root() + if root is None: + return False + for y in packed_layer_name_tensor_dict_list: + root_name = root.get_friendly_name() + if root_name.find(y["name"]) != -1: + max_fp16 = np.array([[[[-np.finfo(np.float16).max]]]]).astype(np.float32) + new_tensor = ops.constant(max_fp16, ov.Type.f32, name="Constant_4431") + root.set_arguments([root.input_value(0).node, new_tensor]) + packed_layer_name_tensor_dict_list.remove(y) + + return True + + self.register_matcher(passes.Matcher(param, "ReplaceTensor"), callback) + + packed_layer_tensor_dict_list = [{"name": "aten::mul/Multiply"}] + + manager = passes.Manager() + manager.register_pass(ReplaceTensor(packed_layer_tensor_dict_list)) + manager.run_passes(model.model) + model.reshape(1, 512) + + +def convert_chat_model(model_type: str, precision: str, model_dir: Path, access_token: str) -> Path: + """ + Convert chat model + + Params: + model_type: selected mode type and size + precision: model precision + model_dir: dir to export model + Returns: + Path to exported model + """ + output_dir = model_dir / model_type + model_name = MODEL_MAPPING[model_type] + + if access_token is not None: + os.environ["HUGGING_FACE_HUB_TOKEN"] = access_token + + # load model and convert it to OpenVINO + model = OVModelForCausalLM.from_pretrained(model_name, export=True, compile=False, load_in_8bit=False, token=access_token) + # change precision to FP16 + model.half() + + if precision != "fp16": + # select quantization mode + quant_config = OVWeightQuantizationConfig(bits=4, sym=False, ratio=0.8) if precision == "int4" else OVWeightQuantizationConfig(bits=8, sym=False) + config = OVConfig(quantization_config=quant_config) + + suffix = "-INT4" if precision == "int4" else "-INT8" + output_dir = output_dir.with_name(output_dir.name + suffix) + + # create a quantizer + quantizer = OVQuantizer.from_pretrained(model, task="text-generation") + # quantize weights and save the model to the output dir + quantizer.quantize(save_directory=output_dir, weights_only=True, ov_config=config) + else: + output_dir = output_dir.with_name(output_dir.name + "-FP16") + # save converted model + model.save_pretrained(output_dir) + + # export also tokenizer + tokenizer = AutoTokenizer.from_pretrained(model_name) + tokenizer.save_pretrained(output_dir) + + return Path(output_dir) / "openvino_model.xml" + + +def convert_embedding_model(model_type: str, model_dir: Path) -> Path: + """ + Convert embedding model + + Params: + model_type: selected mode type and size + model_dir: dir to export model + Returns: + Path to exported model + """ + output_dir = model_dir / model_type + output_dir = output_dir.with_name(output_dir.name + "-FP32") + model_name = MODEL_MAPPING[model_type] + + # load model and convert it to OpenVINO + model = OVModelForFeatureExtraction.from_pretrained(model_name, export=True, compile=False) + optimize_model_for_npu(model) + model.save_pretrained(output_dir) + + # export tokenizer + tokenizer = AutoTokenizer.from_pretrained(model_name) + tokenizer.save_pretrained(output_dir) + + return Path(output_dir) / "openvino_model.xml" + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--chat_model_type", type=str, choices=["qwen2-7B", "llama3.2-3B"], + default="qwen2-7B", help="Chat model to be converted") + parser.add_argument("--embedding_model_type", type=str, choices=["bge-small", "bge-large", "bge-m3"], + default="bge-large", help="Embedding model to be converted") + parser.add_argument("--precision", type=str, default="int4", choices=["fp16", "int8", "int4"], help="Model precision") + parser.add_argument("--hf_token", type=str, help="HuggingFace access token") + parser.add_argument("--model_dir", type=str, default="model", help="Directory to place the model in") + + args = parser.parse_args() + convert_embedding_model(args.embedding_model_type, Path(args.model_dir)) + convert_chat_model(args.chat_model_type, args.precision, Path(args.model_dir), args.hf_token) diff --git a/ai_ref_kits/agentic_llm_rag/css/gradio.css b/ai_ref_kits/agentic_llm_rag/css/gradio.css new file mode 100644 index 00000000..de0cca44 --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/css/gradio.css @@ -0,0 +1,174 @@ +body { + padding: 15px; + box-sizing: border-box; + overflow-x: hidden; +} + +#agent-steps { + border: 2px solid #ddd; + border-radius: 8px; + padding: 12px; + background-color: #f9f9f9; + margin-top: 0; /* Remove top margin to align with other components */ + height: 100%; /* Ensure the same height as other components */ + box-sizing: border-box; /* Include padding in height calculation */ +} + +#shopping-cart { + border: 2px solid #4CAF50; + border-radius: 8px; + padding: 12px; + background-color: #f0f8f0; + margin-top: 0; /* Remove top margin to align with other components */ + height: 100%; /* Ensure the same height as other components */ + box-sizing: border-box; /* Include padding in height calculation */ +} + +/* Fix row alignment issues */ +.gradio-row { + align-items: flex-start !important; /* Align all items to the top of the row */ +} + +/* Make all components in the main row the same height */ +.gradio-row > .gradio-column { + height: 100%; + display: flex; + flex-direction: column; +} + +/* Ensure the chatbot and other components align properly */ +.gradio-chatbot { + margin-top: 0 !important; +} + +/* Improve shopping cart table styling */ +#shopping-cart table { + width: 100%; + border-collapse: collapse; + table-layout: auto; /* Let the browser calculate column widths based on content */ +} + +#shopping-cart th, +#shopping-cart td { + padding: 8px; + text-align: left; + min-width: 50px; /* Ensure minimum width for all columns */ +} + +#shopping-cart th:nth-child(2), /* Qty column */ +#shopping-cart td:nth-child(2) { + text-align: center; + width: 50px; +} + +#shopping-cart th:nth-child(3), /* Price column */ +#shopping-cart td:nth-child(3), +#shopping-cart th:nth-child(4), /* Total column */ +#shopping-cart td:nth-child(4) { + text-align: right; + min-width: 80px; +} + +#shopping-cart th:first-child, /* Product column */ +#shopping-cart td:first-child { + width: auto; /* Let product name take remaining space */ +} + +.sample-prompt-btn { + min-height: 35px !important; + font-size: 0.85em !important; + margin: 2px !important; + padding: 4px 8px !important; +} + +.intel-header { + margin: 0px; + padding: 0 15px; + background: #0054ae; + height: 60px; + width: 100%; + display: flex; + align-items: center; + position: relative; + box-sizing: border-box; + margin-bottom: 15px; +} + +.intel-logo { + margin-left: 20px; + margin-right: 20px; + width: 60px; + height: 60px; +} + +.intel-title { + height: 60px; + line-height: 60px; + color: white; + font-size: 24px; +} + +.gradio-container { + max-width: 100% !important; + padding: 0 !important; + box-sizing: border-box; + overflow-x: hidden; +} + +/* Override Gradio's generated padding classes */ +.padding.svelte-phx28p, +[class*="padding svelte-"], +.gradio-container [class*="padding"] { + padding: 0 !important; +} + +.intel-header-wrapper { + width: 100%; + max-width: 100%; + margin-left: 0; + position: relative; + padding: 0; + box-sizing: border-box; +} + +.gradio-container > .main { + padding: 20px !important; + max-width: 1800px; + margin: 0 auto; + box-sizing: border-box; +} + +/* Fix label alignment issues */ +.gradio-column > .label-wrap { + margin-top: 0; +} + +/* Ensure consistent spacing for all components */ +.gradio-box, .gradio-chatbot, .gradio-markdown { + margin-top: 0 !important; +} + +/* Responsive adjustments */ +@media (max-width: 768px) { + #agent-steps, #shopping-cart { + padding: 8px; + } + + .intel-logo { + margin-left: 10px; + margin-right: 10px; + width: 50px; + height: 50px; + } + + .intel-title { + font-size: 20px; + } + + /* Adjust table for mobile */ + #shopping-cart th, + #shopping-cart td { + padding: 4px; + font-size: 0.9em; + } +} \ No newline at end of file diff --git a/ai_ref_kits/agentic_llm_rag/data/Sample_Prompts.txt b/ai_ref_kits/agentic_llm_rag/data/Sample_Prompts.txt new file mode 100644 index 00000000..60fbf67d --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/data/Sample_Prompts.txt @@ -0,0 +1,25 @@ +*Sample prompts* + +These are sample prompts that would work depends on the context. + +RAG: +- what paint is the best for kitchens? +- what is the price of it? +- what else do I need to complete my project? +- I want to paint my room. The size is 500 sqft. which products do you recommend? +- create a table with paint products sorted by price +- I have a room 1000 sqft, I'm looking for supplies to paint the room + +Paint Calculations tools: +- how many gallons of paint do I need to cover 600 sq ft ? +- Calculate the paint cost for a 600 sqft room using Sherwin-Williams Emerald + +Shopping Cart tools: +- add them to my cart +- add brushes to my cart +- add rollers to my shopping cart +- add 3 gallons of Benjamin Moore Aura Revere Pewter to my cart +- add 3 gallons of that paint to my cart +- add gloves to my cart +- clear shopping cart +- I want to see my current cart \ No newline at end of file diff --git a/ai_ref_kits/agentic_llm_rag/data/test_painting_llm_rag.pdf b/ai_ref_kits/agentic_llm_rag/data/test_painting_llm_rag.pdf new file mode 100644 index 00000000..7774c804 Binary files /dev/null and b/ai_ref_kits/agentic_llm_rag/data/test_painting_llm_rag.pdf differ diff --git a/ai_ref_kits/agentic_llm_rag/main.py b/ai_ref_kits/agentic_llm_rag/main.py new file mode 100644 index 00000000..6f73f75f --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/main.py @@ -0,0 +1,29 @@ +import argparse +from pathlib import Path + +import app +import convert_and_optimize_llm as chat + + +def main(args): + embedding_model_dir = chat.convert_embedding_model(args.embedding_model_type, Path(args.model_dir)) + chat_model_dir = chat.convert_chat_model(args.chat_model_type, args.chat_precision, Path(args.model_dir), args.hf_token) + + app.run(str(chat_model_dir.parent), str(embedding_model_dir.parent), Path(args.rag_pdf), "AUTO:GPU,CPU", args.public) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + + parser.add_argument("--chat_model_type", type=str, choices=["qwen2-7B", "llama3.2-3B"], + default="llama3.2-3B", help="Chat model to be converted") + parser.add_argument("--embedding_model_type", type=str, choices=["bge-small", "bge-large", "bge-m3"], + default="bge-small", help="Embedding model to be converted") + parser.add_argument("--chat_precision", type=str, default="int4", choices=["fp16", "int8", "int4"], help="Chat model precision") + parser.add_argument("--hf_token", type=str, help="HuggingFace access token to get Llama3") + parser.add_argument("--model_dir", type=str, default="model", help="Directory to place the model in") + parser.add_argument("--rag_pdf", type=str, default="data/test_painting_llm_rag.pdf", + help="Path to the PDF file which is an additional context") + parser.add_argument("--public", default=False, action="store_true", help="Whether interface should be available publicly") + + main(parser.parse_args()) \ No newline at end of file diff --git a/ai_ref_kits/agentic_llm_rag/requirements.txt b/ai_ref_kits/agentic_llm_rag/requirements.txt new file mode 100644 index 00000000..89d8dbd1 --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/requirements.txt @@ -0,0 +1,23 @@ +--extra-index-url https://download.pytorch.org/whl/cpu + +openvino==2025.0.0 +optimum-intel==1.21.0 +optimum==1.23.3 +nncf==2.14.1 + +llama-index==0.12.9 +llama-index-llms-openvino==0.4.0 +llama-index-embeddings-openvino==0.5.1 +llama-index-postprocessor-openvino-rerank==0.4.1 +llama-index-vector-stores-faiss==0.3.0 +faiss-cpu==1.9.0 +onnx==1.17.0; +onnxruntime==1.17.3 +torch==2.5.1 + +transformers==4.46.3 +librosa==0.10.2 +pyyaml==6.0.1 +PyMuPDF==1.24.10 + +gradio==5.12.0 diff --git a/ai_ref_kits/agentic_llm_rag/system_prompt.py b/ai_ref_kits/agentic_llm_rag/system_prompt.py new file mode 100644 index 00000000..5adc91ff --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/system_prompt.py @@ -0,0 +1,58 @@ +## DO NOT modify this prompt. This prompt is the ReactAgent default. +## You can modify the ## Additional Rules section if you want to add more rules +## Note: Adding extra context can confuse the model to go into "roleplay" mode instead of using tools. + +react_system_header_str = """\ + +You are designed to help with a variety of tasks, from answering questions \ + to providing summaries to other types of analyses. + +## Tools +You have access to a wide variety of tools. You are responsible for using +the tools in any sequence you deem appropriate to complete the task at hand. +This may require breaking the task into subtasks and using different tools +to complete each subtask. + +You have access to the following tools: +{tool_desc} + +## Output Format +To answer the question, please use the following format. + +``` +Thought: I need to use a tool to help me answer the question. +Action: tool name (one of {tool_names}) if using a tool. +Action Input: the input to the tool, in a JSON format representing the kwargs (e.g. {{"input": "hello world", "num_beams": 5}}) +``` + +Please ALWAYS start with a Thought. + +Please use a valid JSON format for the Action Input. Do NOT do this {{'input': 'hello world', 'num_beams': 5}}. + +If this format is used, the user will respond in the following format: + +``` +Observation: tool response +``` + +You should keep repeating the above format until you have enough information +to answer the question without using any more tools. At that point, you MUST respond +in the one of the following two formats: + +``` +Thought: I can answer without using any more tools. +Answer: [your answer here] +``` + +``` +Thought: I cannot answer the question with the provided tools. +Answer: Sorry, I cannot answer your query. +``` + +## Additional Rules +- End every sentence with a polite question to engage with the customer, include emojis about painting. + +## Current Conversation +Below is the current conversation consisting of interleaving human and assistant messages. + +""" \ No newline at end of file diff --git a/ai_ref_kits/agentic_llm_rag/tools.py b/ai_ref_kits/agentic_llm_rag/tools.py new file mode 100644 index 00000000..0714a678 --- /dev/null +++ b/ai_ref_kits/agentic_llm_rag/tools.py @@ -0,0 +1,103 @@ +import math + +class PaintCalculator: + + @staticmethod + def calculate_paint_cost(area: float, price_per_gallon: float, add_paint_supply_costs: bool = False) -> float: + """ + Calculate the total cost of paint needed for a given area. + + Args: + area: Area to be painted in square feet + price_per_gallon: Price per gallon of paint + add_paint_supply_costs: Whether to add $50 for painting supplies + + Returns: + Total cost of paint and supplies if requested + """ + gallons_needed = math.ceil((area / 400) * 2) # Assuming 2 gallons are needed for 400 square feet + total_cost = round(gallons_needed * price_per_gallon, 2) + if add_paint_supply_costs: + total_cost += 50 + return total_cost + + @staticmethod + def calculate_paint_gallons_needed(area: float) -> int: + """ + Calculate the number of gallons of paint needed for a given area. + + Args: + area: Area to be painted in square feet + + Returns: + Number of gallons needed (rounded up to ensure coverage) + """ + # Using the same formula as in PaintCostCalculator: 2 gallons needed for 400 square feet + gallons_needed = math.ceil((area / 400) * 2) + return gallons_needed + +class ShoppingCart: + # In-memory shopping cart + _cart_items = [] + + @staticmethod + def add_to_cart(product_name: str, quantity: int, price_per_unit: float) -> dict: + """ + Add an item to the shopping cart. + Add a product to a user's shopping cart. + This function ensures a seamless update to the shopping cart by specifying each required input clearly. + + Args: + product_name: Name of the paint product + quantity: Number of units/gallons + price_per_unit: Price per unit/gallon + + Returns: + Dict with confirmation message and current cart items + """ + item = { + "product_name": product_name, + "quantity": quantity, + "price_per_unit": price_per_unit, + "total_price": round(quantity * price_per_unit, 2) + } + + # Check if item already exists + for existing_item in ShoppingCart._cart_items: + if existing_item["product_name"] == product_name: + # Update quantity + existing_item["quantity"] += quantity + existing_item["total_price"] = round(existing_item["quantity"] * existing_item["price_per_unit"], 2) + return { + "message": f"Updated {product_name} quantity to {existing_item['quantity']} in your cart", + "cart": ShoppingCart._cart_items + } + + # Add new item + ShoppingCart._cart_items.append(item) + + return { + "message": f"Added {quantity} {product_name} to your cart", + "cart": ShoppingCart._cart_items + } + + @staticmethod + def get_cart_items() -> list: + """ + Get all items currently in the shopping cart. + + Returns: + List of items in the cart with their details + """ + return ShoppingCart._cart_items + + @staticmethod + def clear_cart() -> dict: + """ + Clear all items from the shopping cart. + + Returns: + Confirmation message + """ + ShoppingCart._cart_items = [] + return {"message": "Shopping cart has been cleared"} \ No newline at end of file