You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: ai_ref_kits/README.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -96,7 +96,7 @@ The Custom AI Assistant, powered by the OpenVINO™ toolkit, integrates voice-ac
96
96
Understanding why computer vision models make certain predictions using data and model explainability can help us refine our models to be more efficient and performant. This solution demonstrates how to leverage the OpenVINO™ toolkit, Datumaro, and Ultralytics to generate data quality measurements and saliency maps to understand the predictions and performance of computer vision models during inference.
[](https://github.com/openvinotoolkit/openvino_build_deploy/blob/master/LICENSE.txt)
10
9
11
-
The Multimodal AI Visual Generator is designed for rapid prototyping, instant iteration, and seamless visualization of complex concepts. The kit integrates image creation with generative AI, automatic speech recognition (ASR), speech synthesis, large language models (LLMs), and natural language processing (NLP). It processes multimodal inputs from sources such as cameras, voice commands, or typed text to generate AI-driven visual outputs. Utilizing the Intel OpenVINO™ toolkit, the system enables seamless deployment of deep learning models across hardware platforms. Explore the demo to see its real-time visual generative AI workflow in action.
10
+
The kit integrates image creation with generative AI, voice activity detection (VAD), automatic speech recognition (ASR), large language models (LLMs), and natural language processing (NLP). A live voice transcription pipeline is connected to an LLM, which makes intelligent decisions about whether the user is describing the scene to an adventure game. When the LLM detects a new scene, the LLM will produce a detailed text prompt suitable for stable diffusion, which the application uses to illustrate the image. Utilizing the OpenVINO™ GenAI framework, this kit demonstrates the use of text2image, LLM pipeline, and whisper speech2text APIs.
Contributors: Ria Cheruvu, Garth Long, Arisha Kumar, Paula Ramos, Dmitriy Pastushenkov, Zhuo Wu, and Raymond Lo.
25
+
Contributors: Ryan Metcalfe, Garth Long, Arisha Kumar, Ria Cheruvu, Paula Ramos, Dmitriy Pastushenkov, Zhuo Wu, and Raymond Lo.
27
26
28
27
### What's New
29
28
@@ -44,18 +43,18 @@ Now, let's dive into the steps starting with installing Python.
44
43
45
44
## Installing Prerequisites
46
45
47
-
Now, let's dive into the steps starting with installing Python. We recommend using Ubuntu to set up and run this project. This project requires Python 3.8 or higher and a few libraries. If you don't have Python installed on your machine, go to https://www.python.org/downloads/ and download the latest version for your operating system. Follow the prompts to install Python, making sure to check the option to add Python to your PATH environment variable.
46
+
Now, let's dive into the steps starting with installing Python. This project requires Python 3.10 or higher and a few libraries. If you don't have Python installed on your machine, go to https://www.python.org/downloads/ and download the latest version for your operating system. Follow the prompts to install Python, making sure to check the option to add Python to your PATH environment variable.
48
47
49
48
Install libraries and tools:
50
49
50
+
If you're using Ubuntu, install required dependencies like this:
_NOTE: If you are using Windows, you will probably need to install [Microsoft Visual C++ Redistributable](https://aka.ms/vs/16/release/vc_redist.x64.exe) also._
56
55
57
56
## Setting Up Your Environment
58
-
### Cloning the Repository
57
+
### Cloning the Repository and Installing Dependencies
59
58
60
59
To clone the repository, run the following command:
61
60
@@ -69,58 +68,37 @@ The above will clone the repository into a directory named "openvino_build_deplo
69
68
cd openvino_build_deploy/ai_ref_kits/multimodal_ai_visual_generator
70
69
```
71
70
72
-
Next, you’ll download and optimize the required models. This will involve the creation of a temporary virtual environment and the running of a download script. Your requirements.txt file will depend on the Python version you're using (3.11 or 3.12).
71
+
Nextthe below will create a virtual environment, activate the environment, and install the required dependencies for the setup and execution of the project.
73
72
74
-
- nanoLLaVA (multimodal): Image recognition/captioning from webcam
75
-
- Whisper: Speech recognition
76
-
- Llama3-8b-instruct: Prompt refinement
77
-
- Latent Consistency Models: Image generation
78
-
79
-
**Note:** If you would like to run Latent Consistency Models on the NPU, as shown in the demo above, please follow the following steps: Download the model from this location "https://huggingface.co/Intel/sd-1.5-lcm-openvino" and compile it via the steps located at https://github.com/intel/openvino-ai-plugins-gimp/blob/v2.99-R3-staging/model_setup.py.
80
-
81
-
- AI Super Resolution: Increase the resolution of the generated image
82
-
- Depth Anything v2: Create 3d parallax animations
83
-
73
+
Linux:
84
74
```shell
85
-
python3 -m venv model_installation_venv
86
-
sourcemodel_installation_venv/bin/activate
75
+
python3 -m venv run_env
76
+
sourcerun_env/bin/activate
87
77
pip install -r requirements.txt
88
-
python3 download_and_prepare_models.py
89
-
```
90
-
After model installation, you can remove the `model_installation_venv` virtual environment as it is no longer needed.
91
-
92
-
### Creating a Virtual Environment
93
-
94
-
To create a virtual environment, open your terminal or command prompt and navigate to the directory where you want to create the environment. Then, run the following command:
95
-
96
-
```shell
97
-
python3 -m dnd_env
98
78
```
99
-
This will create a new virtual environment named "dnd_env" in the current directory.
100
-
101
-
### Activating the Environment
102
-
103
-
Activate the virtual environment using the following command:
104
79
80
+
Windows:
105
81
```shell
106
-
source dnd_env/bin/activate # For Unix-based operating systems such as Linux or macOS
107
-
```
108
-
109
-
_NOTE: If you are using Windows, use the `dnd_env\Scripts\activate` command instead._
110
-
111
-
This will activate the virtual environment and change your shell's prompt to indicate that you are now working within that environment.
82
+
python -m venv run_env
83
+
run_env/Scripts/activate
84
+
pip install -r requirements.txt
85
+
```
112
86
113
-
### Installing the Packages
87
+
### Downloading and Preparing Models
88
+
Next, you’ll download and optimize the required models via the running of a download script.
114
89
115
-
To install the required packages, run the following commands:
90
+
- Whisper: Speech recognition
91
+
- Llama3-8b-instruct: Intelligent LLM helper
92
+
- Latent Consistency Models: Image generation
93
+
- Super Resolution: Increase the resolution of the generated image
94
+
- Depth Anything v2: Create 3d parallax animations
To interact with the animated GIF outputs, host a simple web server on your system as the final output. To do so, please install Node.js via [its Download page](https://nodejs.org/en/download/package-manager) and [http-server](https://www.npmjs.com/package/http-server).
126
104
@@ -130,38 +108,62 @@ Run the following command to start an HTTP server within the repository. You can
130
108
http-server -c10
131
109
```
132
110
133
-
Open a terminal or you can use the existing one with `dnd_env` environment activated and start the Gradio GUI - <br>
111
+
Open a terminal or you can use the existing one with `run_env` environment activated and start the GUI - <br>
134
112
135
113
```shell
136
-
python3 gradio_ui.py
114
+
python app.py
137
115
```
138
116
139
-
Click on the web link to open the GUI in the web browser.
Take or upload a picture of any object via the Gradio image interface. Your "theme" will become the image description, if the object in the image is clearly captured.
120
+
### ➕ Set the theme for your story
121
+
This theme is passed as part of the system message to the LLM, and helps the LLM make more a more educated decision about whether you are describing the scene to a story, or not.
145
122
146
-
### 🗣 Speak your prompt
147
-
Start or upload a recording, wait for the server to listen, and speak your prompt to life. Click the “Stop” button to stop the generation.
123
+
### ➕ Click the Start Button
124
+
The start button will activate the listening state (Voice Activity Detection & Whisper Transcription pipelines) on the system's default input device (microphone).
148
125
149
-
### ➕ Add a theme to prompt
150
-
Now, your prompt is transcribed! Click the "Add Theme to Prompt" button to combine your prompt and theme.
126
+
### 🗣 Describe a scene to your story
127
+
Go ahead and describe a scene your story. For example, "You find yourself at the gates of a large, abandoned castle."
151
128
152
-
### ⚙️ Refine it with an LLM
153
-
You can optionally ask an LLM model to refine your model by clicking the LLM button. It will try its best to generate a prompt infusing the elements.
129
+
### 🖼️ Wait for your illustration
130
+
The scene that you just described will be passed to the LLM, which should detect it as a new scene to your story. The detailed prompt that is generated by the LLM will show up in real-time in the UI caption box, followed soon after by the illustration generated from the stable diffusion pipeline.
154
131
155
-
### 🖼️ Generate your image and depth map
156
-
Click "Generate Image" to see your image come to life. A depth map will automatically be generated for the image as well. Feel free to adjust the advanced parameters to control the image generation model.
132
+
### 🗣 Talk about something not relevant to your story
133
+
You can test the intelligence of the LLM helper and say something not relevant to the story. For example, "Hey guys, do you think we should order a pizza?". You should find that the LLM will make the decision to disregard this, and not try to illustrate anything.
157
134
158
135
### 🪄🖼️ Interact with the animated GIF
159
136
To interact with the 3D hoverable animation created with depth maps, start an HTTP server as explained above, and you will be able to interact with the parallax.
160
137
161
-
<palign="right"><ahref="#top">Back to top ⬆️</a></p>
138
+
## :bulb: Additional Tips
139
+
* Feel free to modify `main.py` to select different OpenVINO devices for the llm, stable diffusion pipeline, whisper, etc.
140
+
Look toward the bottom of the script, for a section that looks like this:
141
+
```
142
+
if __name__ == "__main__":
143
+
app = QApplication(sys.argv)
144
+
145
+
llm_device = 'GPU'
146
+
sd_device = 'GPU'
147
+
whisper_device = 'CPU'
148
+
super_res_device = 'GPU'
149
+
depth_anything_device = 'GPU'
150
+
```
151
+
If you're running on an Intel Core Ultra Series 2 laptop, and you want to set ```llm_device = 'NPU'```, be sure to have latest NPU driver installed, from [here](https://www.intel.com/content/www/us/en/download/794734/intel-npu-driver-windows.html)
152
+
153
+
* Based on the resolution of your display, you may want to tweak the default resolution of the illustrated image, as well as caption font size.
154
+
To adjust the resolution of the illustrated image, look for and modify this line:
155
+
```
156
+
self.image_label.setFixedSize(1216, 684)
157
+
```
158
+
It's recommended to choose a 16:9 ratio resolution. You can find a convenient list [here](https://pacoup.com/2011/06/12/list-of-true-169-resolutions/).
159
+
160
+
The caption font size can be adjusted by modifying this line:
161
+
```
162
+
fantasy_font = QFont("Papyrus", 18, QFont.Bold)
163
+
```
162
164
163
165
# Additional Resources
164
166
- Learn more about [OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html)
0 commit comments