Add remove adapters for LLMpipeline #1852

wenyi5608 · 2025-03-06T03:35:21Z

tickets: CVS-160984

remove adapters

wenyi5608 · 2025-03-17T08:32:58Z

Will using mmap for adapter content be helpful in your use case and eliminate the need for remove_adapters?

I had used mmap for adapter content and eliminate 'romove_adapters'. The user need to encrypt the lora adapter, so I added new API to read the lora adapter from the buffer.

slyalin · 2025-03-17T09:26:23Z

src/cpp/src/lora_adapter.cpp

-            "Tensor shape ", ov::shape_size(shape), " for tensor \"", name, "\" from Safetensors file \"", filename, "\" doesn't match the expected tensor size ",
+            "Tensor shape ", ov::shape_size(shape), " for tensor \"", name, "\" from Safetensors file \"", "safetensor", "\" doesn't match the expected tensor size ",


Please don't lose a better diagnostics for a file due to the code unification. Most users will still use adapters from files, so this message becomes less useful for them.

src/cpp/src/icontinuous_batching.cpp

src/cpp/src/lora_adapter.cpp

iefode · 2025-03-17T11:13:17Z

src/cpp/src/lora_adapter.cpp

+}
+
+// Reads a safetensor and creates a constantmap from the memory.
+ConstantMap read_safetensors(const ov::Tensor& safetensor) {


Why do we need this function? It it whole duplication of safetensors_init(safetensor). We can move second one to public section in this case

src/cpp/include/openvino/genai/lora_adapter.hpp

wenyi5608 added 20 commits March 5, 2025 15:43

Update continuous_batching_pipeline.hpp for remove adapters

cd3452b

Update llm_pipeline.hpp

58d7928

remove adapters

Update lora_adapter.hpp

913086f

remove adapters

Update continuous_batching_adapter.hpp

e10844d

remove adapters

Update continuous_batching_impl.cpp

63a260c

remove adapters

Update continuous_batching_impl.hpp

dc1e445

remove adapters

Update continuous_batching_pipeline.cpp

a61e966

Update icontinuous_batching.cpp

9d03f49

remove adapters

Update icontinuous_batching.hpp

d180c87

remove adapters

Update llm_pipeline.cpp

3fe9c56

remove adapters

Update llm_pipeline_base.hpp

c15f23d

remove adapters

Update llm_pipeline_stateful.cpp

57f75cf

remove adapters

Update llm_pipeline_stateful.hpp

4fd3200

remove adapters

Update llm_pipeline_static.cpp

2052795

remove adapters

Update llm_pipeline_static.hpp

8c35a3f

remove adapters

Update lora_adapter.cpp

78f4be7

remove adapters

Update prompt_lookup_impl.cpp

2b076fc

remove adapters

Update prompt_lookup_impl.hpp

837070b

remove adapters

Update speculative_decoding_impl.cpp

6e14676

remove adapters

Update speculative_decoding_impl.hpp

972527e

remove adapters

ilya-lavrenov requested review from Wovchena and removed request for Wovchena March 6, 2025 07:19

ilya-lavrenov assigned slyalin Mar 6, 2025

wenyi5608 added 12 commits March 17, 2025 15:45

Update llm_pipeline.cpp

aeff485

Update llm_pipeline_base.hpp

56d5561

Update llm_pipeline_stateful.cpp

c3a0db0

Update llm_pipeline_stateful.hpp

5b02537

Update llm_pipeline_static.cpp

7de647a

Update llm_pipeline_static.hpp

8b4bf86

Update lora_adapter.cpp

352a82b

Update prompt_lookup_impl.cpp

2a2e3fe

Update prompt_lookup_impl.hpp

7cb3fbb

Update speculative_decoding_impl.cpp

cd5337d

Update speculative_decoding_impl.hpp

b418105

Update lora_adapter.hpp

4abb135

slyalin requested changes Mar 17, 2025

View reviewed changes

iefode removed category: speculative decoding Speculative decoding category: prompt lookup labels Mar 17, 2025

iefode reviewed Mar 17, 2025

View reviewed changes

ilya-lavrenov reviewed Mar 17, 2025

View reviewed changes

src/cpp/include/openvino/genai/lora_adapter.hpp Show resolved Hide resolved

ilya-lavrenov added this to the 2025.2 milestone Mar 17, 2025

Update icontinuous_batching.cpp

5d90e7a

github-actions bot added category: speculative decoding Speculative decoding category: prompt lookup labels Mar 19, 2025

wenyi5608 added 3 commits March 19, 2025 11:31

Update llm_pipeline_static.cpp

7b98968

Update lora_adapter.cpp

4d3ff30

Update py_openvino_genai.pyi

1e5411d

github-actions bot added the category: Python API Python API for GenAI label Mar 19, 2025

Update py_lora_adapter.cpp

de4c06a

ilya-lavrenov requested a review from iefode March 19, 2025 06:59

ilya-lavrenov assigned iefode Mar 19, 2025

ilya-lavrenov requested a review from slyalin March 19, 2025 06:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add remove adapters for LLMpipeline #1852

Add remove adapters for LLMpipeline #1852

wenyi5608 commented Mar 6, 2025 •

edited by peterchen-intel

Loading

wenyi5608 commented Mar 17, 2025

slyalin Mar 17, 2025

iefode Mar 17, 2025

		"Tensor shape ", ov::shape_size(shape), " for tensor \"", name, "\" from Safetensors file \"", filename, "\" doesn't match the expected tensor size ",
		"Tensor shape ", ov::shape_size(shape), " for tensor \"", name, "\" from Safetensors file \"", "safetensor", "\" doesn't match the expected tensor size ",

Add remove adapters for LLMpipeline #1852

Are you sure you want to change the base?

Add remove adapters for LLMpipeline #1852

Conversation

wenyi5608 commented Mar 6, 2025 • edited by peterchen-intel Loading

wenyi5608 commented Mar 17, 2025

slyalin Mar 17, 2025

Choose a reason for hiding this comment

iefode Mar 17, 2025

Choose a reason for hiding this comment

wenyi5608 commented Mar 6, 2025 •

edited by peterchen-intel

Loading