Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix jit memory issue #627

Merged
merged 3 commits into from
Mar 26, 2024
Merged

fix jit memory issue #627

merged 3 commits into from
Mar 26, 2024

Conversation

jiqing-feng
Copy link
Collaborator

@jiqing-feng jiqing-feng commented Mar 22, 2024

Fix #604

Hi @echarlaix . As we discussed before, the jit trace memory issue can only workaround for now. I think the best way is to skip the save and load step and just return the cls after jit tracing. WDYT?

Jit trace memory issue: 96726

@jiqing-feng jiqing-feng marked this pull request as ready for review March 22, 2024 05:29
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yao-matrix
Copy link

@jiqing-feng , do you observe memory saving with your PR? any before vs after proof data?

@jiqing-feng
Copy link
Collaborator Author

jiqing-feng commented Mar 22, 2024

@jiqing-feng , do you observe memory saving with your PR? any before vs after proof data?

Script:

import torch
from optimum.intel.ipex import IPEXModelForCausalLM
model = IPEXModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.bfloat16, export=True)

Original model size: 13.5G

Memory size before change: 51.7G
image

Memory size after my change: 13.6G
image

jiqing-feng and others added 3 commits March 22, 2024 09:22
Copy link
Collaborator

@echarlaix echarlaix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks @jiqing-feng

@echarlaix echarlaix merged commit e261e21 into huggingface:main Mar 26, 2024
10 checks passed
@jiqing-feng jiqing-feng deleted the jit_memory branch April 9, 2024 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory not release after jit.trace
4 participants