Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LCM to DIFFUSERS_TASKS_TO_MODEL_LOADERS #1762

Closed
wants to merge 2 commits into from
Closed

Add LCM to DIFFUSERS_TASKS_TO_MODEL_LOADERS #1762

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
317a7b4
Add LCM to DIFFUSERS_TASKS_TO_MODEL_LOADERS
l-bat Mar 21, 2024
f36e61e
Apply review comments
l-bat Apr 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions optimum/exporters/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,9 @@ class TasksManager:

_DIFFUSERS_TASKS_TO_MODEL_LOADERS = {
"stable-diffusion": "StableDiffusionPipeline",
"stable-diffusion-xl": "StableDiffusionXLImg2ImgPipeline",
"stable-diffusion-xl": "StableDiffusionXLPipeline",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_optional_components was introduced in diffusers v0.22.0 https://github.com/huggingface/diffusers/blob/v0.22.0/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L144, so we would need to upgrade DIFFUSERS_MINIMUM_VERSION as well to avoid any potential issue when loading a SDXL model (which don't have optional components)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DIFFUSERS_MINIMUM_VERSION was upgraded

"stable-diffusion-xl-refiner": "StableDiffusionXLImg2ImgPipeline",
"latent-consistency": "LatentConsistencyModelPipeline",
Comment on lines +192 to +193
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you extend on why this is needed ? To my knowledge this shouldn't be needed (both pipelines can be correctly loaded using the current _DIFFUSERS_TASKS_TO_MODEL_LOADERS

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also at the moment inferred task for diffusers models will either be "stable-diffusion-xl" or "stable-diffusion"

optimum/optimum/exporters/tasks.py

Line 1564 in dedb852

inferred_task_name = "stable-diffusion-xl" if "StableDiffusionXL" in class_name else "stable-diffusion"

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary to correctly determine the model class from task. Hybrid quantization supports the stable-diffusion-xl task, but doesn't support the stable-diffusion-xl-refiner. Without these changes, it is not possible to apply hybrid quantization via CLI.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure that we need it in the TaskManager, from what I see it's only needed to get the class to load the model in the optimum-intel CLI for diffusers models, could we move it in optimum-intel directly instead ? We could get this information by combining the task (text-to-image, image-to-image, inpaint, ..) and the architecture (SD, SDXL, LCMs, ..) using the model's config

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you suggesting to remove only the latent-consistency from _DIFFUSERS_TASKS_TO_MODEL_LOADERS?

As far as I understand, there are no text-to-image, image-to-image, inpaint,.. tasks in optimum. Should I create these tasks only for optimum-intel instead of stable-diffusion, stable-diffusion-xl, stable-diffusion-xl-refiner and latent-consistency? In this case, we cann't use TasksManager.infer_task_from_model for diffusion models.

}

_TIMM_TASKS_TO_MODEL_LOADERS = {
Expand Down Expand Up @@ -1561,7 +1563,12 @@ def _infer_task_from_model_name_or_path(

if library_name == "diffusers":
class_name = model_info.config["diffusers"]["class_name"]
inferred_task_name = "stable-diffusion-xl" if "StableDiffusionXL" in class_name else "stable-diffusion"
if "LatentConsistency" in class_name:
inferred_task_name = "latent-consistency"
elif "StableDiffusionXL" in class_name:
inferred_task_name = f"stable-diffusion-xl{'-refiner' if 'Img2Img' in class_name else ''}"
else:
inferred_task_name = "stable-diffusion"
elif library_name == "timm":
inferred_task_name = "image-classification"
else:
Expand Down
2 changes: 1 addition & 1 deletion optimum/utils/import_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ def _is_package_available(pkg_name: str, return_version: bool = False) -> Union[

TORCH_MINIMUM_VERSION = version.parse("1.11.0")
TRANSFORMERS_MINIMUM_VERSION = version.parse("4.25.0")
DIFFUSERS_MINIMUM_VERSION = version.parse("0.18.0")
DIFFUSERS_MINIMUM_VERSION = version.parse("0.22.0")
AUTOGPTQ_MINIMUM_VERSION = version.parse("0.4.99") # Allows 0.5.0.dev0


Expand Down