Register Model as LoRA Hook 🎭🅐🅓¶
Documentation¶
- Class name:
ADE_RegisterModelAsLoraHook
- Category:
Animate Diff 🎭🅐🅓/conditioning/register lora hooks
- Output node:
False
This node is designed to register a model as a LoRA hook within the AnimateDiff framework, enabling dynamic modification and control of model behavior during the generation process. It facilitates the integration of LoRA (Low-Rank Adaptation) techniques with models, allowing for enhanced customization and fine-tuning of model outputs based on specific conditions or inputs.
Input types¶
Required¶
model
- The model parameter represents the target model to which the LoRA hook will be applied. It is crucial for defining the scope and context of the LoRA adaptation, affecting how the model's behavior is modified during the generation process.
- Comfy dtype:
MODEL
- Python dtype:
Union[ModelPatcher, ModelPatcherAndInjector]
clip
- The clip parameter specifies the CLIP model involved in the process, if any, providing context for where LoRA adaptations may be applied alongside the primary model.
- Comfy dtype:
CLIP
- Python dtype:
CLIP
ckpt_name
- The ckpt_name parameter indicates the checkpoint name for loading the model's state, essential for initializing the model with specific pre-trained weights or configurations.
- Comfy dtype:
COMBO[STRING]
- Python dtype:
str
strength_model
- This parameter controls the intensity of the LoRA modifications applied to the model. It allows for fine-tuning the impact of LoRA on the model's behavior, enabling precise adjustments to the generation process.
- Comfy dtype:
FLOAT
- Python dtype:
float
strength_clip
- Similar to strength_model, this parameter adjusts the intensity of LoRA modifications but specifically for the CLIP model involved in the process. It enables separate control over the adaptation strength for different components of the generation framework.
- Comfy dtype:
FLOAT
- Python dtype:
float
Output types¶
model
- Comfy dtype:
MODEL
- The model with LoRA hooks applied, reflecting the dynamic adaptations specified through the input parameters.
- Python dtype:
ModelPatcher
- Comfy dtype:
clip
- Comfy dtype:
CLIP
- The CLIP model with LoRA hooks applied, if a CLIP model was involved in the process. It indicates the application of LoRA modifications to the CLIP component.
- Python dtype:
CLIP
- Comfy dtype:
lora_hook
- Comfy dtype:
LORA_HOOK
- The LoRA hook that has been registered and applied to the model and/or CLIP, encapsulating the specific adaptations made.
- Python dtype:
LoraHook
- Comfy dtype:
Usage tips¶
- Infra type:
CPU
- Common nodes: unknown
Source code¶
class MaskableSDModelLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"model": ("MODEL",),
"clip": ("CLIP",),
"ckpt_name": (folder_paths.get_filename_list("checkpoints"), ),
"strength_model": ("FLOAT", {"default": 1.0, "min": -20.0, "max": 20.0, "step": 0.01}),
"strength_clip": ("FLOAT", {"default": 1.0, "min": -20.0, "max": 20.0, "step": 0.01}),
}
}
RETURN_TYPES = ("MODEL", "CLIP", "LORA_HOOK")
CATEGORY = "Animate Diff 🎭🅐🅓/conditioning/register lora hooks"
FUNCTION = "load_model_as_lora"
def load_model_as_lora(self, model: ModelPatcher, clip: CLIP, ckpt_name: str, strength_model: float, strength_clip: float):
ckpt_path = folder_paths.get_full_path("checkpoints", ckpt_name)
out = comfy.sd.load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings"))
model_loaded = out[0]
clip_loaded = out[1]
lora_hook = LoraHook(lora_name=ckpt_name)
lora_hook_group = LoraHookGroup()
lora_hook_group.add(lora_hook)
model_lora, clip_lora = load_model_as_hooked_lora_for_models(model=model, clip=clip,
model_loaded=model_loaded, clip_loaded=clip_loaded,
lora_hook=lora_hook,
strength_model=strength_model, strength_clip=strength_clip)
return (model_lora, clip_lora, lora_hook_group)