feat(sd3): support LoRAs

docs/src/content/docs/Users Guide/03.Models/03.model-format-support.mdx

@@ -104,7 +104,8 @@ not check, so a quantized file there fails with an unclear error or decodes inco
 | FLUX.2, Z-Image | ✓ (LyCORIS, Diffusers) | FLUX.2 [dev] (single-file checkpoint), Z-Image | |
 | Qwen-Image, Krea-2, Wan 2.2, Anima | ✓ (LyCORIS/kohya, including LoKR where noted on the family page) | Qwen-Image (single-file checkpoint), Anima (ControlNet-LLLite) | |
 | LTX-2, MiniMax H3 | ✓ (plain low-rank; LoKR, LoHA and DoRA are refused) | ✗ | |
-| SD 3, CogView4, Ideogram 4, ERNIE-Image, Qwen-Image-2.1 | ✗ | ✗ | |
+| SD 3 / 3.5 | ✓ (diffusers, PEFT, SimpleTuner LyCORIS, kohya; CLIP-L/G too; DoRA is refused) | ✗ | |
+| CogView4, Ideogram 4, ERNIE-Image, Qwen-Image-2.1 | ✗ | ✗ | |
 
 [PiD decoders](/users-guide/image-generation/advanced-topics/pid-decode/) load for FLUX.1, FLUX.2, SD 3, SDXL and
 Qwen-Image (v1 and v1.5, bf16 and int8 builds).

docs/src/content/docs/Users Guide/03.Models/Local Models/sd-3.mdx

@@ -1,7 +1,7 @@
 ---
 title: Stable Diffusion 3.5
 description: Generate images with Stability AI's Stable Diffusion 3.5 Medium and Large models, including text encoder options and PiD super-resolution decoding.
-lastUpdated: 2026-09-30
+lastUpdated: 2026-10-12
 sidebar:
   order: 11
 ---
@@ -83,7 +83,29 @@ reference settings:
 ## Canvas
 
 On the Canvas, SD3 supports text-to-image, image-to-image, inpainting and outpainting. ControlNet,
-reference images and regional guidance are **not** available for SD3. LoRAs are not supported either.
+reference images and regional guidance are **not** available for SD3.
+
+## LoRA
+
+SD 3 and SD 3.5 LoRAs are supported in the layouts trainers publish them in: the diffusers DreamBooth script (a folder
+with `pytorch_lora_weights.safetensors`, or the file itself), ai-toolkit, PEFT adapters, SimpleTuner's LyCORIS files
+(LoKR) and kohya sd-scripts (`lora_unet_joint_blocks_…`). A LoRA that also trains the CLIP-L and CLIP-G text encoders
+applies to them too; T5 layers in a LoRA are skipped with a warning, because the T5 encoder takes no LoRA. DoRA files,
+and files with layers that match no SD 3 module, are refused when you install them.
+
+- **Medium and Large.** A LoRA is trained for one size: SD 3 Medium and SD 3.5 Medium share one, SD 3.5 Large (and
+  Large Turbo) the other. Invoke records the size of LoRAs and of SD 3 models when you install them, and the concept
+  picker only offers LoRAs that fit the selected model. SD 3 models installed with an earlier version carry no size;
+  the picker then offers every SD 3 LoRA. To add it, select the model in the Model Manager and use **Re-identify**
+  (which also resets the model's default settings). A LoRA that does not fit is refused when the generation starts,
+  before anything is applied.
+- **SD 3.5 Medium LoRAs on SD 3 Medium.** SD 3.5 Medium adds a second attention to its first 13 blocks. A LoRA that
+  trains it does not fit SD 3 Medium and is refused there.
+- **FP8 Storage.** With FP8 Storage the LoRA is applied alongside the weights instead of being merged into them; it then
+  stays in VRAM for the whole generation, and InvokeAI reserves room for it.
+
+Earlier versions identified some SD 3 LoRAs as Qwen-Image LoRAs and others not at all. Select such a LoRA in the Model
+Manager and use **Re-identify** to fix it.
 
 ## PiD super-resolution decode
 

docs/src/content/docs/Users Guide/04.Image Generation/generate-panel.mdx

@@ -101,7 +101,7 @@ Concepts, also called LoRAs, are small add-on models that teach the main model a
 
 </Steps>
 
-You can add as many concepts as you like and combine them. Use the switch on each row to turn a concept off without removing it, and the trash button to remove it. If you change to a model from a different family, its concepts are marked **Incompatible** or removed. SD3, CogView4, ERNIE Image, Ideogram 4, Qwen-Image-2.1 and external API models don't support concepts, so with one of them selected the section says so instead of offering LoRAs.
+You can add as many concepts as you like and combine them. Use the switch on each row to turn a concept off without removing it, and the trash button to remove it. If you change to a model from a different family, its concepts are marked **Incompatible** or removed. CogView4, ERNIE Image, Ideogram 4, Qwen-Image-2.1 and external API models don't support concepts, so with one of them selected the section says so instead of offering LoRAs.
 
 ## Adding reference images
 

invokeai/app/invocations/sd3/sd3_denoise.py

@@ -1,3 +1,4 @@
+from contextlib import ExitStack
 from typing import Callable, Optional, Tuple
 
 import torch
@@ -24,19 +25,35 @@
 from invokeai.app.invocations.text_encoder.sd3_text_encoder import SD3_T5_MAX_SEQ_LEN
 from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.backend.flux.sampling_utils import clip_timestep_schedule_fractional
-from invokeai.backend.model_manager.taxonomy import BaseModelType
+from invokeai.backend.model_manager.taxonomy import BaseModelType, SD3VariantType
+from invokeai.backend.patches.layer_patcher import LayerPatcher, PatchSpec
+from invokeai.backend.patches.layers.lora_layer import LoRALayer
+from invokeai.backend.patches.lora_conversions.sd3_lora_constants import (
+    SD3_LORA_TRANSFORMER_PREFIX,
+    sd3_variant_from_width,
+)
+from invokeai.backend.patches.model_patch_raw import ModelPatchRaw
+from invokeai.backend.quantization.dequantizing_linear import requires_sidecar_patching
 from invokeai.backend.rectified_flow.rectified_flow_inpaint_extension import RectifiedFlowInpaintExtension
 from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState
 from invokeai.backend.stable_diffusion.diffusion.conditioning_data import SD3ConditioningInfo
 from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.fp8 import FP8_STORAGE_DTYPES
+
+# How the sizes read in a message; the UI uses the same names.
+_SD3_SIZES = {SD3VariantType.Medium: "SD 3 / 3.5 Medium", SD3VariantType.Large: "SD 3.5 Large"}
+
+# What one sidecar-patched layer holds while it computes its residual -- a second copy of the layer's output, or for a
+# LoKR/LoHA layer its full weight delta. Same margin as LTX-2 and ERNIE-Image.
+_SIDECAR_TRANSIENT_BYTES = 1024**3
 
 
 @invocation(
     "sd3_denoise",
     title="Denoise - SD3",
     tags=["image", "sd3"],
     category="latents",
-    version="1.2.0",
+    version="1.3.0",
 )
 class SD3DenoiseInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Run denoising process with a SD3 model."""
@@ -280,8 +297,43 @@ def _run_diffusion(
             ),
         )
 
-        with transformer_info.model_on_device() as (cached_weights, transformer):
+        # Loaded before the lock, so a LoRA that does not fit fails before the transformer moves, and the
+        # reservation below can be sized from the patches.
+        lora_patch_specs = self._materialize_lora_patches(context)
+        self._raise_unless_loras_fit(context, transformer_info.model, lora_patch_specs)
+        transformer_config = context.models.get_config(self.transformer.transformer)
+        force_sidecar = bool(lora_patch_specs) and requires_sidecar_patching(
+            transformer_info.model, transformer_config.format
+        )
+        # The patcher also picks the sidecar on its own for fp8 weights (FP8 Storage), which cannot be added to.
+        lora_on_device = force_sidecar or (
+            bool(lora_patch_specs) and any(p.dtype in FP8_STORAGE_DTYPES for p in transformer_info.model.parameters())
+        )
+        # This node has no activation estimate, so the cache floors its request at `device_working_mem_gb`, which then
+        # stands in for the activations. Sidecar patches stay on the device for the whole run, so they come on top of
+        # that floor rather than out of it. Patches merged into the weights leave nothing resident.
+        working_mem_bytes: Optional[int] = None
+        if lora_on_device:
+            floor = int(context.config.get().device_working_mem_gb * 2**30)
+            lora_bytes = sum(spec[0].calc_size() for spec in lora_patch_specs)
+            working_mem_bytes = floor + lora_bytes + _SIDECAR_TRANSIENT_BYTES
+
+        with ExitStack() as exit_stack:
+            (cached_weights, transformer) = exit_stack.enter_context(
+                transformer_info.model_on_device(working_mem_bytes=working_mem_bytes)
+            )
             assert isinstance(transformer, SD3Transformer2DModel)
+            if lora_patch_specs:
+                exit_stack.enter_context(
+                    LayerPatcher.apply_smart_model_patches(
+                        model=transformer,
+                        patches=lora_patch_specs,
+                        prefix=SD3_LORA_TRANSFORMER_PREFIX,
+                        dtype=inference_dtype,
+                        cached_weights=cached_weights,
+                        force_sidecar_patching=force_sidecar,
+                    )
+                )
 
             # 6. Denoising loop
             for step_idx, (t_curr, t_prev) in tqdm(
@@ -329,6 +381,55 @@ def _run_diffusion(
 
         return latents
 
+    def _materialize_lora_patches(self, context: InvocationContext) -> list[PatchSpec]:
+        """Load every LoRA on the transformer field into (patch, weight, cache-pin) specs."""
+        patch_specs: list[PatchSpec] = []
+        for lora in self.transformer.loras:
+            lora_info = context.models.load(lora.lora)
+            if not isinstance(lora_info.model, ModelPatchRaw):
+                raise TypeError(
+                    f"Expected ModelPatchRaw for LoRA '{lora.lora.key}', got {type(lora_info.model).__name__}."
+                )
+            patch_specs.append((lora_info.model, lora.weight, lora_info.model_in_ram()))
+        return patch_specs
+
+    def _raise_unless_loras_fit(
+        self, context: InvocationContext, transformer: SD3Transformer2DModel, patch_specs: list[PatchSpec]
+    ) -> None:
+        """Refuse a LoRA trained for another SD 3 size, or for modules this model does not have.
+
+        Medium and Large name the same modules at different widths, so the patcher would only log and skip every
+        layer; and a LoRA for SD 3.5 Medium's dual attention (`attn2`) targets modules SD 3 Medium lacks.
+        """
+        width = int(transformer.config.num_attention_heads) * int(transformer.config.attention_head_dim)
+        model_variant = sd3_variant_from_width(width)
+        for lora, spec in zip(self.transformer.loras, patch_specs, strict=True):
+            name = lora.lora.name or lora.lora.key
+            lora_variant = getattr(context.models.get_config(lora.lora), "variant", None)
+            if lora_variant is not None and model_variant is not None and lora_variant is not model_variant:
+                # A hand-built workflow can route another base's LoRA here, with a variant SD 3 has no name for.
+                trained_for = _SD3_SIZES.get(lora_variant, getattr(lora_variant, "value", lora_variant))
+                raise ValueError(
+                    f"LoRA '{name}' was trained for {trained_for} and does not fit this "
+                    f"{_SD3_SIZES[model_variant]} model."
+                )
+            for key, layer in spec[0].layers.items():
+                if not key.startswith(SD3_LORA_TRANSFORMER_PREFIX):
+                    continue
+                path = key.removeprefix(SD3_LORA_TRANSFORMER_PREFIX)
+                try:
+                    module = transformer.get_submodule(path)
+                except AttributeError:
+                    hint = " (it was trained for SD 3.5 Medium's dual attention)" if ".attn2." in path else ""
+                    raise ValueError(f"LoRA '{name}' patches `{path}`, which this SD 3 model does not have{hint}.")
+                if isinstance(layer, LoRALayer) and (
+                    layer.up.shape[0] != module.weight.shape[0] or layer.down.shape[1] != module.weight.shape[1]
+                ):
+                    raise ValueError(
+                        f"LoRA '{name}' does not fit this SD 3 model: `{path}` is {tuple(module.weight.shape)} here. "
+                        "It was trained for another SD 3 size."
+                    )
+
     def _prepare_noise_tensor(
         self, context: InvocationContext, num_channels_latents: int, inference_dtype: torch.dtype, device: torch.device
     ) -> torch.Tensor:

invokeai/app/invocations/sd3/sd3_lora_loader.py

@@ -0,0 +1,171 @@
+from typing import Optional
+
+from invokeai.app.invocations.baseinvocation import (
+    BaseInvocation,
+    BaseInvocationOutput,
+    Classification,
+    invocation,
+    invocation_output,
+)
+from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField
+from invokeai.app.invocations.metadata import LoRAMetadataField
+from invokeai.app.invocations.model import CLIPField, LoRAField, ModelIdentifierField, TransformerField
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelType
+
+
+@invocation_output("sd3_lora_loader_output")
+class SD3LoRALoaderOutput(BaseInvocationOutput):
+    """SD3 LoRA loader output."""
+
+    transformer: Optional[TransformerField] = OutputField(
+        default=None, description=FieldDescriptions.transformer, title="Transformer"
+    )
+    clip_l: Optional[CLIPField] = OutputField(default=None, description=FieldDescriptions.clip, title="CLIP L")
+    clip_g: Optional[CLIPField] = OutputField(default=None, description=FieldDescriptions.clip, title="CLIP G")
+    # The model fields carry LoRAField entries, which Core Metadata cannot take. This echoes the applied LoRA in the
+    # shape `core_metadata.loras` wants, so a workflow can record it without retyping the model key.
+    lora_metadata: LoRAMetadataField = OutputField(
+        description="The applied LoRA, in the shape Core Metadata's `loras` field takes.",
+        title="LoRA Metadata",
+    )
+
+
+@invocation_output("sd3_lora_collection_loader_output")
+class SD3LoRACollectionLoaderOutput(BaseInvocationOutput):
+    """SD3 LoRA collection loader output."""
+
+    transformer: Optional[TransformerField] = OutputField(
+        default=None, description=FieldDescriptions.transformer, title="Transformer"
+    )
+    clip_l: Optional[CLIPField] = OutputField(default=None, description=FieldDescriptions.clip, title="CLIP L")
+    clip_g: Optional[CLIPField] = OutputField(default=None, description=FieldDescriptions.clip, title="CLIP G")
+    lora_metadata: list[LoRAMetadataField] = OutputField(
+        default=[],
+        description="The applied LoRAs, in the shape Core Metadata's `loras` field takes.",
+        title="LoRA Metadata",
+    )
+
+
+def _raise_unless_sd3_lora(context: InvocationContext, lora_key: str) -> None:
+    """Refuse a key that does not resolve to an installed SD 3 LoRA.
+
+    The identifier's own base/type are client-supplied: a hand-written workflow can label any model key as an SD 3
+    LoRA and so reach model patching. Only the stored config is authoritative.
+    """
+    if not context.models.exists(lora_key):
+        raise ValueError(f"Unknown lora: {lora_key}!")
+    stored_config = context.models.get_config(lora_key)
+    if stored_config.type is not ModelType.LoRA or stored_config.base is not BaseModelType.StableDiffusion3:
+        raise ValueError(
+            f"Model '{lora_key}' is not an SD 3 LoRA (resolved to "
+            f"type={getattr(stored_config.type, 'value', stored_config.type)}, "
+            f"base={getattr(stored_config.base, 'value', stored_config.base)})."
+        )
+
+
+def _with_lora(field, lora: LoRAField):
+    """A copy of a model field with ``lora`` appended, or ``None`` for an unconnected input."""
+    if field is None:
+        return None
+    copied = field.model_copy(deep=True)
+    copied.loras.append(lora)
+    return copied
+
+
+@invocation(
+    "sd3_lora_loader",
+    title="Apply LoRA - SD3",
+    tags=["lora", "model", "sd3"],
+    category="model",
+    version="1.0.0",
+    classification=Classification.Prototype,
+)
+class SD3LoRALoaderInvocation(BaseInvocation):
+    """Apply a LoRA model to an SD3 transformer and its CLIP encoders.
+
+    A LoRA patches only what it trained; T5 layers in a LoRA are dropped, since the T5 encoder takes no LoRA.
+    """
+
+    lora: ModelIdentifierField = InputField(
+        description=FieldDescriptions.lora_model,
+        title="LoRA",
+        ui_model_base=BaseModelType.StableDiffusion3,
+        ui_model_type=ModelType.LoRA,
+    )
+    weight: float = InputField(default=0.75, description=FieldDescriptions.lora_weight)
+    transformer: TransformerField | None = InputField(
+        default=None, description=FieldDescriptions.transformer, input=Input.Connection, title="Transformer"
+    )
+    clip_l: CLIPField | None = InputField(
+        default=None, description=FieldDescriptions.clip, input=Input.Connection, title="CLIP L"
+    )
+    clip_g: CLIPField | None = InputField(
+        default=None, description=FieldDescriptions.clip, input=Input.Connection, title="CLIP G"
+    )
+
+    def invoke(self, context: InvocationContext) -> SD3LoRALoaderOutput:
+        lora_key = self.lora.key
+        _raise_unless_sd3_lora(context, lora_key)
+
+        for role, field in (("transformer", self.transformer), ("CLIP L", self.clip_l), ("CLIP G", self.clip_g)):
+            if field is not None and any(lora.lora.key == lora_key for lora in field.loras):
+                raise ValueError(f'LoRA "{lora_key}" already applied to {role}.')
+
+        lora = LoRAField(lora=self.lora, weight=self.weight)
+        return SD3LoRALoaderOutput(
+            transformer=_with_lora(self.transformer, lora),
+            clip_l=_with_lora(self.clip_l, lora),
+            clip_g=_with_lora(self.clip_g, lora),
+            lora_metadata=LoRAMetadataField(model=self.lora, weight=self.weight),
+        )
+
+
+@invocation(
+    "sd3_lora_collection_loader",
+    title="Apply LoRA Collection - SD3",
+    tags=["lora", "model", "sd3"],
+    category="model",
+    version="1.0.0",
+    classification=Classification.Prototype,
+)
+class SD3LoRACollectionLoader(BaseInvocation):
+    """Apply a collection of LoRAs to an SD3 transformer and its CLIP encoders."""
+
+    loras: Optional[LoRAField | list[LoRAField]] = InputField(
+        default=None,
+        description="LoRAs to apply. May be a single LoRA or a collection.",
+        title="LoRAs",
+        ui_model_base=[BaseModelType.StableDiffusion3],
+        ui_model_type=ModelType.LoRA,
+    )
+    transformer: Optional[TransformerField] = InputField(
+        default=None, description=FieldDescriptions.transformer, input=Input.Connection, title="Transformer"
+    )
+    clip_l: CLIPField | None = InputField(
+        default=None, description=FieldDescriptions.clip, input=Input.Connection, title="CLIP L"
+    )
+    clip_g: CLIPField | None = InputField(
+        default=None, description=FieldDescriptions.clip, input=Input.Connection, title="CLIP G"
+    )
+
+    def invoke(self, context: InvocationContext) -> SD3LoRACollectionLoaderOutput:
+        output = SD3LoRACollectionLoaderOutput(
+            transformer=self.transformer.model_copy(deep=True) if self.transformer is not None else None,
+            clip_l=self.clip_l.model_copy(deep=True) if self.clip_l is not None else None,
+            clip_g=self.clip_g.model_copy(deep=True) if self.clip_g is not None else None,
+        )
+        if self.loras is None:
+            return output
+
+        for lora in self.loras if isinstance(self.loras, list) else [self.loras]:
+            lora_key = lora.lora.key
+            _raise_unless_sd3_lora(context, lora_key)
+            # A LoRA already on a field (from an earlier loader, or twice in the collection) is not applied again.
+            fields = [f for f in (output.transformer, output.clip_l, output.clip_g) if f is not None]
+            missing = [f for f in fields if not any(applied.lora.key == lora_key for applied in f.loras)]
+            for field in missing:
+                field.loras.append(lora)
+            if missing:
+                output.lora_metadata.append(LoRAMetadataField(model=lora.lora, weight=lora.weight))
+        return output