Skip to content

add PAG support for SD Img2Img#9463

Merged
yiyixuxu merged 24 commits into
huggingface:mainfrom
SahilCarterr:main
Oct 9, 2024
Merged

add PAG support for SD Img2Img#9463
yiyixuxu merged 24 commits into
huggingface:mainfrom
SahilCarterr:main

Conversation

@SahilCarterr

@SahilCarterr SahilCarterr commented Sep 18, 2024

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds PAG (Perturbed-Attention Guidance) support for SD models (StableDiffusionPAGImg2ImgPipeline).Continuation of #7944

Fixes #8710 (partially)

Before submitting

Who can review?

@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed.


Usage

from diffusers import AutoPipelineForImage2Image
from diffusers.utils import load_image, make_image_grid
import torch

# Set device to CUDA for GPU processing
device = "cuda"

# Load the image-to-image pipeline with pre-trained weights
pipe = AutoPipelineForImage2Image.from_pretrained("Lykon/dreamshaper-8", enable_pag=True)
pipe.to(device)

# Load an initial image
init_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"
)

# Set a random generator with a specific seed
generator = torch.Generator("cuda").manual_seed(31)

# Define the prompt
prompt = "cat wizard"

# Generate the output image using the pipeline
image = pipe(prompt, image=init_image, generator=generator, pag_scale=0.4).images[0]

# Create a grid of the initial and generated images
make_image_grid([init_image, image], rows=1, cols=2)

cat_wizrd

@yiyixuxu yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! I left some comments
can we see some outputs too? (saw it)!

Comment thread src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py
Comment thread src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py
Comment thread src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py
Comment thread src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated
Comment thread src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated
Comment thread src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated
Comment thread tests/pipelines/pag/test_pag_sd_img2img.py Outdated
Comment thread tests/pipelines/pag/test_pag_sd_img2img.py Outdated
@yiyixuxu yiyixuxu added the PAG label Sep 18, 2024
@yiyixuxu

Copy link
Copy Markdown
Collaborator

don't forget to call make style and make fix-copies too

@SahilCarterr

Copy link
Copy Markdown
Contributor Author

added "# copied from" , removed depreciated methods and fix test

@yiyixuxu yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! I think we still have one test needs to be removed, and then can merge!

), f"output is different from expected, {image_slice.flatten()}"

@slow
@require_torch_gpu

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test is not needed? (same as the one above it?)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i remove StableDiffusionPAGImg2ImgPipelineIntegrationTests(unittest.TestCase): also ?

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yiyixuxu yiyixuxu mentioned this pull request Sep 20, 2024
6 tasks
Comment thread src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated
@yiyixuxu

Copy link
Copy Markdown
Collaborator

@SahilCarterr can you look into failing tests?

@SahilCarterr

Copy link
Copy Markdown
Contributor Author
Screenshot 2024-09-25 at 10 47 58 PM

3 errors remaining to be fix

@SahilCarterr

Copy link
Copy Markdown
Contributor Author

Help Needed to fix the remaining errors

@SahilCarterr

Copy link
Copy Markdown
Contributor Author

The errors are now fixed @yiyixuxu
Screenshot 2024-10-06 at 1 25 56 PM

@yiyixuxu

yiyixuxu commented Oct 9, 2024

Copy link
Copy Markdown
Collaborator

can you run make style and make fix-copies?

@SahilCarterr

Copy link
Copy Markdown
Contributor Author

fixed make error

@yiyixuxu yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@yiyixuxu yiyixuxu merged commit af28ae2 into huggingface:main Oct 9, 2024
leisuzz pushed a commit to leisuzz/diffusers that referenced this pull request Oct 11, 2024
* added pag to sd img2img pipeline


---------

Co-authored-by: YiYi Xu <yixu310@gmail.com>
sayakpaul pushed a commit that referenced this pull request Dec 23, 2024
* added pag to sd img2img pipeline


---------

Co-authored-by: YiYi Xu <yixu310@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add PAG support to SD1.5

3 participants