→

"a photo of a pink toy horse on the beach"

"a photo of a bronze horse in a museum"

"a photo of a robot horse"

→

"a photo of robots dancing"

"a cartoon of a couple dancing"

"a photo of a wooden sculpture of a couple dancing"

→

"a photorealistic image of bear cubs in the snow"

"a polygonal illustration of a cat and a bunny"

→

"a photo of sharks in the ocean"

"a polygonal illustration of fish in the ocean"

→

"a photo of a white wedding cake"

"a photo of a skyscraper"

"a photo of a tower in snow"

Previous

Next

Abstract

Large-scale text-to-image generative models have been a revolutionary breakthrough in the evolution of generative AI, allowing us to synthesize diverse images that convey highly complex visual concepts. However, a pivotal challenge in leveraging such models for real-world content creation tasks is providing users with control over the generated content. In this paper, we present a new framework that takes text-to-image synthesis to the realm of image-to-image translation -- given a guidance image and a target text prompt, our method harnesses the power of a pre-trained text-to-image diffusion model to generate a new image that complies with the target text, while preserving the semantic layout of the source image. Specifically, we observe and empirically demonstrate that fine-grained control over the generated structure can be achieved by manipulating spatial features and their self-attention inside the model. This results in a simple and effective approach, where features extracted from the guidance image are directly injected into the generation process of the target image, requiring no training or fine-tuning and applicable for both real or generated guidance images. We demonstrate high-quality results on versatile text-guided image translation tasks, including translating sketches, rough drawings and animations into realistic images, changing of the class and appearance of objects in a given image, and modifications of global qualities such as lighting and color.

Additional Results

CVPR 2023 Video

Paper

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation Narek Tumanyan *, Michal Geyer *, Shai Bagon, Tali Dekel. (* indicates equal contribution) [paper]

Supplementary Material

[supplementary page]

Bibtex

    @InProceedings{Tumanyan_2023_CVPR,
        author    = {Tumanyan, Narek and Geyer, Michal and Bagon, Shai and Dekel, Tali},
        title     = {Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation},
        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
        month     = {June},
        year      = {2023},
        pages     = {1921-1930}
    }