Skip to content

bearstonem/StoryDiffusion

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation Paper page

[Paper]   [Project Page]   [🤗 Comic Generation Demo ]


Official implementation of StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation.

Demo Video

28a42f-2cf8-a5b-ec52-f3aaba486f.mp4

🌠 Key Features:

StoryDiffusion can create a magic story by generating consistent images and videos. Our work mainly has two parts:

  1. Consistent self-attention for character-consistent image generation over long-range sequences. It is hot-pluggable and compatible with all SD1.5 and SDXL-based image diffusion models. For the current implementation, the user needs to provide at least 3 text prompts for the consistent self-attention module. We recommend at least 5 - 6 text prompts for better layout arrangement.
  2. Motion predictor for long-range video generation, which predicts motion between Condition Images in a compressed image semantic space, achieving larger motion prediction.

🔥 Examples

Comics generation

1

Image-to-Video generation (Results are HIGHLY compressed for speed)

Leveraging the images produced through our Consistent Self-Attention mechanism, we can extend the process to create videos by seamlessly transitioning between these images. This can be considered as a two-stage long video generation approach.

Note: results are highly compressed for speed, you can visit our website for the high-quality version.

Two-stage Long Videos Generation (New Update)

Combining the two parts, we can generate very long and high-quality AIGC videos.

Video1 Video2 Video3

Long Video Results using Condition Images

Our Image-to-Video model can generate a video by providing a sequence of user-input condition images.

Video1 Video2 Video3
Video4 Video5 Video6

Short Videos

Video1 Video2 Video3
Video4 Video5 Video6

🚩 TODO/Updates

  • Commic Results of StoryDiffusion.
  • Video Results of StoryDiffusion.
  • Source code of Comic Generation
  • Source code of gradio demo
  • Source code of Video Generation Model
  • Pretrained weight of Video Generation Model

🔧 Dependencies and Installation

conda create --name storydiffusion python=3.10
conda activate storydiffusion
pip install -U pip

# Install requirements
pip install -r requirements.txt

How to use

Currently, we provide two ways for you to generate comics.

Use the jupyter notebook

You can open the Comic_Generation.ipynb and run the code.

Start a local gradio demo

Run the following command:

python gradio_app_sdxl.py

Disclaimer

This project strives to impact the domain of AI-driven image and video generation positively. Users are granted the freedom to create images and videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.

BibTeX

If you find StoryDiffusion useful for your research and applications, please cite using this BibTeX:

@article{Zhou2024storydiffusion,
  title={StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation},
  author={Zhou, Yupeng and Zhou, Daquan and Cheng, Ming-Ming and Feng, Jiashi and Hou, Qibin},
  year={2024}
}

About

Create Magic Story!

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Jupyter Notebook 99.3%
  • Python 0.7%