Text-to-video generation models are AI systems that create video content from textual descriptions. These models leverage advanced machine learning techniques, such as neural networks, to understand and translate text inputs into corresponding video sequences. These models are gaining popularity because many professionals from various fields are using video generation tools for their tasks or creative content.
Here is a list of 6 open-source video generation models:
VideoGPT: a straightforward architecture for generating natural videos using likelihood-based modeling. It uses VQ-VAE with 3D convolutions and axial self-attention to learn compressed video representations. A GPT-like model autoregressively processes these latents with spatio-temporal encodings. VideoGPT matches GAN models in video quality and produces high-fidelity videos from UCF-101 and TGIF. → Read more
Stable Video Diffusion by Stability AI is a latent video diffusion model for text-to-video and image-to-video generation. It is built with two image-to-video models that generate 14 and 25 frames at customizable rates between 3 and 30 frames per second. This model offers a robust multi-view 3D-prior and can be used to fine-tune a multi-view diffusion model. → Read more
LVDM (Latent Video Diffusion Model) by Tencent AI Lab can be used for high-fidelity long video generation. LVDM is lightweight video diffusion model that uses a low-dimensional 3D latent space. Hierarchical diffusion approach helps to generate videos over a thousand frames long. LVDM addresses errors in long video generation as well. → Read more
Dreamix is an image and video diffusion model that achieves high realism. It uses a diffusion-based method for text-guided motion and appearance editing of general videos. Dreamix merges low-resolution spatio-temporal data from the original video with new high-resolution information. It also supports subject-driven video generation. → Read more
MAV3D by Meta AI is a method for generating 3D dynamic scenes from text descriptions using a 4D dynamic Neural Radiance Field (NeRF). This approach optimizes scene appearance, density and motion by querying a text-to-video diffusion model. The generated videos can be viewed from any angle and integrated into any 3D environment. → Read more
StyleGAN-V is a continuous-time video generator that uses neural representations. Designing continuous motion representations with positional embeddings shows that training on sparse videos, even with just 2 frames per clip, can be effective. Built on StyleGAN2, this model offers high-quality video generation with spatial manipulations. → Read more
Bonus → 4 useful text- and image-to-video generators that you can try for free: