Awesome Autoregressive Models in Vision

If you like our project, please give us a star ⭐ on GitHub for the latest update.

Autoregressive models have shown significant progress in generating high-quality content by modeling the dependencies sequentially. This repo is a curated list of papers about the latest advancements in autoregressive models in vision. This repo is being actively updated, please stay tuned!

📣 Update News

[2024-10-13] We released the repository.

⚡ Contributing

We welcome feedback, suggestions, and contributions that can help improve this survey and repository so as to make them valuable resources to benefit the entire community. We will actively maintain this repository by incorporating new research as it emerges. If you have any suggestions regarding our taxonomy, find any missed papers, or update any preprint arXiv paper that has been accepted to some venue.

If you want to add your work or model to this list, please do not hesitate to pull requests. Markdown format:

* [**Name of Conference or Journal + Year**] Paper Name. [[paper]](link) [[code]](link)

📖 Table of Contents

• 📣 Update News
• ⚡ Contributing
• 📖 Table of Contents
  • Unconditional/Class-Conditioned Image Generation
  • Text-to-Image Generation
  • Image-to-Image Translation
  • Image Editing
  • Medical Tasks in Vision
  • Motion Generation
  • Video Generation
  • Multi-Modal Tasks
  • Other Generation (Motion, Robot, Scene, 3D， Outpainting)
  • Accelerating & Stability
  • Evaluation Metrics
• Star History
• ♥️ Contributors
• 👍 Acknowledgement
• 📑 Citation

Unconditional/Class-Conditioned Image Generation

• Pixel-wise Generation

  • [ICML, 2020] ImageGPT: Generative Pretraining from Pixels Paper
  • [ICML, 2018] Image Transformer Paper Code
  • [ICML, 2018] PixelSNAIL: An Improved Autoregressive Generative Model paper Code
  • [ICML, 2017] Parallel Multiscale Autoregressive Density Estimation Paper
  • [ICLR workshop, 2017] Gated PixelCNN: Generating Interpretable Images with Controllable Structure Paper
  • [ICLR, 2017] PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications Paper Code
  • [NeurIPS, 2016] PixelCNN Conditional Image Generation with PixelCNN Decoders Paper Code
  • [ICML,2016] PixelRNN Pixel Recurrent Neural Networks Paper Code

• Token-wise Generation

  • [Arxiv, 2024.09] Emu3: Next-Token Prediction is All You NeedPaper Name. Paper Code Project
  • [Arxiv, 2024.09] Open-MAGVIT2: Democratizing Autoregressive Visual Generation Paper Code
  • [Arxiv, 2024.06] OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation Paper Code
  • [Arxiv, 2024.06] Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99% Paper Code
  • [Arxiv, 2024.06] Titok An Image is Worth 32 Tokens for Reconstruction and Generation Paper Code
  • [Arxiv, 2024.06] Wavelets Are All You Need for Autoregressive Image Generation*[Tokenizer]* Paper
  • [Arxiv, 2024.06] LlamaGen Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation Paper Code
  • [ICLR, 2024] MAGVIT-v2 Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation Paper
  • [ICLR, 2024] FSQ Finite scalar quantization: Vq-vae made simple Paper Code
  • [ICCV, 2023] Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformers Paper
  • [CVPR, 2023] Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization Paper Code
  • [CVPR, 2023, Highlight] MAGVIT: Masked Generative Video Transformer Paper
  • [AAAI, 2023] Exploring Stochastic Autoregressive Image Modeling for Visual Representation Paper Code
  • [NeurIPS, 2023] MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation Paper
  • [BMVC, 2022] Unconditional image-text pair generation with multimodal cross quantizer Paper Code
  • [ICLR, 2022] ViT-VQGAN Vector-quantized Image Modeling with Improved VQGAN Paper
  • [PMLR, 2021] Generating images with sparse representations*[Tokenizer]* Paper
  • [CVPR, 2021] VQGAN Taming Transformers for High-Resolution Image Synthesis Paper Code
  • [NeurIPS, 2019] Generating Diverse High-Fidelity Images with VQ-VAE-2 Paper Code
  • [NeurIPS, 2017] VQ-VAE Neural Discrete Representation LearningPaper
  • [Arxiv, 2024.10] ImageFolder: Autoregressive Image Generation with Folded Tokens Paper Code
  • [Arxiv, 2024.08] Scalable Autoregressive Image Generation with Mamba Paper Code
  • [Arxiv, 2024.06] Autoregressive Pretraining with Mamba in Vision Paper Code
  • [Arxiv, 2024.06] Autoregressive Image Generation without Vector Quantization Paper Code
  • [Arxiv, 2024.06] LlamaGen Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation Paper Code
  • [ICML, 2024] DARL: Denoising Autoregressive Representation Learning Paper
  • [ICML, 2024] DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents Paper Code
  • [ICML, 2024] DeLVM: Data-efficient Large Vision Models through Sequential Autoregression Paper Code
  • [CVPR, 2024] Beyond Text: Frozen Large Language Models in Visual Signal Comprehension Paper Code
  • [ICLR, 2024] Finite Scalar Quantization: VQ-VAE Made Simple Paper Code
  • [NeurIPS, 2021] ImageBART: Context with Multinomial Diffusion for Autoregressive Image Synthesis Paper Code
  • [CVPR, 2021] VQGAN Taming Transformers for High-Resolution Image Synthesis Paper Code
  • [ECCV, 2020] RAL: Incorporating Reinforced Adversarial Learning in Autoregressive Image Generation Paper
  • [NeurIPS, 2019] Generating Diverse High-Fidelity Images with VQ-VAE-2 Paper Code
  • [NeurIPS, 2017] VQ-VAE Neural Discrete Representation LearningPaper

• Scale-wise Generation

  • [Arxiv, 2024.10] Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling Paper Code
  • [CVPR, 2022] RQ-Transformer Autoregressive Image Generation Using Residual Quantization Paper Code
  • [Arxiv, 2024.04] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction Paper Code

Text-to-Image Generation

• [Arxiv, 2024.10] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens Paper
• [Arxiv, 2024.10] HART: Efficient Visual Generation with Hybrid Autoregressive Transformer Paper Code
• [Arxiv, 2024.10] DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation Paper Code
• [Arxiv, 2024.10] DnD-Transformer: A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Fine-grained Image Generation Paper Code
• [Arxiv, 2024.08] VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling Paper Code
• [Arxiv, 2024.08] Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining Paper Code
• [Arxiv, 2024.07] MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis Paper Code
• [Arxiv, 2024.06] LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation Paper Code
• [Arxiv, 2024.06] STAR: Scale-wise Text-to-image generation via Auto-Regressive representations Paper Code
• [Arxiv, 2024.05] Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling Paper
• [TOG, 2023] IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformers (svg image) Paper Code
• [NeurIPS, 2023] LQAE Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment Paper Code
• [TMLR, 2022.06] Parti: Scaling Autoregressive Models for Content-Rich Text-to-Image Generation Paper Code
• [NeurIPS, 2022] CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers Paper Code
• [ECCV, 2022] Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors Paper
• [CVPR, 2022] VQ-Diffusion: Vector Quantized Diffusion Model for Text-to-Image Synthesis Paper Code
• [CVPR, 2022] Make-A-Story: Visual Memory Conditioned Consistent Story Generation (storytelling) Paper
• [NeurIPS, 2021] CogView: Mastering Text-to-Image Generation via Transformers Paper Code
• [Arxiv, 2021.02] DALL-E 1: Zero-Shot Text-to-Image Generation Paper

Image-to-Image Translation

• [Arxiv, 2024,06] CAR: Controllable Autoregressive Modeling for Visual Generation Paper Code
• [Arxiv, 2024,06] ControlAR: Controllable Image Generation with Autoregressive Models Paper Code
• [Arxiv, 2024,06] ControlVAR: Exploring Controllable Visual Autoregressive Modeling Paper Code
• [ICML Workshop, 2024] MIS Many-to-many Image Generation with Auto-regressive Diffusion Models Paper
• [Arxiv, 2024.03] SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model Paper Project

Image Editing

• [ECCV, 2022] VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance Paper
• [ECCV, 2022] Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors Paper
• [NeurIPS, 2021] ImageBART: Context with Multinomial Diffusion for Autoregressive Image Synthesis Paper Code
• [ECCV, 2022] QueryOTR: Outpainting by Queries Paper Code

Medical Tasks in Vision

• [arxiv, 2024] Autoregressive Sequence Modeling for 3D Medical Image Representation paper
• [arxiv, 2024] Medical Vision Generalist: Unifying Medical Imaging Tasks in Context paper code
• [MIDL, 2024] Conditional Generation of 3D Brain Tumor ROIs via VQGAN and Temporal-Agnostic Masked Transformer paper
• [NMI, 2024] Realistic morphology-preserving generative modelling of the brain paper code
• [Arxiv, 2023] Generating 3D Brain Tumor Regions in MRI using Vector-Quantization Generative Adversarial Networks paper medical image-to-image translation
• [ICCV, 2023] Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers paper Code
• [MICCAI, 2022] Morphology-preserving Autoregressive 3D Generative Modelling of the Brain paper code medical image generation
• [ICIP, 2021] Generating Annotated High-Fidelity Images Containing Multiple Coherent Objects paper Code

Motion Generation

• [AAAI, 2024] AMD: Autoregressive Motion Diffusion Paper Code
• [CVPR, 2023] T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations Paper
• [Arxiv, 2022] HiT-DVAE: Human Motion Generation via Hierarchical Transformer Dynamical VAE Paper
• [ICCV, 2021 oral] HuMoR: 3D Human Motion Model for Robust Pose Estimation Paper Code

Video Generation

• Unconditional Video Generation

  • [ICML, 2017] Video Pixel Networks Paper
  • [CVPR, 2018] MoCoGAN: Decomposing Motion and Content for Video Generation paper Code
  • [ICLR, 2020] Scaling Autoregressive Video Models paper
  • [Arxiv, 2020.06] Latent Video Transformer paper Code
  • [Arxiv, 2021.04] VideoGPT: Video generation using VQ-VAE and transformers paper
  • [ECCV, 2022] Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer Paper Code
  • [CVPR, 2023] PVDM Video Probabilistic Diffusion Models in Projected Latent Space Paper
  • [ECCV 2024] ST-LLM: Large Language Models Are Effective Temporal Learners Paper Code
  • [ICLR, 2024] MAGVIT-v2 Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation Paper

• Conditional Video Generation

  • Text-to-Video Generation

    • [IJCAI, 2021] IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation. paper
    • [Arxiv, 2022.05] GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions paper
    • [Arxiv, 2022.05] CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers paper Code
    • [ECCV, 2022] NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion paper Code
    • [NeurIPS, 2022] NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis paper Code
    • [CVPR, 2024] ART-V: Auto-Regressive Text-to-Video Generation with Diffusion Models paper
    • [Arxiv, 2024.02] LWM World Model on Million-Length Video And Language With Blockwise RingAttention Paper Code
    • [Arxiv, 2024.06] ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Models paper Code
    • [Arxiv, 2024.06] iVideoGPT: Interactive VideoGPTs are Scalable World Models Paper Code
    • [Arxiv, 2024.06] Pandora: Towards General World Model with Natural Language Actions and Video States Paper Code
    • [Arxiv, 2024.10] Loong: Generating Minute-level Long Videos with Autoregressive Language Models Paper
    • [Arxiv, 2024.10] Pyramid Flow: Pyramidal Flow Matching for Efficient Video Generative Modeling Paper Code
    • [Arxiv, 2024.10] Progressive Autoregressive Video Diffusion Models Paper Code

  • Visual Conditional Video Generation

    • [NeurIPS, 2015] Convolutional LSTM network: A machine learning approach for precipitation nowcasting paper
    • [NeurIPS, 2017] Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms paper
    • [ICLR, 2019] Eidetic 3d lstm: A model for video prediction and beyond paper
    • [ICLR, 2018] Stochastic variational video prediction paper
    • [CVPR, 2021] Stochastic image-to-video synthesis using cinns paper Code
    • [Arxiv, 2021.03] Predicting Video with VQVAE paper
    • [ICIP, 2022] HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator paper
    • [CVPR, 2024] LVM Sequential Modeling Enables Scalable Learning for Large Vision Models Paper Code

  • Multimodal Conditional Video Generation

    • [CVPR, 2022] Make it move: controllable image-to-video generation with text descriptions paper Code
    • [CVPR, 2023] MAGVIT: Masked Generative Video Transformer paper
    • [ICML, 2024] VideoPoet: A Large Language Model for Zero-Shot Video Generation paper
    • [ICML, 2024] Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization paper code

Multi-Modal Tasks

• [Arxiv, 2024.10] Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation paper Code
• [Arxiv, 2024.10] MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling paper Code
• [Arxiv, 2024.10] vACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correction paper Code
• [Arxiv, 2024.08] Show-o: One Single Transformer to Unify Multimodal Understanding and Generation paper Code
• [Arxiv, 2024.08] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model paper Code
• [Arxiv, 2024.07] SEED-Story: Multimodal Long Story Generation with Large Language Model Paper Code
• [Arxiv, 2024.05] Chameleon: Mixed-Modal Early-Fusion Foundation Models paper Code
• [Arxiv, 2024.04] SEED-X Multimodal Models with UnifiedMulti-granularity Comprehension and Generation Paper Code
• [ICML, 2024] Libra: Building Decoupled Vision System on Large Language Models paper Code
• [CVPR, 2024] Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Actionpaper Code
• [CVPR, 2024] Anole: An Open, Autoregressive and Native Multimodal Models for Interleaved Image-Text Generation paper Code
• [Arxiv, 2023.11] InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation paper Code
• [ICLR, 2024] Kosmos-G: Generating Images in Context with Multimodal Large Language Models Paper Code
• [ICLR, 2024] LaVIT: Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization paper code
• [ICLR, 2024] SEED-LLaMA Making LLaMA SEE and Draw with SEED Tokenizer Paper Code
• [ICLR, 2024] EMU Generative Pretraining in Multimodality Paper Code
• [Arxiv, 2023.09] CM3Leon: Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning Paper Code
• [Arxiv, 2023.07] SEED Planting a SEED of Vision in Large Language Model Paper Code
• [NeurIPS, 2023] SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMspaper
• [ICLR, 2023] Unified-IO: A unified model for vision, language, and multi-modal tasks paper Code
• [ICML, 2023] Grounding Language Models to Images for Multimodal Inputs and Outputs paper Code
• [NeurIPS, 2022] Flamingo: a Visual Language Model for Few-Shot Learning Paper
• [Arxiv, 2021.12] ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation Paper
• [KDD, 2021] M6: A Chinese Multimodal Pretrainer Paper

Other Generation (Motion, Robot, Scene, 3D， Outpainting)

• [Arxiv, 2024.10] DART: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control Paper
• [Arxiv, 2024.10] Autoregressive Action Sequence Learning for Robotic Manipulation Paper Code
• [Arxiv, 2024.07] Video In-context Learning Paper
• [CVPR, 2024] Sequential Modeling Enables Scalable Learning for Large Vision Models Paper Code
• [AAAI, 2024] Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation Paper Code
• [arxiv, 2024] LM4LV: A Frozen Large Language Model for Low-level Vision Tasks paper code
• [CVPR, 2023] Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings Paper
• [ECCV, 2022] Autoregressive 3D Shape Generation via Canonical Mapping Paper
• [NeurIPS, 2022] Visual Prompting via Image Inpainting Paper Code
• [EMNLP, 2022] MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning Paper
• [NeurIPS, 2021] Multimodal Few-Shot Learning with Frozen Language Models Paper
• [ECCV, 2020] Autoregressive Unsupervised Image Segmentation paper

Accelerating & Stability

• [NeurIPS, 2024] Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective Paper Code
• [Arxiv, 2024.10] Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding Paper
• [Arxiv, 2024.09] Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation Paper
• [ECCV, 2024] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models Paper Code

Evaluation Metrics

Metric Name	Quantitative or Qualitative	Higher is better↑ / Lower is better↓	Paper proposing the metric
Inception Score (IS)	Quantitative	↑	Salimans et al., 2016
Fréchet Inception Distance (FID)	Quantitative	↓	Heusel et al., 2017
Kernel Inception Distance (KID)	Quantitative	↓	Binkowski et al., 2018
Precision and Recall	Quantitative	Depends on metric	Powers, 2020
CLIP Maximum Mean Discrepancy	Quantitative	↓	Jayasumana et al., 2023
CLIP Score	Quantitative	↑	Hessel et al., 2021
R-precision	Quantitative	↑	Craswell et al., 2009
Perceptual Path Length	Quantitative	↓	Karras et al., 2019
Fréchet Video Distance (FVD)	Quantitative	↓	Unterthiner et al., 2019
Aesthetic (Expert Evaluation)	Qualitative	↑	Based on domain expertise
Turing Test	Qualitative	Pass/fail	Turing, 1950
User Studies (ratings, satisfaction)	Qualitative	↑	Various, depending on the user study methodology

Star History

♥️ Contributors