ViPE | University of Tübingen

ViPE: Visualise Prettymuch Everything

ViPE is the first language model designed for assisting in text-to-image generation. It translates any arbitrary piece of text into a visualizable prompt. It helps any text-to-image model in figurative or non-lexical language visualizations. Below is a comparison between SDXL with and without ViPE given infinity as a prompt.

Below is another example of using DALLE 2 with and without ViPE for a highly abstract prompt. The image on the left shows the prompts and the generated image. The images on the right, show ViPE's interpretaion of how the initial prompt could be visualized and the genrated images.

How ViPE is Built?

Building ViPE involves three main steps

Data Collection: Scraping all the English lyrics from the Genius platform, preprocessing and noise removal
Synthetic Label Generation: Applying GPT3.5 Turbo to generate visual translation (elaborations) for the lyrics based on human instructions and the context of the songs. Compiling the LyricCanvas dataset comprising of 10M samples.
Training: Obtaining a robust and lightweight model by training GPT2 on the LyricCanvas dataset with causal language modeling objective conditioned on the lyrics

Down Stream Applications

ViPE's robust generalization capabilities offers a wide range of usage including

Music Video Generation: https://github.com/Hazel1994/ViPE-Videos
Long Document Summarization: https://github.com/Hazel1994/summagary
Prompt Generation: https://huggingface.co/spaces/fittar/ViPE

ViPE Paper

ViPE, won the outstanding paper award at the main conference of EMNLP 2023.

Paper: https://aclanthology.org/2023.emnlp-main.333/

List of outstanding papers: https://2023.emnlp.org/program/best_papers/

Privacy settings