Auto-Illustrating Poems and Songs with Style

Automatically generated illustrations of the first text lines (bottom) of the song "The Mamas The Papas - CALIFORNIA DREAMIN" in three different styles (left).

Katharina Schwarz (University of Tübingen)

Tamara L. Berg (University of North Carolina)

Hendrik P. A. Lensch (University of Tübingen)

Asian Conference on Computer Vision (ACCV) 2016

We develop an optimization based framework to automatically illustrate poems and songs. Our method is able to produce both semantically relevant and visually coherent illustrations, all while matching a particular user selected visual style. We demonstrate our method on a selection of 200 popular poems and songs collected from the internet and operate on around 14M Flickr images. A user study evaluates variations on our optimization procedure. Finally, we present two applications, identifying textual style, and automatic music video generation.

Paper

Supplemental material

Bibtex