Image Recoloring Based on Object Color Distributions

Mahmoud Afifi1, Brian Price2, Scott Cohen2, and Michael S. Brown1

1York University, 2Adobe Research


Figure


(A) An input image and its semantic segmentation (object mask) obtained by RefineNet. (B) Recolored images produced by Photoshop's variation tool. (C) Recolored images from our method that considers the color distribution of objects in the image.


Abstract

We present a method to perform automatic image recoloring based on the distribution of colors associated with objects present in an image. For example, when recoloring an image containing a sky object, our method incorporates the observation that objects of class 'sky' have a color distribution with three dominant modes for blue (daytime), yellow/red (dusk/dawn), and dark (nighttime). Our work leverages recent deep-learning methods that can perform reasonably accurate object-level segmentation. By using the images in datasets used to train deep-learning object segmentation methods, we are able to model the color distribution of each object class in the dataset. Given a new input image and its associated semantic segmentation (i.e., object mask), we perform color transfer to map the input image color histogram to a set of target color histograms that were constructed based on the learned color distribution of the objects in the image. We show that our framework is able to produce compelling color variations that are often more interesting and unique than results produced by existing methods.



Figure


Our image recoloring framework. We use a deep-learning method to obtain the input image's semantic segmentation as an object mask. We generate a color distribution for each object as a color palette and select one object in the image as the primary object. Next, we visit each cluster in the training images of the primary object class. For each cluster, we select an object instance with the most dissimilar colors to our input object's colors and add its colors to our target histogram. Within this same cluster, we search for the other (non-primary) objects found in the input image and use the most dissimilar instance in the target histogram. Lastly, color transfer is applied to map the input image to our generated target histogram. This is repeated for each cluster, producing several variations on the input image.


Files
Paper Supplementary Materials Presentation Source Code
Paper Supplementary Materials PPSX | PPTX Code


BibTeX
@inproceedings{Afifi2019ImageRecoloring,
booktitle = {Eurographics 2019 - Short Papers},
title = {Image Recoloring Based on Object Color Distributions},
author = {Afifi, Mahmoud and Price, Brian and Cohen, Scott and Brown, Michael S.},
year = {2019},
publisher = {The Eurographics Association},
ISSN = {1017-4656},
DOI = {10.2312/egs.20191008}
}


Results:
Input image Recolored images
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Image Image Image Image
Download input/recolored images in high resolution from here.


This page contains files that could be protected by copyright. They are provided here for reasonable academic fair use.
Copyright © 2019