Project Review

On
“Style Transfer and Image Compositing on a Painting”
By
407115 Kaustubha Kuldip Rao

407116 Rishikesh Rameshwar Jadhav

407070 Ayush Jain

Guide
“ Prof. S. P. Potdar”
Thursday, 12th December 2019
 CONTENTS
■ Project Details
■ Problem Statement
■ Objectives
■ Scope
■ Introduction
■ VGG16 Architecture
■ Robust Coarse Harmonisation
■ Losses
■ High Quality Refinement
■ UML Diagrams
 PROJECT DETAILS
● Project Title: “Style Transfer and Image composting on a painting”
● Project Domain: Image Processing
● Project Group Members:
○ 71707531B, Kaustubha Kuldip Rao
○ 71707550J, Rishikesh Rameshwar Jadhav
○ 71706706J, Ayush Jain
 PROBLEM STATEMENT
To build a neural network based algorithm that can transfer relevant
statistics of neural responses from the painting to an object which is being
copied into the painting and harmonize the appearance of the pasted
object so that it matches that of the painting.
 OBJECTIVES
To cope with the diversity of abstraction levels and types of paintings, we introduce
a technique to adjust the parameters of the transfer depending on the painting. We ensure
both spatial and inter-scale statistical consistency and demonstrate that both aspects are
key to generating quality results.
1. Inter-scale Consistency:
Preserving the correlation of the neural responses between the layers of the
network also improves the output quality.
1. Spatial Consistency:
Transferring statistics that come from a small set of regions in the painting yields
better results than using many isolated locations.
 SCOPE

● Implementing VGG16, one of the popular CNN methods would result into more
accurate results.
● Using two pass algorithm to maintain a balance between style and content.
● Pass one focuses on higher layers of the CNN while pass two focuses on upper layers
for styling.
● To implement this project on Torch Framework and CUDA processor.
 INTRODUCTION
Our proposed algorithm proceeds in three steps:

1. The input image I and style S are processed with the VGG network to produce a set
of activation values as feature representations. Intuitively, these capture the statistics
that represent the style of each image.
2. The style activations are mapped to the input ones. In this step we use a two pass
algorithm which proceeds as:
a. Robust Coarse Harmonization
b. High-Quality Refinement
3. The output image O is reconstructed that seeks to preserve the content of the input
image while at the same time match the visual appearance of the style image. These
objectives are modeled using losses.
 LITERATURE SURVEY
■ SUNKAVALLI K., JOHNSON M. K., MATUSIK W., PFISTER H.: Multi-scale image
harmonization. In ACM Trans. Graph. (2010), vol. 29, ACM, p. 125.
- Advantage:
Smooth histogram and noise matching which handles fine texture on top of color
- Disadvantage:
Does not capture more structured textures like brush strokes which often appear in
paintings.
■ Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram
Losses
- Advantage:
Can be improved in stability and quality by imposing histogram losses, which better
constrain the dispersion of the texture statistics.
- Disadvantage:
- Focus mostly on the style pattern rather than content.
 LITERATURE SURVEY
■ GATYS L. A., ECKER A. S., BETHGE M.: Image style transfer using convolutional
neural networks. In The IEEE Conference on Computer Vision and Pattern
Recognition(CVPR) (June 2016).
This paper we process the input image and the style image using the VGG16, one of the
popular CNN Architecture to produce a set of feature representation or activation values which
can be used to map the input photo with the painting. The idea of the paper is to minimise the
loss by calculating the appropriate value of style image and input image factors which leads to
a perfect desired output.
Disadvantages:
● Focuses more on content.
● Texture is lost on pasted object.
VGG16 ARCHITECTURE
 ROBUST COARSE HARMONIZATION
● Mask all other parts of image except the object.
● Nearest neighbour and reconstruction losses.
● Independent Mapping

First Pass
 LOSSES
● Total Variation Loss:

Sums the difference between adjacent pixel values which will ensure
that the result is soother.

● Histogram Loss:

Mean-squared error between the output features and their remapped version.
 HIGH-QUALITY REFINEMENT
● Consistent Mapping.
● Final Reconstruction loss considering histogram and total-variation losses.

Second Pass
 SOFTWARE AND HARDWARE ANALYSIS
● Software Requirements :
○ Ubuntu 16.04 OS
○ Torch Framework

● Hardware Requirements :
○ Nvidia Quadro 4000 CUDA Processor
USE CASE DIAGRAM
CLASS DIAGRAM
SEQUENCE DIAGRAM
ACTIVITY DIAGRAM
STATE DIAGRAM
 REFERENCES
● GATYS L. A., ECKER A. S., BETHGE M.: Image style transfer using
convolutional neural networks. In The IEEE Conference on Computer Vision and
Pattern Recognition(CVPR)(June 2016).
● WILMOT P., RISSER E., BARNES C.: Stable and Controllable Neural Texture
Synthesis and Style Transfer Using Histogram Losses. (Feb 2017).
● SUNKAVALLI K., JOHNSON M. K., MATUSIK W., PFISTER H.: Multi-scale
image harmonization. In ACM Trans. Graph. (2010), vol. 29, ACM, p. 125.
 CONCLUSION
We have described an algorithm to copy an object in a photograph and paste it to a
painting seamlessly,i.e.,the composite still looks like a genuine painting. We have
introduced a two-pass algorithm that first transfers the overall style of the painting to the
input and then refines the result to accurately match the painting’s color and texture. This
latter pass relies on mapping neural response statistics that ensures consistency across the
network layers and in image space.