Church

Living room

Restaurant

256x256 images synthesized by a Progressive GAN [Karras, et al 2017]

 Church

To render a beautiful scene,

What does a GAN need to know?
Bedroom

What causes the mistakes?

GAN Dissection: Visualizing and Understanding
Generative Adversarial Networks
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou
Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba
Which units correlate to an object class?
 Which units correlate to an object class?
 :the current layer
: the first half
: the second half
Which units correlate to an object class?

generated image

generate
h r f

z
generator
Which units correlate to an object class?
featuremap

single unit u

generated image

generate
h r f

z
generator
Which units correlate to an object class?
featuremap

single unit u

generated image segmentation

generate
h r f
segment
z
generator
Which units correlate to an object class?
featuremap thresholded

single unit u upsample

generated image segmentation

generate
h r f
segment
z
generator
Which units correlate to an object class?
featuremap thresholded

single unit u upsample

agreement
IoU
generated image segmentation

generate
h r f
segment
z
generator
 Which units correlate to an object class?
Church samples

Unit #119
Tree

Unit #32
Dome
 Which units correlate to an object class?
Dining room samples

Unit #139
Window

Unit #65
Table
Which units cause an object class?
 Which units cause an object class?

unforced units causal units

z
 Which units cause an object class?

unforced units causal units

h
ablated image
z
f

force units off

 Which units cause an object class?

unforced units causal units

h
ablated image segmentation
z
f

segment
force units off
 Which units cause an object class?

force units on inserted image

f f o r c e U o ff

unforced units causal units

h
ablated image segmentation
z
f

segment
force units off
 Which units cause an object class?

force units on inserted image segmentation

f f o r c e U o ff

unforced units causal units

segment

h
ablated image segmentation
z
f

segment
force units off
 Which units cause an object class?

force units on inserted image segmentation

f f o r c e U o ff

unforced units causal units

segment

h causal effect
ablated image segmentation
z
f

segment
force units off
 Removing or Adding Units

0 5 10 20
Number of tree units ablated
 Removing or Adding Units

0 5 10 20
Number of tree units ablated
 Removing or Adding Units

0 5 10 20
Number of tree units ablated
 GAN Dissection: Comparing Datasets
Units in scene generator Unit class distribution
table #96 iou=0.30 person-b #91 iou=0.21 seat #83 iou=0.13
dining room conference rm

3 2

1 2 o bj e ct s

2 7 p a rt s

4 m at e ri a l s
u n its
1 6

chandelier-l #184 iou=0.21 chair-l #456 iou=0.19 table #89 iou=0.31

2 0

1 0 o b j e ct s

2 9 pa rt s

2 m a t e ri a l s
un i t s
1 0

stove-t #312 iou=0.12 chair-b #166 iou=0.11 cabinet-b #70 iou=0.15

2 7

6 ob j e ct s

3 4 p art s

2 m a t e ri a l s
kitchen

un i t s
1 4

tree #157 iou=0.32 grass #14 iou=0.25 dome #43 iou=0.07

church/outdoor

2 4

7 o b j e ct s

13 p a rt s

2 m at e r i a l s
u n it s
1 2

Top objects: ceiling, wall, person, table…

Scene: conference room
 GAN Dissection: Comparing Datasets
Units in scene generator Unit class distribution
table #96 iou=0.30 person-b #91 iou=0.21 seat #83 iou=0.13
conference rm

3 2

1 2 o b je c ts

2 7 p a r ts

4 m a te r ia ls
u n its
1 6

chandelier-l #184 iou=0.21 chair-l #456 iou=0.19 table #89 iou=0.31

2 0
dining room

1 0 o b je c ts

2 9 p a r ts

2 m a te r ia ls
u n its
1 0

stove-t #312 iou=0.12 chair-b #166 iou=0.11 cabinet-b #70 iou=0.15

2 7

6 o b je c ts

3 4 p a r ts

2 m a te r ia ls
kitchen

u n its
1 4

tree #157 iou=0.32 grass #14 iou=0.25 dome #43 iou=0.07

church/outdoor

2 4

7 o b je c ts

1 3 p a r ts

2 m a te r ia ls
u n its
1 2

Top objects: ceiling, window, chair, table…

Scene: dining room
 church/outdoor kitchen dining room conference rm

table #96

tree #157
stove-t #312

iou=0.32 grass #14

chandelier-l #184 iou=0.21 chair-l #456

iou=0.12 chair-b #166

iou=0.30 person-b #91
Units in scene generator
iou=0.21 seat #83

iou=0.19 table #89

iou=0.25 dome #43

iou=0.11 cabinet-b #70

iou=0.07
iou=0.15
iou=0.31
iou=0.13

u n it s u n it s u n it s u n it s
1
12
24
1
14
27
1
10
20
1
16
32

ce
ce

ce

gr il il
w

as ilin
in

in ing g
g

fl o d
w

tre s p e wa
e w in or ch rs ll
sk o r p e d ow a o
t
ir

do y k rs
su o ta n
o le bl
r

rfa n se e
ow ab s

w r oa ch pain k
in
y

ta c e at
d d
w

an tin
w w w ab l e d g o
i

bu 6 o b je c ts pa d
nd
o se
ll -

ild t cu elie
p

in 7 o b je c ts c ow rt r m t
fl o i n o i n i

-
n
r w g

sk g-t ca bu ain si on
y g ff lv i
c n p et e c to r
r

sk -b
t s w s ha
ei bi ab

cl y-r fl o l a te
o
lin e in

ce 1 0 o b je c ts iv cre ir
e

ou t r-b el e
d
b b -t -b -t

ilin t ch n
g
- h le t w air
c

w floo -b 1 2 o b je c ts
i

y in r- ce e all
d b i
y -

-t
- r-b

t l

e ce ow c ling
ab c a ei

bu - b ilin -t li -r
li in wa

sk sk tr e tre
e
g- ll-

ild - c h ng - ta
t
b b

flo g-l
in
g ki or a t bl
e-
ou d o m - l tc c - ch ir-r
a
a

he b t c e to r s t
nt e n w ir-t
w in

al ilin o
is
al et

gr a i n
l-t -

l-b
as - l la r
n w se g-t
i
c

e i a t-
flo nd

se s-t w flood-t lin t

a- in r
d t t g
-r -t

st

ob 1 3 p a r ts ce ow-r ce abl c e-l

il

w e-
w

o ili -r in l
or ow in

o gr
r
l g

k stong- -
- r

ne od

flo -b
s

2 m a t e r ia ls ur v e r pe
fa -t ch or-
l s r-b
ab eil floo r o

c
n-

st c - ei air- se
b

ov lin l
t

a b g- c h a t-
t r

c le l ai b
a
t

ll- a -b r-
w r
Unit class distribution

w bin

in t le p e wa b
e e- a e -

do l l rs ll-l
b wa

w p a w l-
on
- lt

w -l m tab -t
le
o

ki ch all in al
-
n

tc t n l-l sw
h ai -l
g

r i v ta r
e
b

n flo -b w fl oo - el t
ito ble

w is or in r
d -
o
r - -b

la -l p a c ha
r

n
in

c h d ow t si in ir-
w k s c d-l a o -l lv tin b
c

or u g
r

k fa ir h d w- er
w na

t
ha c

su n el i e b sc floo-b
d

r fa e r- re r-
- -b

l e l
w t ce
s

or s abl -l ch iw ilv fl o n - b
e

k to e - or
r -r
h

an rs
lie wa do

re su ve t d c
r-b ll- w-

fri rfa -b
r

ge c pa e l i
m
ai een
c r c

r e- i n er -
ic
r-r r

owato r ti r ha -
ca ng- ir -
a r-t rp l 2 9 p a rts l 2 7 p a rts
sk

ve

ca
in

gl - 3 4 p a r ts g l et
as as pa rpe
s
l s

2 m a t e r ia ls in t
t
GAN Dissection: Comparing Datasets

tl
e 2 m a te r ia ls fa e d
br
ic
4 m a te r ia ls
GAN Dissection: Comparing Models

SWD (Sliced Wasserstein Distance) [Karras, et al 2017]: the lower, the better
GAN Dissection: Comparing Models

SWD (Sliced Wasserstein Distance) [Karras, et al 2017]: the lower, the better
GAN Dissection: Comparing Models

SWD (Sliced Wasserstein Distance) [Karras, et al 2017]: the lower, the better
GAN Dissection: Comparing Layers

Hard to find object concepts

GAN Dissection: Comparing Layers

Objects and object parts

GAN Dissection: Comparing Layers

Texture, color, edges

 Debugging and Improving GANs
Unit #63

Bedroom images with artifacts

Unit #231

Example artifact-causing units Ablating “artifact” units improves results

 Object-Scene Relationship

ablate person units ablate curtain units

ablate window units ablate table units ablate chair units

 Object-Scene Relationship

conference room church

kitchen living room bedroom

 Object-Scene Relationship

Yellow bounding box: highlight every location where we can insert doors.
Paint with GANs
Paint with GANs
Paint with GANs
 Online Demo
http://bit.ly/ganpaint