Modified Slides

Analysis of Different Frameworks for Deep
Neural Network and its Implementation
Abdur Rahman, Mohamed Ismail

University of Erlangen-Nürnberg
Institute for Digital Communications
February 1,2019
Outline
1 Analysis of Frameworks
Motivation
Comparison of Frameworks
Analysis Based on time
2 Implementation Task
Implementation of a BPSK Detector
1 / 49 Abdur Rahman, Mohamed Ismail –

Analysis of Different Frameworks for Deep Neural Network and its Implementation
1. Analysis of Frameworks Motivation
Goal
Problem
A large number of Deep learning frameworks available
Each stands out with its own purpose
Which suits my application ?
Various studies on performance based on speed and

complexity
Our focus is on the facilities available with each
framework

1. Analysis of Frameworks Comparison of Frameworks
Frameworks

Frameworks
Overview:
TensorFlow
Creator: Google
Platform: Windows, Linux, macOS, Android
Interface Language: Python, C, C++, JavaScript, R,
Julia
Keras
Creator: François Chollet
Platform: Windows, Linux, macOS
Interface Language: Python,R
MATLAB
Creator: Mathworks
Interface Language: MATLAB

Frameworks
Overview:
Caffe
Creator: Berkeley Vision and Learning Center
Interface Language: Python, MATLAB, C++
PyTorch
Creator: Facebook AI Group
Interface Language: Python

Parameters for comparison
The frameworks are compared based on:

Available layers
Available activation functions
User defined layers and functions
GUI availability

Available Layers
Fully connected Network:
Layers: Input,Hidden,Output
Each neuron is connected to all the other neurons in
the next layer

Available Layers
Layer names to create a Fully Connected Network in

different frameworks:
TensorFlow Matlab Keras Caffe Pytorch

Dense fullyConnectedLayer Dense Linear InnerProductLayer

Available Layers
Convolutional Neural Network:
Convolution operation is applied to the input data
Sliding Kernel is applied
Can be followed by a pooling operation

Available Layers
Layer names to create a Convolution Neural Network in


Conv1D convolution2dLayer Conv1D Conv1d
Conv2D Conv2D Convolution Layer Conv2d
Conv3D Conv3D Conv3d
Conv1DTranspose Convtranspose1d
Conv2DTranspose transposedConv2dLayer Conv2DTranspose Deconvolution Layer Convtranspose2d
Conv3DTranspose Conv3DTranspose Convtranspose3d
SeparableConv1D SeparableConv1D
SeparableConv2D SeparableConv2D InnerProductLayer

Available Layers
Convolution 2D:

Available Layers
Redefining Convolution:
Figure: Kernel

Available Layers

Available Layers
Deconvolution 2D:

Available Layers
Depthwise Seperable Convolution 2D:

To reduce the computational cost of convolution
Depthwise convolution
Pointwise convolution

Available Layers
Recurrent Neural Network:

Motivation:
Traditional Neural Networks don’t capture dependencies
between the input which is highly critical for applications
with sequential data.
Shortcomings:
Short dependencies
Vanishing gradient problem

Available Layers
LSTM
It interacts with subsequent layers through four different
entities, and the key cell state assures the long term
dependency.

Available Layers
Layer names to create a Recurrent Neural Network in


layrecnet RNN RNN RNN
SimpleRNN
lstmLayer LSTM LSTM LSTM
ConvLSTM2D
SimpleRNNCell RNNCell
GRUCell GRUCell
LSTMCell LSTMCell
CuDNNGRU
bilstmLayer

Activation Function
Provide a non linear mapping between the input and
response variable.
Assure universal function approximation
They should be differentiable mathematical function.

Available Activation Function
Common Activation Functions available in different

frameworks: :

Sigmoid Sigmoid Sigmoid Sigmoid
tanh tanh TanH tanh
elu elu elu elu
selu selu elu selu
relu relulayer relu relu relu
Leakyrelulayer leakyrelu leakyrelu
Prelu Prelu Prelu
softplus softplus softplus
softsign softsign softsign
softmax softmax

Available Activation Function
Other Activation Functions :

relu6 clippedrelulayer ThresholdedReLU log Hardshrink
Crelu Power Hardtanh
relu-x Bias LogSigmoid
Scale Rrelu
Threshold softshrink
BNLL tanhshrink
Absolutevalue softmax2d
Scale logsoftmax

Activation Function
Sigmoid:
+ Non linear and differentiable
+ Probabilistic output
- Saturates for x 0 and x 0
- Not zero-centered
Activation Function
Tanh:
+ Non linear and Differentiable
+ Zero Centered
- Vanishing gradient problem

Activation Function
Relu:
+ Avoids vanishing gradient problem
- Not zero centered

Activation Function
Leaky Relu:
+ Fixes dying ReLU
Parametric ReLU: paramater ’a’

Activation Function
Exponential Relu
+ No vanishing gradient problem
+ Can keep zero mean activation

User Defined Layers
Need:
Though most frameworks come with number of
sophisticated predefined layers, there are instances when
application-specific layers either from scratch or as the
composition of existing layers is necessary.
Custom layers can be created in Tensor Flow with

Keras
Matlab has detailed documentation on creating
custom layers
Custom layers can also be created on Caffe and
Pytorch

GUI Availability
Visualization of networks in different frameworks:

Tensorboard nnstart Visdom

MATLAB- GUI

MATLAB- GUI

MATLAB- GUI

Tensorboard - TensorFlow’s GUI

Tensorboard is a suite of web applications for inspecting
and understanding the TensorFlow runs and graphs.
Figure: Visualisation of implemented network


Tensorboard allows comparison of different graphs from
different programs
Figure: Comparison of graphs of different learning rate

Figure: Different visualisations available in Tensorboard

Implications of comparisons
All 5 frameworks has equal facilities for Fully

connected network
TensorFlow, Keras and Pytorch makes working with 1D
Convolution ,3D convolution and Separable
convolution easy
TensorFlow has many modification of Relu activation
function
Creating a new activation function is simple in Matlab
TensorFlow has a more useful and intutive
GUI-Tensorboard

1. Analysis of Frameworks Analysis Based on time
Analysis Based on Time
Model setup:
A Fully connected neural network is considered
MNIST dataset with 6000 training images and 10,000
test images are taken as input
Varying the network “depth” (the number of internal
layers) with a fixed number of neurons in each layer
Varying the network “width” (the number of neurons)
with a fixed number of layers

Figure: Increase the network depth
Figure: Increase the network width

Figure: Results on increasing network depth

Figure: Results on increasing network width

2. Implementation Task Implementation of a BPSK Detector
Implementation of BPSK Detector
Goal
To implement a BPSK detector considering an AWGN
channel
Framework
Framework used: Tensorflow with Keras
Motivation
Computation time
Available libraries
Detailed documentation

Figure: Block Diagram - Implementation

Figure: BPSK Signal Generation

Figure: Noise Generation

Figure: Received Signal

Figure: Core Network

Figure: Training

Figure: Testing Stage

Figure: Testing Stage

Figure: Performance: SNR=5dB


Modified Slides

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Modified Slides

Hochgeladen von

Copyright:

Verfügbare Formate

Analysis of Diﬀerent Frameworks for Deep

Neural Network and its Implementation

Abdur Rahman, Mohamed Ismail

1 / 49 Abdur Rahman, Mohamed Ismail –

Various studies on performance based on speed and

2 / 49 Abdur Rahman, Mohamed Ismail –

3 / 49 Abdur Rahman, Mohamed Ismail –

4 / 49 Abdur Rahman, Mohamed Ismail –

5 / 49 Abdur Rahman, Mohamed Ismail –

Parameters for comparison

The frameworks are compared based on:

6 / 49 Abdur Rahman, Mohamed Ismail –

7 / 49 Abdur Rahman, Mohamed Ismail –

Layer names to create a Fully Connected Network in

TensorFlow Matlab Keras Caﬀe Pytorch

8 / 49 Abdur Rahman, Mohamed Ismail –

9 / 49 Abdur Rahman, Mohamed Ismail –

Layer names to create a Convolution Neural Network in

TensorFlow Matlab Keras Caﬀe Pytorch

10 / 49 Abdur Rahman, Mohamed Ismail –

11 / 49 Abdur Rahman, Mohamed Ismail –

12 / 49 Abdur Rahman, Mohamed Ismail –

13 / 49 Abdur Rahman, Mohamed Ismail –

14 / 49 Abdur Rahman, Mohamed Ismail –

Depthwise Seperable Convolution 2D:

15 / 49 Abdur Rahman, Mohamed Ismail –

Recurrent Neural Network:

16 / 49 Abdur Rahman, Mohamed Ismail –

17 / 49 Abdur Rahman, Mohamed Ismail –

Layer names to create a Recurrent Neural Network in

TensorFlow Matlab Keras Caﬀe Pytorch

18 / 49 Abdur Rahman, Mohamed Ismail –

19 / 49 Abdur Rahman, Mohamed Ismail –

Available Activation Function

Common Activation Functions available in diﬀerent

TensorFlow Matlab Keras Caﬀe Pytorch

20 / 49 Abdur Rahman, Mohamed Ismail –

Available Activation Function

Other Activation Functions :

21 / 49 Abdur Rahman, Mohamed Ismail –

23 / 49 Abdur Rahman, Mohamed Ismail –

24 / 49 Abdur Rahman, Mohamed Ismail –

25 / 49 Abdur Rahman, Mohamed Ismail –

26 / 49 Abdur Rahman, Mohamed Ismail –

User Defined Layers

Custom layers can be created in Tensor Flow with

27 / 49 Abdur Rahman, Mohamed Ismail –

Visualization of networks in diﬀerent frameworks:

TensorFlow Matlab Keras Caﬀe Pytorch

28 / 49 Abdur Rahman, Mohamed Ismail –

29 / 49 Abdur Rahman, Mohamed Ismail –

30 / 49 Abdur Rahman, Mohamed Ismail –

31 / 49 Abdur Rahman, Mohamed Ismail –

Tensorboard - TensorFlow’s GUI

Figure: Visualisation of implemented network

32 / 49 Abdur Rahman, Mohamed Ismail –

Tensorboard - TensorFlow’s GUI

Figure: Comparison of graphs of diﬀerent learning rate

33 / 49 Abdur Rahman, Mohamed Ismail –

Tensorboard - TensorFlow’s GUI

Figure: Diﬀerent visualisations available in Tensorboard

34 / 49 Abdur Rahman, Mohamed Ismail –

All 5 frameworks has equal facilities for Fully

35 / 49 Abdur Rahman, Mohamed Ismail –