Improving Robustness of the NSLS-II X-ray Scattering
Image Neural Network with Data Augmentation
INTRODUCTION
Figure 1: Example Synchrotron: Figure 2: Diagram of x-ray
National Synchrotron Light scattering process in
Source II (NSLS-II)
Image taken by bnl.gov
(Brookhaven National Lab) [11]
synchrotron.
Diagram from Kevin Yager [6]
Synchrotrons:
- Allow researchers to peer into the nano-scale to
understand the structure of new materials
- By understanding the structure of materials,
scientists can accelerate their research because
they better understand the function of the
material (form equals function)
- How Synchrotrons Works:
- Accelerates electrons into a large circular orbit
- Use magnetic fields to deflect the electrons
and release extremely bright x-ray light
- Light is directed to a chamber that contains a
sample of interest
- X-ray light scatters off molecule
- A detector picks up the scattered light and
produces X-ray scattering images
- Images reveal structural information with
patterns of rings, halos, spots, etc. that arise
when scattered waves interfere.
- Images provide insight into size, orientation,
packing of atoms
Figure 3: Example of material with 6-fold symmetry
structure from an X-ray scattering image (left)
compared to image from an electron microscope
(right). The x-ray scattering image reveals much more
structural detail. Images taken from Kevin Yager [6]
Applications:
- Many different fields depend on X-ray scattering
images to accelerate their research
- Biology:
- Observe molecules undergo conformational
changes (from the environment) [2]
- Design novel drugs (understand how the
molecule of interest fits into biological
systems) [4]
- Chemistry:
- How molecules interact with each other
- How enzymes perform catalysis [3]
- Electronics:
- Design microelectronics [5]
- Material Science:
- Apply changes in pressure or temperature to
see how a molecule reacts (ex: developing a
material harder than diamond)
Issue: Big Data
- Synchrotrons produce 1-4 TB/day
- 60 beam lines (60 concurrent experiments)
- Manual image analysis is complex
- Image streams outpace manual image analysis
Convolutional Neural Networks (CNN):
- CNNs are specifically useful for processing data
that has a “grid-like topology” such as “image
data, which can be thought of as a 2D grid of
pixels” [7]
- CNNs simulate how the brain interprets images
- As an infant learns, he/she trains with
auditory cues to learn how to identify the
world around them
RESEARCH QUESTION
How will Data Augmentation and Instrument-
Specific Noise Injection improve the accuracy of
the Convolutional Neural Network?
PROCEDURE
1. Neural Network
- TensorFlow’s deep CNN (AlexNet) [14] for image labeling [12]
Figure 4: Architecture for AlexNet. Image taken from Hao Gao [15]
- To train the CNN, images are passed through initial convolutional
layers and a subsequent output layer of softmax regression
- Stochastic gradient descent algorithm used to minimize the loss
- Training terminated when the loss was below 0.10.
2. Synthetic Dataset
- Impossible to tag 50,000 X-ray scattering images
- Generate pre-tagged images
Figure 5: Chart that compares the real X-ray scattering images and
synthetic X-ray scattering images to illustrate the synthetic and real
images look very similar.
Image taken from Kevin Yager [6]
3. Noise Injection
Gaussian Noise: image pixels take on values that are Gaussian-
distributed
• Cause: beam line doesn’t align correctly, blurry lenses
• Apply Gaussian distribution to values in image array to
Salt and Pepper Noise: sparsely occurring black and white pixels
• Cause: sharp sudden disturbances in signal
• Apply Poisson distribution to randomly corrupt values in image array
Poisson Counting Statistic : capturing an x-ray scattering image is a
chance observation [10]
• Cause: If a few photons in a beam of light hit a detector every
second, the fluctuations in the number of photons is significant.
• If the pixel has a higher brightness, it has a higher chance to be
varied
• λ = value in image array
• Poisson distribution: right tailed distribution
• Apply Poisson Distribution (right tailed distribution) to model a
chance observation
METHODOLOGY
X-ray Scattering Image CNN:
- Issue:
- Low accuracy
- Neural Network is not robust to noise
- Regularization Technique: Specialized Data Augmentation (DA) and
Noise Injection (NI) [8]
- DA: Increase size of training dataset
- NI: Train neural network with deliberately distorted images to
allow the CNN to view images with greater diversity [9]
- Traditional DA: Linear Transformations (translation, cropping),
rotation, change in saturation or hue, etc.
- Specialized DA: Apply instrument-specific noise
PROCEDURE RESULTS
Figure 10: Example of x-ray scattering image
with 6-fold symmetry. Image taken from Kevin
Yager.
Figure 11. Increase in accuracy for each class shows that data augmentation
affects different classes and different types of noise differently. The neural
network was able to classify certain labels extremely well (Higher Order,
Rings) with a mAP over 80%, and therefore the performance did not increase
much after data augmentation since the initial performance was so high.
There were some attributes such as Diffuse High-q that were initially hard to
classify (40% accuracy) and data augmentation improved the classification of
those by 30%.
Figure 6: Chart that demonstrates the types of noise and the varying level of
noise applied to the X-ray scattering images. Class-conditional data augmentation as proposed by Salamon and Bello [16]
could improve the performance of a neural network for specific classes.
Chart created by the Finalist Class-conditional DA consists of increasing the number of images that the
neural network cannot classify well (ie. symmetric rings) in training may
4. Training/Testing CNN: Data Augmentation increase the overall accuracy (mAP).
Figure created by the Finalist.
• Generate 50k noise-free x-ray scattering images
Train • Train the Neural Network
(Baseline)
• Generate 5k x-ray scattering images
• Apply Noise to Images using Python script
CONCLUSION
Apply
Noise • Evaluate image performance in neural network
- Newly trained CNN is successful in recognizing images
• Generate 50k noise-free images
• Generate 50k images with random amounts of noise with noise
Data
Augment- • Retrain Neural Network with 100k images - Newly trained CNN always outperform the previous
ation • Evaluate and compare the performance of the CNN that was only trained on noise-free images
original 5k images in the newly trained neural - Data augmentation, specifically noise injection, is a
Evaluate network
successful regularization technique for recognizing X-
Figure 7: Figure created by the Finalist
ray scattering images
RESULTS - This discovery advances the novel and groundbreaking
research scientists are performing with synchrotrons
Comparing CNN Before and After Training with because the image labeling process is now automated
Augmented Data and the algorithm can classify images with noise.
0.7
- With higher accuracy, we are closer to solving the big
mAP (Mean Average Precision)
0.6
data issue and taking advantage of a synchrotron’s full
0.5
0.4 potential
No Noise
0.3
0.2 Data
Augmentation
- Future Work:
0.1
0
- Train the CNN with real NSLS-II X-ray scattering
images
- Transfer learning: apply information learned from
synthetic images to classify real image data [13]
- Class Conditional Data Augmentation
Figure 8: Comparison of CNN performance with and without data
augmentation. After data augmentation, the neural network
consistently better classifies images with different types of noise and
different levels of noise. The performance neural network increases REFERENCES
the most on images with Gaussian noise, then Salt and Pepper noise,
the least Poisson Statistics
[1] B. Wang, K. G. Yager, D. Yu, and M. Hoai, "X-ray scattering image classification using deep
Figure created by the Finalist.
learning" 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), 1: 697–704,
2017. [2] B. A. Wallace, “Conformational changes by synchrotron radiation circular dichroism
spectroscopy.” Nature Structural Biology 7, 708–709 (2000), doi:10.1038/78915 [3] J. Hajdu, K.R.
Acharya, D.I. Stuart, P.J. McLaughlin, D. Barford, N.G. Oikonomakos, H. Klein, and L.N. Johnson,
“Catalysis in the crystal: synchrotron radiation studies with glycogen phosphorylase B.” The EMBO
Journal 6 (2): 539-546, 1987. [4] A. C. Anderson, “The process of structure-based drug design.”
Chemistry & Biology 10 (9): 787–97, 2013. [5] J. Jordan-Sweet, “Synchrotron X-ray scattering
techniques for microelectronics-related materials studies” IBM Journal of Research and
Development 44(4): 457 – 476, 2000. [6] K. Yager, “Deep learning for automated synchrotron
experiments”. IEEE 2017 New York Scientific Data Summit, New York.
http://www.esrf.eu/about/synchrotron-science/synchrotron [7] I. Goodfellow, Y. Bengio, and A.
Courville, “Deep learning.” MIT Press, 2016. [8] J. Sietsma and R. J. F. Dow, “Creating artificial
neural networks that generalize.” Neural Networks 4 (1): 67–79, 1991. [9] R. M. Zur, Y. Jiang, L. L.
Pesce, and K. Drukker, “Noise injection for training artificial neural networks: A comparison with
weight decay and early stopping”. Medical Physics, 36(10), 4810–4818, 2009. [10] D. V.
Perepelitsa, “Johnson noise and shot noise”. Dept. of Physics, MIT, 2006.
http://web.mit.edu/dvp/Public/noise-paper.pdf [11] Department of Energy Approves
Figure 9. Impact of the different types of noise on the CNN’s ability to
classify the images. After DA, the performance of the CNN does not Construction Start of NSLS-II Project | BNL Newsroom. (2018). [12]
https://www.tensorflow.org/tutorials/deep_cnn [13] S. J. Pan and Q. Yang, “A survey on transfer
deteriorate as much as before DA when exposed to images with higher learning”. IEEE Transactions on Knowledge and Data Engineering 22(10):1345 - 1359 · November
levels of Gaussian noise and Salt and Pepper noise. This demonstrates
2010. [14] Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet classification with deep
that the network has truly become more robust to noise. After trained convolutional neural networks. In NIPS, pp. 1106–1114, 2012 [15] Gao, H. (2018). A Walk-through
of AlexNet – Hao Gao – Medium. [16] J. Salamon and J. P. Bello, “Deep convolutional neural
on noisy images, it becomes better at generalizing and classifying
different types of images. networks and data augmentation for environmental sound classification” IEEE Signal Processing
Letters, 4(3):279-283, 2017.
Figure created by the Finalist.