Filter Selection for Spectral Estimation Using a Trichromatic Camera

David Collin Day B.S. Imaging Science Rochester Institute of Technology (2001)

Filter Selection for Spectral Estimation Using a Trichromatic Camera By David Collin Day B.S. Imaging Science Rochester Institute of Technology (2001) A thesis submitted in partial fulfillment of the requirements of for the degree of Master of Science in the Chester F. Carlson Center for Imaging Science of the College of Science Rochester Institute of Technology October 2003

Signature of Author______________________________________________

Accepted by____________________________________________________
Coordinator, M.S. Degree Program Date

ii

CHESTER F. CARLSON CENTER FOR IMAGING SCIENCE COLLEGE OF SCIENCE ROCHESTER INSTITUTE OF TECHNOLOGY ROCHESTER, NEW YORK CERTIFICATE OF APPROVAL

M.S. DEGREE THESIS

The M.S. Degree Thesis of David Collin Day has been examined and approved by the thesis committee as satisfactory for the thesis requirement for the Master of Science degree

__________________________________________ Dr. Roy S. Berns, Thesis Advisor __________________________________________ Dr. Francisco H. Imai __________________________________________ Dr. Mark D. Fairchild __________________________________________ Date

iii

THESIS RELEASE PERMISSION ROCHESTER INSTITUTE OF TECHNOLOGY COLLEGE OF SCIENCE CHESTER F. CARLSON CENTER FOR IMAGING SCIENCE

Filter Selection for Spectral Estimation Using a Trichromatic Camera I, David Collin Day, hereby grant permission to the Wallace Memorial Library of R.I.T. to reproduce my thesis in whole or in part. Any reproduction will not be for commercial use or profit.

Signature:________________________________

Date:____________________________________

iv

Filter Selection for Spectral Estimation Using a Trichromatic Camera By David Collin Day A thesis submitted in partial fulfillment of the requirements of for the degree of Master of Science in the Chester F. Carlson Center for Imaging Science of the College of Science Rochester Institute of Technology

Abstract: Current imaging practices are based on exploiting metamerism to record and reproduce images. As a result, the data obtained in these images are dependent on the viewing conditions and the observer. While these methods produce acceptable results for day to day use, they often do not exhibit the type of accuracy and control required for scientific purposes such as art conservation. As a solution, many research institutions are now advocating the use of multispectral imaging to record the objects fundamental spectral properties to remove the datas dependency on the observer and viewing environment. The research described in this thesis involved determining if a trichromatic camera and readily available filters can be used for spectral estimation purposes. The Pixel Physics TerraPix camera system was characterized, its response to a target and 105 Kodak Wratten Filters under tungsten illumination was simulated, and spectral reflectance estimations were generated. The top filter candidates were chosen based on their simulated performance. These filters were then used in an imaging experiment designed to approximate conditions that would be found in an art gallery or other place where copy work is performed. The results of the imaging experiment were compared with the simulation, and shortcomings of the model were identified. The results of the experiment show that a camera model can be used as a guiding tool to make filter selections for spectral estimation.

Acknowledgements: Special thanks go to the following individuals: Dr. Roy S. Berns, for giving me the opportunity to work on this project, making it possible for me to see various works of art in a manner that most people do not know exist, and helping me improve as an individual. Dr. Francisco H. Imai, who gave me help and pointers, but more importantly made me look at everything in a different manner, helped me realize that things are not as bad as they seem, and for simply being an incredibly supportive friend throughout this entire process. Dr. Mark D. Fairchild, for his advice and recommendations, and helping me out in a bind. Lawrence Taplin, for spending time answering my questions, helping me improve my writing and research techniques, and listening to my problems and concerns. All the people at the Munsell Color Science Laboratory, who have at one point or another given me a hand. My sponsors, the Andrew W. Mellon Foundation, the National Gallery of Art, Washington D.C., the Museum of Modern Art, New York City, and Pixel Physics for their generous funding and use of equipment which made this thesis possible. My parents, for being there when I needed them most. Taryn, for all her patience, support and making me feel like someone when I felt like I was nothing. Thank you all

vi

Table of Contents:
Table of Contents:............................................................................................................. vii List of Figures:................................................................................................................... ix List of Tables: .................................................................................................................. xiv Munsell Color Science Laboratory Spectral Notation:..................................................... xv Chapter 1 - Introduction and Overview: .......................................................................... 1-1 Introduction:................................................................................................................. 1-1 Overview:..................................................................................................................... 1-6 Chapter 2 - Spectral Imaging Systems Background: ....................................................... 2-1 Narrow band capture:................................................................................................... 2-1 Wide band capture: ...................................................................................................... 2-2 Trichromatic camera with absorption filters:............................................................... 2-3 Conclusions:................................................................................................................. 2-4 Chapter 3 - Equipment:.................................................................................................... 3-1 Purpose:........................................................................................................................ 3-1 Pixel Physics TerraPix camera system: ....................................................................... 3-1 Elinchrome Scanlite Digital 1000:............................................................................... 3-2 Targets: ........................................................................................................................ 3-3 Filters: .......................................................................................................................... 3-6 Processing: ................................................................................................................... 3-7 Chapter 4 - Filter Selection for Spectral Estimation using a Noiseless Camera Model: . 4-1 Purpose:........................................................................................................................ 4-1 Filter Selection Method: .............................................................................................. 4-1 Spectral Reconstruction Theory:.................................................................................. 4-2 Experimental:............................................................................................................... 4-4 Results and Discussion: ............................................................................................... 4-9 Conclusions:............................................................................................................... 4-22 Chapter 5 - Modeling the Camera Noise for Simulation: ................................................ 5-1 Purpose:........................................................................................................................ 5-1 Noise Sources: ............................................................................................................. 5-2 Experimental:............................................................................................................... 5-3 Model Results: ........................................................................................................... 5-13 Conclusions:............................................................................................................... 5-17 Chapter 6 - Filter Selection for Spectral Estimation Incorporating a Noise Model: ....... 6-1 Purpose:........................................................................................................................ 6-1 Theory of Spectral Reconstruction using a direct pseudo-inverse transformation:..... 6-1 Experimental:............................................................................................................... 6-2 Results and Discussion: ............................................................................................... 6-3 Conclusions:............................................................................................................... 6-29 Chapter 7 - Imaging and Data Comparison: .................................................................... 7-1 Purpose:........................................................................................................................ 7-1 Imaging: ....................................................................................................................... 7-1 Data Comparison and Error Source Analysis: ............................................................. 7-7 Model limitations:.................................................................................................... 7-7

vii

Equipment Limitations: ......................................................................................... 7-17 Conclusions:............................................................................................................... 7-18 Chapter 8 - Filter Combination Analysis ......................................................................... 8-1 Purpose:........................................................................................................................ 8-1 Spectral estimation and performance evaluation: ........................................................ 8-1 Results and Discussion: ............................................................................................... 8-4 Wratten 55 and NF combination.............................................................................. 8-6 Wratten 60 and NF filter combination ................................................................... 8-19 Wratten 2C and NF combination ........................................................................... 8-29 Sensitivity of the pseudo-inverse transformation matrix:.......................................... 8-33 Target and lighting analysis:...................................................................................... 8-36 Conclusions:............................................................................................................... 8-39 Chapter 9 - Conclusions and Future Research:................................................................ 9-1 Conclusions:................................................................................................................. 9-1 Future Improvements and Research: ........................................................................... 9-5 Chapter 10 - References:................................................................................................ 10-1 A. Matlab Programs ....................................................................................................... A-1 generate_dc: Used to generate the digital counts based on equation 3.6................... A-1 combinations_a.m used to calculate data for a noiseless simulation ....................... A-2 simulate_noise_il2_pyth simulates pixels and noise using Matlab imnoise() function ..................................................................................................................................... A-6 make_transforms_pyth.m computes transforms from simulated data ................... A-11 reflectance_estimates_from_noise_pyth.m creates reflectance estimates from simulated noise.......................................................................................................... A-13 pullDC_30.m takes digital counts from 1/30th second images .............................. A-16 pullDC.m pull digital counts from optimized images............................................ A-18 make_pixel_transforms.m calculates the transforms based on experimental data A-21 r_final_2.m calculates the results of the experimental data................................... A-22

viii

List of Figures:
Figure 1.1 General process serving as the basis for this thesis..................................... 1-5 Figure 2.1 Quantix camera with LCTF attached. ......................................................... 2-2 Figure 2.2 The VASARI system (National Gallery, London). ..................................... 2-3 Figure 2.3 IBM Pro3000 scanning camera. .................................................................. 2-4 Figure 3.1 - TerraPix camera system. .............................................................................. 3-1 Figure 3.2 - TerraPix unfiltered spectral sensitivities...................................................... 3-2 Figure 3.3 - Scanlite with light diffuser. .......................................................................... 3-3 Figure 3.4 - Elinchrome relative spectral power distribution. ......................................... 3-3 Figure 3.5 Reflectance spectra of all samples on the Esser and Blues characterization target (top) and the Gamblin paint sample verification target (bottom). ................. 3-5 Figure 3.6 - Targets imaged: Blue pigments (upper left), Macbeth Colorchecker (center), Esser test target (upper right), Kodak Gray Scale (left/center), halon disk (right/center), Macbeth CCDC (lower left), Gamblin pigments (lower right) ........ 3-6 Figure 4.1 - Balzers UV/IR filter transmittance............................................................... 4-7 Figure 4.2 - TerraPix spectral sensitivities with UV/IR cutoff applied. .......................... 4-8 Figure 4.3 - Histogram of the average percent RMS spectral error calculated from all filter combinations. .................................................................................................. 4-9 Figure 4.4 - Histogram of the average CIEDE2000 calculated from all filter combinations. ......................................................................................................... 4-10 Figure 4.5 - Average CIEDE2000 vs. average. RMS spectral error plot used to aid in selecting threshold sorting criteria, with blue representing the entire set, green representing the first sort, and red representing the second sort............................ 4-12 Figure 4.6 - Maximum CIEDE2000 vs. maximum RMS spectral error plot used in selecting threshold sorting criteria, with blue representing the entire set, green representing the first sort, and red representing the second................................... 4-13 Figure 4.7 Flowchart outlining general selection process for this case. ..................... 4-15 Figure 4.8 - Wratten 81 yellowish filter spectral transmittance..................................... 4-16 Figure 4.9 - Filtered and unfiltered TerraPix spectral sensitivities after using the Wratten 81............................................................................................................................ 4-17 Figure 4.10 - Transformation matrix resulting from the Wratten 81 and unfiltered combination. Note the extremely large scale........................................................ 4-19 Figure 4.11 - Spectral Transmittance of the Wratten 40 and 80A filters....................... 4-20 Figure 4.12 - TerraPix Camera sensitivities after filtration with the Wratten 40 and 80A filters. ..................................................................................................................... 4-21 Figure 4.13 - Transformation matrix coefficients resulting from the Wratten 40 and 80A combination. The y scale is much smaller than in figure 4.10, suggesting less sensitivity to noise.................................................................................................. 4-22 Figure 5.1 - a, b, and c - TerraPix signal variance relationship determined from images of the Macbeth ColorChecker DC for each channel. ................................................... 5-6 Figure 5.2 - a, b, and c - Calculated vs. measured average digital count verifying that the slopes are approximately one and that constants are correct. .................................. 5-9 Figure 5.3 - a, b, and c Measured pixel distributions of CCDC patch D2 (red), verifying that the distribution of digital counts for a patch is approximately Gaussian........ 5-12

ix

Figure 5.4 - Camera model flowchart outlining the pixel simulation process............... 5-13 Figure 5.5 - a, b, and c - Average measured digital counts vs. simulated digital counts of the CCDC after being simulated with the model including flat fielding for each channel. .................................................................................................................. 5-15 Figure 5.6 - a, b and c - Cumulative distribution plots of the measured (black dashed) and simulated (colored solid) digital counts (x), verifying that the pixels are distributed with an approximate Gaussian density. ............................................... 5-17 Figure 6.1 - Average % RMS spectral error histogram calculated using the top 1,351 filter combinations from the noiseless case. ..................................................................... 6-3 Figure 6.2 - Average CIEDE2000 histogram calculated with the noise model and top 1,351 filter combinations from the noiseless case. .................................................. 6-4 Figure 6.3 - Average % RMS spectral error of the noise case vs. the noiseless case plotted to look for correlations. See text for explanation of color coding. ......................... 6-5 Figure 6.4 - Average CIEDE2000 from the noise case vs. the noiseless case. See text for explanation of color coding. .................................................................................... 6-6 Figure 6.5 Spectral Transmittance of the Wratten 55. ................................................ 6-11 Figure 6.6 Camera sensitivities resulting from the use of the Wratten 55 and unfiltered combination. The dashed lines represent the resulting filtered sensitivities......... 6-12 Figure 6.7 Normalized camera signal plots resulting from the Wratten 55 and unfiltered combination. The x axis represents wavelength (nm) and y axis represents normalized spectral sensitivity............................................................................... 6-13 Figure 6.8 Difference plot of estimated spectra from measured spectra of the Gamblin verification target resulting from the use of data simulated with the Wratten 55 and unfiltered camera sensitivities in the noiseless case. ............................................. 6-14 Figure 6.9 - Difference plot of estimated spectra from measured spectra of the Gamblin verification target resulting from the use of data simulated with the Wratten 55 and unfiltered camera sensitivities in the noise case. ................................................... 6-15 Figure 6.10 Individual reflectance spectra for both cases using the Wratten 55 and unfiltered camera sensitivities compared with the measured data. The x axis represents wavelength (nm) and y axis represents reflectance factor. See text for color and line codes. .............................................................................................. 6-17 Figure 6.11 Spectral transmittance of the Wratten 80D and 90 filters. ...................... 6-18 Figure 6.12 Camera sensitivities with the Wratten 80D and 90 filters applied. The dashed lines represent sensitivities resulting from the use of the Wratten 90. ...... 6-19 Figure 6.13 Normalized camera sensitivities resulting from the Wratten 80D and 90 filters. The x axis represents wavelength (nm) and y axis represents normalized spectral sensitivity.................................................................................................. 6-20 Figure 6.14 - Selected reflectance spectra for both cases using the Wratten 80D and 90 filtered camera sensitivities compared with the measured data. The x axis represents wavelength (nm) and y axis represents reflectance factor. See text for color and line codes. ..................................................................................................................... 6-21 Figure 6.15 - Spectral transmittance of the Wratten 38 and 60 filters........................... 6-22 Figure 6.16 - Camera sensitivities with the Wratten 38 and 60 filters applied. The dashed lines represent the sensitivities resulting from the use of the Wratten 60. ............ 6-23

Figure 6.17 Normalized camera sensitivities resulting from the Wratten 38 and 60 filters. The x axis represents wavelength (nm) and y axis represents normalized spectral sensitivity.................................................................................................. 6-24 Figure 6.18 - Difference plot of the measured and estimated Gamblin spectra using the WR-38 and WR-60 filter combination in the noise case. ...................................... 6-25 Figure 6.19 Camera sensitivities resulting from the Wratten 2C and unfiltered combination. The dashed line represent the unfiltered sensitivities. .................... 6-26 Figure 6.20 - Selected reflectance spectra for both cases using the Wratten 2C and unfiltered camera sensitivities compared with the measured data. The x axis represents wavelength (nm) and y axis represents reflectance factor. See text for color and line codes. .............................................................................................. 6-27 Figure 6.21 - Difference plot of estimated and measured spectra resulting from the noiseless simulation using the Wratten 2C and unfiltered data. ............................ 6-28 Figure 6.22 - Difference plot of estimated and measured spectra resulting from the simulation including noise using the Wratten 2C and unfiltered data. .................. 6-29 Figure 7.1 - IR Cutoff Filter mounted on the camera. ..................................................... 7-2 Figure 7.2 Scene setup at MCSL. ................................................................................. 7-2 Figure 7.3 Scene dimensions and geometry. ................................................................ 7-3 Figure 7.4 - Histogram examples used to help determine optimal exposure time. For these two, the unfiltered image was used. The optimized (a) was taken at 1/30 of a second. One stop greater (b.) taken at 1/15th of a second yields over exposure. .... 7-5 Figure 7.5 - Imaging pipeline used for all filter combinations. ....................................... 7-7 Figure 7.6 - a, b, and c - Average measured digital counts versus average simulated digital counts for the unfiltered image of the CCDC............................................. 7-10 Figure 7.7 - Experimental versus simulated variance of the unfiltered CCDC, showing an inability to accurately model variance to match the experiment. .......................... 7-14 Figure 7.8 - a, b, and c - Average experimental versus simulated data for the Esser and blues data, showing good linear fit but off by a scalar for the CCDC................... 7-16 Figure 7.9 - Experimental versus simulated variance for the Esser and blues target showing the models inability to handle texture. ................................................... 7-16 Figure 8.1 Flowchart of the data processing pipeline. .................................................. 8-2 Figure 8.2 - Data processing at the evaluation level........................................................ 8-3 Figure 8.3 - Estimated spectra of samples where RMS spectral error decreased and CIEDE2000 increased for the 55 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor................ 8-7 Figure 8.4 - Difference of estimated reflectance spectra from measured reflectance spectra for both optimized and unoptimized cases for the 55 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference.... 8-8 Figure 8.5 - Reflectance of blank patch 37 for the optimized and unoptimized case compared with measured spectra. .......................................................................... 8-10 Figure 8.6 Histogram of individual pixel %RMS spectral error for blank patch 37 - both optimized and unoptimized case............................................................................ 8-10 Figure 8.7 a and b - Histogram of individual pixel CIEDE2000 for blank patch 37 optimized and unoptimized case............................................................................ 8-11

xi

Figure 8.8 - Estimated reflectance spectra of samples where both RMS spectral error and CIEDE2000 decrease for the 55 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor.............. 8-13 Figure 8.9 - Difference plots for estimated spectra samples where both RMS spectral error and CIEDE2000 decrease for the 55 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference.......................................... 8-14 Figure 8.10 - XYZ arrow plots for 55 and NF combination comparing the unoptimized and optimized exposures........................................................................................ 8-15 Figure 8.11 - L*a*b* arrow plots for the Wratten 55 and NF combination. ................. 8-16 Figure 8.12 -55 and NF transformation matrix channels with the blue line representing the channel from the unoptimized matrix and the green dashed line representing the optimized matrix channels. .................................................................................... 8-18 Figure 8.13 - Estimated reflectance spectra of samples where RMS spectral error decreases and CIEDE2000 increases for the 60 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor. ......... 8-20 Figure 8.14 - Difference of estimated reflectance spectra of samples where RMS spectral error decreases and CIEDE2000 increases for the 60 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference............................... 8-21 Figure 8.15 - Estimated reflectance spectra of samples where both RMS spectral error and CIEDE2000 decrease for the 60 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor.............. 8-22 Figure 8.16 - Difference plots for estimated spectra samples where both RMS spectral error and CIEDE2000 decrease for the 60 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference.......................................... 8-23 Figure 8.17 - XYZ arrow plots for the Wratten 60 and NF filter combination. ........... 8-25 Figure 8.18 - L*a*b* arrow plots for the Wratten 60 and NF combination. ................. 8-26 Figure 8.19 - Wratten 60 and NF transformation matrix channels with the blue line representing the channel from the unoptimized matrix and the green dashed line representing the optimized matrix channels. ......................................................... 8-28 Figure 8.20 Selected spectra estimates for the Wratten 2C and NF combination. The red, dashed line represents the unoptimized estimate, the green x line is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor. ..................................................................................................................... 8-30 Figure 8.21 - Transformation matrix channels for the Wratten 2C and NF combination with the blue line representing the channel from the unoptimized matrix and the green dashed line representing the optimized matrix channels. ............................ 8-32 Figure 8.22 - Progression of estimated reflectance spectra through the simulations and imaging cases for Cobalt Blue (49) on the Gamblin paint target. ......................... 8-34 Figure 8.23 - Plots of curves generated from individual pixels from Cobalt Blue demonstrating the sensitivity of the pseudo-inverse transform. ............................ 8-35

xii

Figure 8.24 Average reflectance over wavelength for the Esser and blues characterization target............................................................................................ 8-37 Figure 8.25 Average Esser and blues target reflectance multiplied by source spectral power distribution over wavelength....................................................................... 8-38 Figure 8.26 Resulting camera signals before integration over wavelength. ............... 8-39

xiii

List of Tables:
Table 4.1 - Cumulative contribution index and metrics for the 1931 standard observer and illuminant D65 calculated in the reconstruction of the Esser and Blues target reflectances. ............................................................................................................. 4-6 Table 4.2 - Metric correlation coefficients. ................................................................... 4-10 Table 4.3 - Sorting Criteria and resulting numbers of combinations............................. 4-13 Table 4.4 Metrics of the top eight filter combinations resulting from the noiseless simulation............................................................................................................... 4-14 Table 4.5 Integrated camera signals of the top eight filters resulting from the noiseless simulation............................................................................................................... 4-14 Table 4.6 - Matrix for Wratten 81 and No Filter Combination. .................................... 4-18 Table 4.7 - Transformation matrix for Wratten 40 and 80A filters. .............................. 4-21 Table 6.1 - Selection criteria used to define magenta, red, and cyan groups representing different types of performance................................................................................. 6-7 Table 6.2 - Selected Filter Combinations sorted by average % RMS spectral error. C=cyan, M=magenta, R=red, G=green see text for definitions............................ 6-8 Table 6.3 Integrated camera signals of the selected filter combinations. ..................... 6-9 Table 7.1 - Tested Kodak Wratten absorption gelatin filter combinations (NF denotes no filter was used)......................................................................................................... 7-4 Table 7.2 Optimized filter times for Kodak Wratten filters.......................................... 7-5 Table 7.3 Maximum digital counts between optimized time and optimized time + 1 stop........................................................................................................................... 7-6 Table 7.4 Comparison of average experimental and model values for the CCDC..... 7-11 Table 7.5 Comparison of model and experimental data with exposure times divided out. ................................................................................................................................ 7-12 Table 8.1 - Average statistics from the Gamblin verification target................................ 8-4 Table 8.2 - Average performance for Esser and Blues characterization data.................. 8-5 Table 8.3 - CIEDE2000 Components for the 55 and NF combination, showing an increase in C. ........................................................................................................ 8-12 Table 8.4 - CIEDE2000 components for the 60/NF filter combination showing increase in C............................................................................................................................ 8-24 Table 8.5 - Average and individual digital counts from Cobalt Blue Patch with resulting quality metrics........................................................................................................ 8-35

xiv

Munsell Color Science Laboratory Spectral Notation:

constants and variables functions vectors matrices transpose inverse mean estimate covariance pseudo-inverse wavelength spatial position number of pixels number of wavelengths number of color samples number of image planes (bands) time spectral reflectance of object SPD of illuminant spectral radiance spectral irradiance spectral sensitivities of detector spectral transmittances of filters digital counts of ith band noise residual error transformation matrix eigenvectors eigenvalues weights a, b,, z a, b, , z a, b, , z A, B, , Z
T -1

bar hat Cov pinv x, y - origin (1,1) k n q b t r r p p L I s s f f di d

R P L S F D A,T E W

e a w

e a w

xv

Chapter 1 - Introduction and Overview:

Introduction: One picture is worth a thousand words.
Fred R. Barnard

Images are used for a variety of purposes. They serve as records and give historical information about time periods ranging from the paintings of cavemen to the art of the middle ages to the digital images of today that will be used by future generations to study the occurrences of our life and times. Images are used to convey ideas and are a form of self expression. They can evoke powerful feelings in an individual and are constantly manipulated in the mass media to sway public opinion. Depending on how images are recorded, the spatial, temporal, and color properties can be related to physical properties providing data used in various scientific disciplines allowing us to learn about ourselves and our environment. The acceptable quality level for the imaged data typically depends upon the application it is used for. The average consumer or business is not as concerned with accuracy as they are with aesthetics. For example, it is often desirable to alter the image to match memory color as opposed to actual color. Many businesses and advertising firms alter or enhance the appearance of a product in order to increase sales. There are many different situations where appearance is more important than truth. For many other applications, the accuracy of the data is of key importance. This is almost always relevant to scholarly, scientific, medical, and military applications. The accuracy and quality of medical imaging can mean the difference between a correct and an incorrect diagnosis. In scholarly and scientific applications, such as art preservation

1-1

and reproduction, accurate high quality imaging is needed for a variety of reasons. The images are often used for archival purposes, where an accurate representation is necessary to preserve the piece long after it has deteriorated (Day, E.A. 2003) or to make sure that a piece is being restored to remain true to the artists original vision. Also, if a reproduction is ever desired, the quality of the data is of paramount importance as the output of any system can only be as good as its worst component. Many imaging and reproduction devices and applications are designed to take advantage of the human visual system. These systems rely on metamerism (Berns 2000), where a visual match can be achieved between two objects with different physical properties. Examples of metameric systems include television, which uses red, green and blue phosphors or digital cameras that are designed with sensors using color filter arrays. The major consequence of relying on metameric systems is that the data acquired by the device are dependent on the illuminant and the observer (Imai 2000b; Imai 2002). This means that images or reproductions will only match the original under a given set of viewing conditions. As a result, images made in this manner lack the accuracy required for any kind of archiving or analysis. An excellent example comes from a case in the 1930s, when much art restoration work was based solely on visual matching. A Picasso painting from his blues period was being restored, and several inpaintings were made by the conservator. While the blue pigments he used appeared to match visually, they were spectrally different from the originals used by Picasso. Later, when this painting was imaged with color film, the areas that had been inpainted by the conservator appeared different to the camera as it had a different set of spectral sensitivities, and the inpainted areas appeared as purple spots in subsequent reproductions.

1-2

A solution to the metamerism problem is the use of multispectral imaging. Multispectral imaging involves the use of sampling multiple channels correlating to different points across the electromagnetic spectrum. Several different systems have been devised at various institutions around the world to research methods of multispectral image capture. Many factors must be taken into consideration to make multispectral imaging work. There are many questions that need to be asked, such as what will the images be used for and how is an acceptable level of quality defined? How many channels are appropriate? Where do the peaks of the channels need to fall in the visible spectrum to create the appropriate sampling interval? How will the imaging be

performed? The types of lights, the targets, the camera characteristics, all must be taken into consideration. The main purpose of this thesis was to develop a method to select the best filters from a set of readily available filters to be used with a trichromatic digital camera that will provide reasonable results with respect to both colorimetric and spectral performance, and to identify the potential problems associated with various components of the system. Most of the previous work done using this method has been performed at MCSL over the last six years, beginning with the work of Imai, Fairchild, and Rosen (Fairchild 2001; Imai 1998a-i; Imai 1999; Imai 2000a,b; Imai 2002; Rosen 1999), and this work is a natural continuation by trying to develop methods to help find the best set of filters to be used with a given trichromatic camera as was suggested by Imai (2000a). The proposed method for achieving these goals was to use the known characteristics of the system components to simulate the cameras filtered response to a known characterization and verification target. Combinations of these data are then used

1-3

to establish a relationship between digital output and spectral reflectance.

This

relationship is then used to make spectral reflectance estimations of a verification target. Once the spectral reflectance of the verification target is estimated, appropriate quality metrics can be used and a filter combination can be chosen for the trichromatic camera in question. Figure 1.1 shows a flowchart of the general process, which was used as the basis for this research.

1-4

System Characteristics (Camera, Lights, Targets)

Filters

Simulated Data and Reflectance Estimates Improve Model Results Unacceptable

Acceptable

Select Filter Combinations

Image

Compare Model and Experiment Draw Conclusions

Figure 1.1 General process serving as the basis for this thesis.

It is hoped that what has been learned from this thesis will lay the groundwork for filter selection methods, serve as a proof of concept for the ability of a trichromatic

1-5

camera to perform spectral reflectance estimation, be used as part of a recommendation to the National Gallery of Art, Washington D.C. and The Museum of Modern Art in New York City in the construction of a spectral imaging system, as well as add to and further document the growing body of spectral imaging research at MCSL.

Overview: Chapters Two and Three give some background on spectral imaging and the equipment used for this thesis. Chapter Two briefly discusses some of the previous research done with regards to spectral imaging. Here, the three main types of spectral imaging systems are discussed and some of their advantages and drawbacks are described. Chapter Three describes the equipment used for this thesis. This includes the camera system, lights, targets, and other major aspects of the system being evaluated. Chapters Four through Six discuss the simulations performed and how the model progresses from a simple linear model that is only concerned with average values to a more complex model that incorporates aspects of system noise. Chapter Seven describes the imaging experiment, where filter selections made based on the results of the simulations are used in a practical situation. The experimental data are then compared with the simulated data, and sources of error are identified. Chapter Eight gives an analysis of three filter combinations, which were chosen based on simulated performance, experimental performance, and an example of poor overall performance. Finally, Chapter Nine gives a recap of this thesis and makes suggestions for future research.

1-6

Chapter 2 - Spectral Imaging Systems Background:

Spectral imaging and reproduction has become an increasingly important area of research at the Munsell Color Science Laboratory (MCSL) as well as at other institutions around the world (Berns 1999; Berns 2003; Burns 1998; Burns 1999; Day E.A. 2003; Day E.A. 2002; Fairchild 2001; Haneishi 2000; Hardeberg 2003; Hardeberg 1999; Imai 1998a-i; Imai 1999; Imai 2000a,b; Imai 2002; Imai 2003; Johnson 1998; Knig 1998; Knig 1999; Quan 2000; Rosen 1999). Several different methods have been used to perform spectral image capture and reproduction. Earlier systems at MCSL were

designed using film as the spectral detector (Rosen 1999). As the availability and quality of digital cameras has increased, CCD cameras have become commonly used. There are currently three methods that are widely used in designing spectral-based digital camera systems. These are narrow band image capture with a monochromatic sensor, wide band image capture with a monochromatic sensor, and wide band image capture with an offthe-shelf trichromatic digital camera. Each method has its advantages and drawbacks.

Narrow band capture: The first method involves the use of a monochromatic digital camera and narrow band sampling. There are many choices of technology available that provide the

spectrally narrow filtration required. Typically, a liquid crystal tunable filter (LCTF) is used as it has the advantages of being solid state and computer controlled meaning that errors from registration and calibration between images becomes minimal. Although the LCTF was originally thought not to be affected by the angular dependencies that other narrow band filtration methods suffer from (Imai 2000b), it has been shown in recent 2-1

experiments that the LCTF still suffers from angular dependencies, but not as severe as in other technologies, such as normal interference filters. While the LCTF delivers the most accuracy as many samples across the visible spectrum are used, it is generally the most costly, time consuming, and generates large amounts of data that must then be managed later. Figure 2.1 shows a LCTF attached to a Roper Scientific Quantix at the MCSL Spectral Imaging laboratory.

Figure 2.1 Quantix camera with LCTF attached.

Wide band capture: The second method uses a smaller set of wide band filters to capture digital image data and then uses reconstruction algorithms to estimate the reflectance spectra of an object. This approach produces reasonable results because both man made and natural materials generally have smooth spectral reflectance shapes, thus the sampling interval can be reduced to between eight and ten channels with broader bandpass filters while still achieving accurate results. This has generally been the accepted method of image capture as it is a good compromise between accuracy and efficiency. Systems using this

approach have been designed and used at MCSL, and are also in use at institutions such

2-2

as the National Gallery in London, England and the Uffizi Gallery in Florence, Italy. These institutions are involved in the VASARI (Visual Arts System for Archiving and Retrieval of Images) program, which has successfully used a seven channel camera to capture multi-spectral information and map it directly to colorimetric space. Figure 2.2 shows the VASARI system.

Figure 2.2 The VASARI system (National Gallery, London).

Trichromatic camera with absorption filters: The third method uses a high quality trichromatic digital camera in conjunction with spectral absorption filters to acquire unique spectral information. This method enables three channels of data to be captured per exposure as opposed to one. This greatly increases the speed of capture and allows the use of technology that is readily available and does not need to be specialized. However, this increase in efficiency usually comes at the price of accuracy, as the system comes initially constrained by the

2-3

red, green, and blue spectral sensitivities of the camera. Figure 2.3 shows the IBM Pro3000 system used in many of the experiments performed at MCSL.

Figure 2.3 IBM Pro3000 scanning camera.

Conclusions: This chapter has given a brief discussion of the three main technologies most commonly used to perform spectral image capture. Usually, the choice of which

technology to use will depend upon several different factors, including the purpose of the images, price, availability, and accuracy. As was mentioned in the introduction, this thesis is mainly concerned with the trichromatic approach and developing methods to help use it more effectively.

2-4

Chapter 3 - Equipment:
Purpose: The main goal of this thesis was to show that it is possible to select filters that can be used with a high resolution trichromatic digital camera to perform spectral reflectance estimation. The following equipment was used for measurement, modeling, simulation, and for the experiment.

Pixel Physics TerraPix camera system: The TerraPix camera by Pixel Physics, shown in Figure 3.1, uses a Megavision T4 camera back. This back is based on the Kodak KAF-16801E CCD sensor. This sensor provides images with a resolution of 4096 X 4096 pixels. The CCD uses a Bayer pattern color filter array, and the raw data must be interpolated to deliver red, green, and blue image planes. The camera back was used with a Contax 645 camera body and an 80mm Carl Zeiss T* Diagnon lens. The image capture software was custom made by Pixel Physics. Other processing software used was created by Lawrence Taplin at

MCSL. For simulations, the cameras spectral sensitivities were also provided by Pixel Physics and are shown in Figure 3.2.

Figure 3.1 - TerraPix camera system.

3-1

0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 400

Red Green Blue

Spectral Sensitivity (Quantum Efficiency)

450

500

550 600 Wavelength (nm)

650

700

Figure 3.2 - TerraPix unfiltered spectral sensitivities.

Elinchrome Scanlite Digital 1000: Two Elinchrome Scanlite Digital 1000 studio lights were used for the simulations and experiment. Chimera Pro Light diffusers were attached to the light sources to provide diffuse, even lighting. Figure 3.3 shows the light sources and Figure 3.4 shows their relative spectral power distribution measured with a Photo Research Spectra Scan PR-650 handheld spectroradiometer. At the image plane, the correlated color

temperature was measured to be approximately 3,334 degrees Kelvin. The luminance was 541 cd/m2.

3-2

Figure 3.3 - Scanlite with light diffuser.

1.8 1.6 1.4 1.2

Relative Power

1 0.8 0.6 0.4 0.2 0 400

450

500

550 600 Wavelength (nm)

650

700

Figure 3.4 - Elinchrome relative spectral power distribution.

Targets: Several targets were imaged during the experiment. For characterization, the reflectance data from the Esser test target (Esser TE 221, made by Esser Test Charts,

3-3

Hummeltal, Germany) was used in combination with the reflectance data of the blue pigment target consisting of phthalocyanine, ultramarine, cobalt, and Prussian blues mixed in different combinations and concentrations with titanium white. The Esser target was chosen because of its spectral variability. It was necessary to add the spectra from pigments of the blues target because these pigments are typically found in art paintings but are not represented on the Esser target. The Gamblin paint target was used as the main verification target. It was composed of 30 pigments commonly found in artwork, each mixed at two concentrations with titanium white. Both the blue pigment and Gamblin target were hand painted onto a piece of canvas board found in many art stores. Figure 3.5 shows the reflectance spectra for all the samples of both these targets. Other targets included a Kodak Gray Scale as well as the GretagMacbeth ColorChecker and ColorChecker DC, shown in Figure 3.6. The targets were measured using a

GretagMacbeth ColorEye XTH handheld integrating sphere spectrophotometer with the specular component excluded.

3-4

Figure 3.5 Reflectance spectra of all samples on the Esser and Blues characterization target (top) and the Gamblin paint sample verification target (bottom).

3-5

Figure 3.6 - Targets imaged: Blue pigments (upper left), Macbeth Colorchecker (center), Esser test target (upper right), Kodak Gray Scale (left/center), halon disk (right/center), Macbeth CCDC (lower left), Gamblin pigments (lower right)

Filters: A set of 105 Kodak Wratten filters were evaluated. These selective absorption filters are used in a wide variety of technical and photographic imaging applications. Transmittance filters were selected because their spectral properties do not have the angular dependence that is found in interference filters, shown to have a significant effect on spectral estimation and reconstruction accuracy. The nominal data used for

simulations were provided by Kodak and span the wavelength range of 400-730 nm. A Unaxis Balzers IR cutoff filter was used to limit the spectrum of interest to the visible region (400 730 nm). The transmittance data used for simulations was

measured with a Macbeth ColorEye 7000 integrating sphere spectrophotometer in the specular excluded mode.

3-6

Processing: All simulations and data processing were performed using Matlab versions 5.3 and 6.5. Code for all processes are in Appendix B, and is adaptable to any platform capable of running the Matlab environment.

3-7

Chapter 4 - Filter Selection for Spectral Estimation using a Noiseless Camera Model:
Purpose: This chapter is concerned with describing filter selection in a noiseless case. First, the theory of spectral reconstruction using multiple camera signals and eigenvector analysis is described, and then the method and metrics used to select an appropriate filter are discussed.

Filter Selection Method: The first factor that was considered was the method of searching for filter combinations. Several different approaches can be used and have been discussed

(Hardeberg 2003). The first and most intuitive method was to simply try to find a combination of filters where the dominant wavelength is equally spaced across the spectrum of interest. While this approach worked well for monochromatic cameras which only consider one channel at a time and have a relatively flat response when unfiltered, it is very difficult to find a filter that will cause three channels in a trichromatic camera to shift significantly without entirely blocking the signal in any one channel. The second, which was first proposed by Maitre and expanded on by

Hardeberg, involves maximizing the orthogonality in the characteristic reflectance vector space (Hardeberg 2003). This method was shown to be fast, but only considered one channel at a time and demonstrated suboptimal results. The final method was an

exhaustive search which considers all possible combinations of filters in question. Due to

4-1

the complex nature of the task at hand, this method was chosen because it will, in theory, find the optimal results from all combinations. Care must be taken with this approach as the number of computations can easily become very large given that the number of evaluations is determined by Eq. 4.1 g= N K N! K!(N-K)! (4.1)

where g is the number of combinations, N is the total number of filters in the set, and K are the number of filters being combined (Hardeberg 2003).

Spectral Reconstruction Theory: Reflectance spectra of objects can be estimated by using a priori spectral analysis with direct measurement and imaging of color patches to establish a relationship between camera digital counts and spectral reflectance factor (Imai 2000a). The spectral reflectances of a set of samples from a characterization target are measured and (n x q) reflectance matrix R is formed, where n is the number of wavelength measurements and q is the number of measured samples. Using principal component analysis, eigenvectors {e1e } and the associated eigenvalues are

calculated and arranged in descending order. The cumulative contribution index (CCI), vi, that describes the amount of variance explained by the first i eigenvalues is given by Eq. 4.2.
i j 1

aj (4.2) aj

vi

j =1

4-2

where a is the vector of eigenvalues. The CCI is generally used to help select the number of eigenvectors to be used in the spectral reflectance reconstruction in conjunction with other colorimetric and spectral metrics (Imai 2000a). The estimated spectral reflectance is defined by Eq. 4.3

R = Ei

(4.3) and R represents the estimated

where Ei=[e1ei], the coefficients reflectances.

T i=[a1ai] ,

A camera system gives (b x q) digital counts, D, where b is the number of channels and q is the number of samples or pixels corresponding to various samples. In general, the number of channels being used should be equal to or greater than the number of eigenvectors (Imai, 2000a) used in the estimation. The relationship between the eigenvalues Eq. 4.4: Te = i pinv(Dc ) (4.4)
i

of the target and digital counts of the characterization target Dc is given by

where pinv() denotes the Matlab function which performs the equivalent of a pseudoinverse calculation. The matrix Te can then be used to estimate eigenvalues digital counts of a verification target Dv as shown in Eq. 4.5:
i

^ i

from

Te D v

(4.5)

Finally, reflectance of the verification target Rv is estimated by substituting Eq. 4.5 into Eq. 4.3.

4-3

R v = Ei Te D v

(4.6)

A camera with linear photometric response can be modeled using Eq. 4.7, and the simulated digital counts can then be used to establish the relationship between camera digital counts and object spectral reflectance:

dk

p( )r ( )s ( )f ( )

(4.7)

where p( ) is the source relative spectral power distribution, r( ) is the spectral reflectance of the object, s( ) is the appropriate camera channel sensitivity, f( ) is the filter being used, and = R, G, or B.

Experimental: An experiment was designed to identify a subset of filters from a set of readily available filters to be used for spectral reflectance estimation with a trichromatic camera. The simulation experiment involved calculating the TerraPix cameras filtered camera signal when imaging a characterization and verification target, estimating the reflectance of the samples, and then making a filter selection that provides good performance. The eigenvectors and eigenvalues of the combination Esser and blues target were calculated using eigenvector analysis and the known reflectance spectra of the patches. The eigenvectors were then ranked from most significant to least significant and a CCI was calculated. The number of eigenvectors selected is typically a compromise between accuracy and the number of channels necessary (Imai 2000a). It has been shown in other work that typically, six channels and, therefore, six eigenvectors are sufficient for most artwork (Haneishi 2000). Table 4.1 shows the CCI and selected metrics for the first 12

4-4

eigenvectors calculated for the CIE 1931 standard observer and illuminant D65, for which all subsequent calculations will be made unless noted otherwise, showing that 99% of the variance in the data can be described with six eigenvectors, the CCI stops increasing significantly at six eigenvectors, and the metrics stop decreasing significantly between six and eight eigenvectors.

4-5

Table 4.1 - Cumulative contribution index and metrics for the 1931 standard observer and illuminant D65 calculated in the reconstruction of the Esser and Blues target reflectances. Average % RMS Average Average Average spectral Metameric Metameric error CIEDE2000 Index D65->A Index A->D65 18.2 6.7 3.3 2.4 1.7 1.3 1.1 0.9 0.8 0.6 0.5 0.3 44.71 18.02 4.18 1.29 0.97 0.52 0.27 0.18 0.16 0.14 0.15 0.06 4.70 1.34 1.27 1.25 0.29 0.20 0.15 0.11 0.04 0.02 0.02 0.02 5.78 2.08 1.49 1.43 0.30 0.23 0.15 0.13 0.04 0.02 0.03 0.03

# of Eigenvectors 1 2 3 4 5 6 7 8 9 10 11 12

CCI 0.63 0.86 0.97 0.98 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00

Based on this information, data from previous experiments analyzing color difference and spectral RMS error showing that six channels is sufficiently accurate (Imai, 2000a), and information obtained from the CCI, six channels obtained by combining the three channels from each image were used for this and the following experiments. Having determined an appropriate number of eigenvectors to use in the reconstruction, all possible combinations of data from two filters simulated with equation 4.6 were evaluated. The data were simulated for digital counts generated from a response integrated over the 400 730 nm range. This range was chosen partly because many pigments have reflectance in the near IR region that can distinguish them from pigments that may have similar reflectance spectra in the visible region and because the available transmittance data of the written filters stopped at 730 nm. This was done by using the TerraPix sensitivities filtered with a Balzers UV/IR cutoff filter. Figures 4.1 and 4.2 4-6

show respectively the spectral transmittance properties and the resulting camera spectral sensitivities of the IR cutoff filter.

1 0.9 0.8

Transmittance Factor

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500 550 600 Wavelength (nm) 650 700

Figure 4.1 - Balzers UV/IR filter transmittance.

4-7

0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 400

Spectral Sensitivity (Quantum Efficiency)

Red Green Blue

450

500

550

600

650

700

Wavelength (nm)
Figure 4.2 - TerraPix spectral sensitivities with UV/IR cutoff applied.

Using the 105 Kodak Wratten filters and the unfiltered case, a total of 106 separate sets of simulated digital counts were created and taken in combinations of two giving a total of 5565 unique combinations. These combinations were used to reconstruct the reflectance spectra of the Gamblin verification target. The mean and maximum CIEDE2000 for illuminant D65 and the 2 degree observer was calculated and the RMS spectral error over the range of 400 730 nm across all the patches was then calculated and used to determine an appropriate filter combination that would yield acceptable results both spectrally and colorimetrically.

4-8

Results and Discussion: Figures 4.3 and 4.4 show the distribution of the filter combinations with respect to the average CIEDE2000 and RMS spectral error. It not only shows that a large

percentage of the combinations perform equally well, but the range of the metrics are relatively small for those that do perform well. This lends to a degree of difficulty in selecting the best combinations from just the mean and maximum metric measures. Another method needed to be devised in order to eliminate combinations that were unlikely to perform well and to reduce the number of filter combinations to a more comprehensible level. The following method was used to sort the filters and make eliminations.

1800 1600 1400

Number of Combinations

1200 1000 800 600 400 200 0 0

10 15 Average %RMS Spectral Error

20

25

Figure 4.3 - Histogram of the average percent RMS spectral error calculated from all filter combinations.

4-9

900 800 700

Number of Combinations

600 500 400 300 200 100 0 0

10 15 Average CIEDE2000

20

25

Figure 4.4 - Histogram of the average CIEDE2000 calculated from all filter combinations.

The metrics correlation coefficients were calculated and the pair that correlated best was plotted against each other. coefficients.
Table 4.2 - Metric correlation coefficients.

Table 4.2 shows the calculated correlation

Average spectral RMS Maximum spectral RMS Average CIEDE2000 Maximum CIEDE2000

Average spectral RMS 1.0000 0.9013 0.9019 0.7780

Maximum spectral RMS .9013 1.0000 0.7810 0.7807

Average CIEDE2000 .9019 .7810 1.0000 0.9009

Maximum CIEDE2000 .7780 .7807 .9009 1.000

4-10

It shows that the average CIEDE2000 and RMS spectral error correlate best and were used as a first criteria. The maximum CIEDE2000 and RMS spectral error were then used as a second selection criteria. The combinations were then sorted by selecting threshold levels on the plot of the average metrics, observing where those points plotted on the maximum metric plot and then selecting thresholds on the maximum and observing where the new group falls on the average plot. This process was repeated until a reasonable number was found. Figures 4.5 and 4.6 show the metric plots and Table 4.3 shows the thresholds and the resulting number of filter combinations after each level of selection. In Figures 4.5 and 4.6, the blue represents the entire set, the green represents members of the set after the first sort, and the red represents members of the set after the second sort. The final sort was not plotted.

4-11

25

20
Average CIEDE2000

15

10

0 0

10 15 Average % Spectral RMS

20

25

Figure 4.5 - Average CIEDE2000 vs. average. RMS spectral error plot used to aid in selecting threshold sorting criteria, with blue representing the entire set, green representing the first sort, and red representing the second sort.

4-12

120

100

Max CIEDE2000

80

60

40

20

0 0

10

20

30 40 50 Maximum % Spectral RMS

60

70

80

Figure 4.6 - Maximum CIEDE2000 vs. maximum RMS spectral error plot used in selecting threshold sorting criteria, with blue representing the entire set, green representing the first sort, and red representing the second.

Table 4.3 - Sorting Criteria and resulting numbers of combinations.

Selection Criteria Average CIEDE2000 < 1.5, Average RMS < 2.5% Maximum CIEDE2000 < 4, Maximum Average RMS < 6% Average CIEDE2000 < .6, Average RMS < 2%

Resulting Number of Filter Combinations 1351 635 71

Once the number of selections reached a reasonable number, the filter combinations were then eliminated by calculating the area of the filtered camera sensitivities for each curve. This would eliminate curves where the signal would be insignificant in the system. Taking filter combinations where all six of the integrated

4-13

signals were greater than 2.5 reduced the list to eight combinations whose metrics are shown in table 4.4 and integrated camera signals are shown in table 4.5.
Table 4.4 Metrics of the top eight filter combinations resulting from the noiseless simulation. Average Maximum % % Spectral Spectral Average Maximum RMS RMS CIEDE2000 CIEDE2000 1.9 1.9 1.9 1.9 1.9 1.9 2.0 2.0 5.0 4.9 4.5 4.9 4.6 4.7 5.5 5.0 0.57 0.58 0.51 0.59 0.54 0.58 0.52 0.49 2.91 3.06 2.79 3.12 3.05 3.30 1.73 2.30

Filter 1 WR-40 LIGHT GREEN WR-40 LIGHT GREEN WR-66 VERY LT GREEN WR-40 LIGHT GREEN WR-66 VERY LT GREEN WR-66 VERY LT GREEN WR-81 YELLOWISH WR-3 LIGHT YELLOW

Filter 2 WR-78A BLUISH WR-80B BLUE WR-78B BLUISH WR-80A BLUE WR-80C BLUE WR-78A BLUISH NF WR-82C BLUISH

Table 4.5 Integrated camera signals of the top eight filters resulting from the noiseless simulation. Integrated Camera Signal Filter 1 WR-40 LIGHT GREEN WR-40 LIGHT GREEN WR-40 LIGHT GREEN Filter 2 WR-78A BLUISH WR-80B BLUE WR-80A BLUE R1 3.18 3.18 5.95 3.18 5.95 5.95 15.89 16.08 G1 5.24 5.24 8.26 5.24 8.26 8.26 12.85 13.07 B1 2.65 2.65 4.79 2.65 4.79 4.79 9.95 6.68 R2 3.86 4.31 7.15 3.39 6.42 3.86 17.83 8.51 G2 4.16 4.38 6.90 3.79 6.14 4.16 8.31 B2 5.00 5.62 6.86 5.33 6.61 5.00 7.68

WR-66 VERY LT GREEN WR-78B BLUISH WR-66 VERY LT GREEN WR-80C BLUE WR-66 VERY LT GREEN WR-78A BLUISH WR-81 YELLOWISH WR-3 LIGHT YELLOW NF WR-82C BLUISH

14.75 11.73

Final selections could only be made after evaluating curve shapes and the resulting camera sensitivities as well as the different metrics measured for this case. Figure 4.7 outlines the entire process in a flowchart. It was found that some unexpected filter combinations performed best. For example, the combination of signals from using the Wratten 81 Trans Pale Yellow filter (Figure 4.8) and unfiltered signals appeared to

4-14

yield a comparatively good RMS spectral error and very good colorimetric results, yet the resulting camera sensitivities are almost equivalent, as shown in Figure 4.9.
Find metrics that correlate best, plot two scatter plots of correlating metrics

Select groups with threshold levels on one plot, plot the groups on the second plot NO - Alternate plots that threshold is selected from and plotted on Is the number in the group being plotted acceptably small?

YES Find area of integrated signals - eliminate filters with low signal Evaluate remaining curves and camera sensitivities - make selections
Figure 4.7 Flowchart outlining general selection process for this case.

4-15

1 0.9 0.8
Transmittance Factor

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400

WR81 SPEC TRANS YELLOWISH NF

450

500

550

600

650

700

Wavelength (nm)
Figure 4.8 - Wratten 81 yellowish filter spectral transmittance.

4-16

0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 400

Spectral Sensitivity (Quantum Efficiency)

450

500

550

600

650

700

Wavelength (nm)
Figure 4.9 - Filtered and unfiltered TerraPix spectral sensitivities after using the Wratten 81.

This combinations performance can be further explained by looking at the transformation matrix that was used to convert the digital counts of an image to the coefficients to be used with the eigenvectors to reconstruct the spectral reflectance. Table 4.6 shows the matrix formed with the WR-81 and no filter combination.

4-17

Table 4.6 - Matrix for Wratten 81 and No Filter Combination. Camera Channel 1 2 Coefficients 3 4 5 6 R1 -12.01 129.64 163.70 -398.18 -34.94 180.80 R2 12.21 -116.10 -145.28 358.61 34.96 -156.08 G1 143.65 -115.57 132.79 400.59 -873.72 -39.73 G2 -123.11 102.58 -118.82 -349.84 753.15 37.43 B1 10.81 -246.16 -215.57 -381.71 400.66 -558.21 B2 -10.28 220.30 188.43 322.87 -337.30 464.52

This matrix is showing the contribution of each channel to a particular coefficient. These matrices work in a noiseless case because it is built off the average simulated values of the Esser and blues target. In reality, a group of pixels associated with a particular reflectance spectrum will have variance. This particular matrix would then amplify the variance and introduce an increased amount of error into the system. For example, the 4th coefficient shows that the contributions of the first two channels are multiplied by approximately -398.18 and 358.61. If the digital counts from a particular patch

corresponding to these channels varied even by a very small amount, the difference would become greatly amplified. This example can be looked at as an example where there is a mathematical solution to the problem with no physical meaning. Figure 4.10 shows plots of the transformation matrix coefficients as a function of wavelength. It appears the transform is very symmetric about the x-axis and is composed of a number of steep slopes, which in turn would amplify pixel differences.

4-18

400 300 200

Transform Coefficient

100 0 100 200 300 400 400

450

500

550 600 Wavelength (nm)

650

700

Figure 4.10 - Transformation matrix resulting from the Wratten 81 and unfiltered combination. Note the extremely large scale.

A much better choice resulting from this filter selection process would be the combination of the Wratten 40 and Wratten 80A filters. Figures 4.11 and 4.12 show respectively the spectral transmittance of the filters and the cameras filtered sensitivities.

4-19

1 0.9 0.8
Transmittance Factor
WR40 WR80A

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500 550 600 Wavelength (nm) 650 700

Figure 4.11 - Spectral Transmittance of the Wratten 40 and 80A filters.

4-20

0.16 0.14 0.12

Spectral Sensitivity

0.1 0.08 0.06 0.04 0.02 0 400

450

500

550 600 Wavelength (nm)

650

700

Figure 4.12 - TerraPix Camera sensitivities after filtration with the Wratten 40 and 80A filters.

For comparison, the transformation matrix to convert camera signals to eigenvector coefficients is shown in table 4.7.
Table 4.7 - Transformation matrix for Wratten 40 and 80A filters. Camera Channel R1 1 1.4936 2 0.089341 Coefficients 3 5.0673 4 -4.4362 5 -22.791 6 4.4082 R2 12.828 -6.4197 5.3384 3.5821 22.761 5.8228 G1 -7.4842 5.3418 9.4926 -12.897 24.121 26.445 G2 14.802 0.74343 -34.009 18.277 -44.337 -31.974 B1 -0.9164 -6.2315 -15.772 -12.911 35.162 -50.31 B2 -3.4039 29.036 21.536 12.465 -13.831 19.202

The values in the transformation matrix are much smaller suggesting a more stable matrix that would be more robust to noise. Differences in the variance would not be as 4-21

amplified with this transformation matrix and would be more likely to perform well in actual imaging. Figure 4.13 shows the transformation matrix coefficients. Notice the scale on the y axis is smaller, indicating that a difference in pixel values will be less amplified. Theoretically, this transform should be more robust to noise.

40 30 20

Transform Coefficient

10 0 10 20 30 40 400

450

500

550 600 Wavelength (nm)

650

700

Figure 4.13 - Transformation matrix coefficients resulting from the Wratten 40 and 80A combination. The y scale is much smaller than in figure 4.10, suggesting less sensitivity to noise.

Conclusions: The noiseless simulation can be used as a preselection method for filter selection or guide if the cameras noise characteristics are unknown. Care must be taken in making selections based on just the metrics calculated. Factors such as filter transmittance and the resulting camera sensitivity as well as transformation matrix stability must also be

4-22

taken into consideration. Otherwise, as shown with the example of the Wratten 2C and no filter combination, erroneous selections can be made. In any filter selection case, it is imperative that noise be taken into account.

4-23

Chapter 5 - Modeling the Camera Noise for Simulation:

Purpose: While a camera with linear photometric response can be modeled as shown in equation 4.6, it is only a measure of the average signal from the camera and does not include noise. A lack of noise on a per pixel basis can lead to filter selections that do not make intuitive sense, as was shown in the noiseless case. This chapter describes how the noise variance in the TerraPix camera was modeled and applied to the basic camera equation to simulate the camera response including noise. The results of the noiseless simulation returned several filter combinations that perform similarly and made it difficult to classify the filters in terms of performance. It was also shown that some of the filter combinations that demonstrated good performance did not make intuitive sense. Filters combinations that fell into this category were usually combinations of filters that were very similar in their transmittance properties and did relatively little to alter the camera signals from each other. It was determined that the addition of noise and a simulation of multiple pixels with the appropriate noise properties could be used to create more realistic reflectance estimates and give more physical meaning to the results of simulating the cameras response to different filters for use in spectral reflectance estimation. Also, the simulation of many pixels would allow the use of a direct pseudo-inverse transform as opposed to the eigenvector analysis used in the noiseless case. By using many pixels, the system becomes overdetermined and a least squares solution can be fit.

5-1

Noise Sources: Noise is defined to be any unwanted signal that contains no information, which is added to the imager output (Eastman Kodak Company 1994). The first major source of noise comes from the CCD imager itself. A CCD camera has several sources of inherent noise that are all dependent on factors such as time, signal, and temperature. The main sources of temporal noise in a CCD imager include dark current, photon shot noise, reset transistor noise, CCD clocking noise, and noise from the output amplifier (Eastman Kodak Company 1994). Dark current noise is dependent on the operating temperature and the integration time. The dark current noise also varies across the pixels of the imager, leaving a fixed pattern noise. The dark noise can be dealt with by taking a dark image or the average of several dark images at the appropriate integration time and using the average pixel referenced data as the zero level for each pixel (Eastman Kodak Company 1994). The reset transistor noise and output amplifier noise is generally dealt with by using different methods of sampling the signal, but is beyond the scope of this research. Finally, the photon shot noise cannot be eliminated, but can also be reduced by taking several images and averaging the data. The second source of noise comes from the scene illumination. While every attempt is made to create uniform illumination across the scene, in practice there is always a certain amount of variation imposed by the unevenness in lighting and optics of the system. This noise is dealt with by taking a reference image of a uniform target, such as an even gray surface of known reflectance and using it to flat field the image, which can be mathematically described in Eq. 5.1.
Dc ( Dr Ddark ) *(Dgray Dgray Ddark Ddark )

(5.1)

5-2

where Dc is the corrected data, Dr is the raw data, Dgray is the gray card, Ddark is the dark exposure and (Dgray

Ddark ) denotes the average gray value over the area of the uniform

gray card. This is a very common method of reducing the variation in output signal across a scene and is easily done in a controlled environment.

Experimental: Equation 4.6 gives the noiseless camera model in terms of a relative, average signal. The equation is now changed to the following form:

d
where t is the integration time, counts and

p( )r ( )s ( )f ( )

(5.2)

is a constant that converts the integrated signal to digital and t

is the noise associated with the signal. In the noiseless case, the

term can be ignored, as they are constants that are dealt with in the transform generated by the eigenvector analysis. It is now necessary to include these terms because the noise is a signal dependent factor that is in turn dependent upon integration time. The noise characteristics of the camera were determined from images taken during an imaging session at the National Gallery of Art, Washington D.C. At the time, the IR cutoff filter was supplied by Pixel Physics and cuts transmittance approximately at 700 nm. Future simulations with the model will be based on the camera sensitivity and the Unaxis Balzers UV/IR blocking filter that extends transmittance into the near IR region to 750 nm, although calculations will stop at 730 nm because that is the cutoff for the nominal filter data. The target used for both characterization of the camera and verification of the model was the GretagMacBeth ColorChecker DC (CCDC), from 5-3

which 240 patches each of 2883 (31 x 31 x 3) pixels were sampled. These patches consisted of 236 individual sample areas measuring 1.3 x 1.3 cm and four samples from the large central square. The first step was to determine the variance characteristics of the camera as a function of mean signal level. This was done by taking the raw, unfiltered images and processing them with the software provided by Pixel Physics using a mode that returned the data scaled to 16 bits with no corrections applied. The mean and variance of the individual patches were then plotted and fit using the robustfit() algorithm available in Matlab for each signal channel at each exposure time. The robustfit() function uses a least squares algorithm that is iteratively reweighted and gives less weight to data that does not fit well. Therefore, the results are less sensitive to outliers in the data. Figure 5.1 a, b and c shows the normalized variance/signal relationship for a 1/15th second exposure time for each unfiltered channel. The figure shows that as the mean signal increases, there is more spread in the variance. This is usually a result of any texture or target nonuniformities. While the model is simplistic, it was determined that it would be sufficient in adding noise to simulated signal values to help eliminate cases such as those discussed in Chapter Four.

5-4

x 10 3.5 3 2.5

Variance

2 1.5 1 0.5 0
y=.5080e4.0312e4

0.5 0.1 0.15 0.2 0.25 0.3 0.35 Mean Signal Level 0.4 0.45 0.5

a. Red
3.5 3 2.5 2 x 10
5

Variance

1.5 1 0.5 0 0.5 0.1

y=.3991e4.0180e4

0.15

0.2 0.25 Mean Signal Level

0.3

0.35

b. Green

5-5

3.5 3 2.5 2

x 10

Variance

1.5 1 0.5 0
y=.3109e4.0210e4

0.5 0.1

0.15

0.2 Mean Signal Level

0.25

0.3

c. Blue Figure 5.1 - a, b, and c - TerraPix signal variance relationship determined from images of the Macbeth ColorChecker DC for each channel.

After determining the relationship between signal and variance, the relationship between the measured and simulated signal values was found by finding the each channel using a pseudo inverse calculation as shown in equation 5.3. value for

(Dm,

O) pinv(Ds, )

(5.3)

where Dm, are the measured digital counts before flat fielding for each patch of the CCDC, O is the offset, which was determined experimentally to be 4,000, Ds, are the simulated digital count from equation 4.7 multiplied by the integration time t. It was found that this value did not remain constant, so an average value over available filtered and unfiltered exposures for each channel was used. Figure 5.2 a, b, and c shows the average values calculated with the first part of equation 5.2 representing the average pixel value before the addition of noise or flat fielding at 1/15th of a second in each channel. It

5-6

shows that the slopes are approximately one, implying that the calculated constants t are correct.

and

5-7

x 10 3.5 3
Calculated Average Digital Count

y=.95x+3.4e02

2.5 2 1.5 1 0.5 0 0

0.5

1 1.5 2 2.5 Measured Average Digital Count

3.5 x 10
4

a. Red
x 10 3.5 3
Calculated Average Digital Count
4

y=1.1x2.1e02

2.5 2 1.5 1 0.5 0 0

0.5

1 1.5 2 2.5 Measured Average Digital Count

3.5 x 10
4

b. Green

5-8

x 10 3.5 3
Calculated Average Digital Count

y=1x1.8e02

2.5 2 1.5 1 0.5 0 0

0.5

1 1.5 2 2.5 Measured Average Digital Count

3.5 x 10
4

c. Blue Figure 5.2 - a, b, and c - Calculated vs. measured average digital count verifying that the slopes are approximately one and that constants are correct.

A correction used to simulate scene illumination was also created and added into the calculated mean signal. Typically, any uneven lighting in a scene will modulate the pixel value by an undetermined amount. This means that the average signal under even illumination has a correction factor of one and the pixels that are subject to uneven lighting will be modulated either above (more light) or below (less light) one. This is simulated by creating a number of random values with mean of one and a variance of .001, which is approximately the variance of a properly exposed gray card, and then using these numbers to multiply the average signal by. It is important to consider that this method assumed that a sample patch can occupy any particular position in space, and for a simulation to be valid, all the images must use the same randomly generated values.

5-9

The actual values of the measured pixels were found to have approximately Gaussian distributions. Figures 5.3 a, b, and c show histograms of the three channels taken from sample patch D2 on the CCDC with a Gaussian density curve superimposed on it. Similar results were found for the other measured patches. This suggests that noise can be simulated by creating a group of pixels all having values approximated to be the mean value for a given reflectance and then using the Matlab imnoise() function on each channel, which can distribute Gaussian noise in an image where the local variance is a function of the intensity value, as described in figure 5.1.

5-10

80 70 60 50
Count

40 30 20 10 0 2.2

2.22

2.24

2.26 2.28 2.3 2.32 Signal Level (Digital Counts)

2.34

2.36 x 10
4

a. Red
120

100

80

Count

60

40

20

0 8200

8400

8600 8800 Signal Level (Digital Counts)

9000

9200

b. Green

5-11

140

120

100

Count

80

60

40

20

0 7300

7400

7500 7600 7700 7800 Signal Level (Digital Counts)

7900

8000

c. Blue Figure 5.3 - a, b, and c Measured pixel distributions of CCDC patch D2 (red), verifying that the distribution of digital counts for a patch is approximately Gaussian.

Having determined the noise characteristics of the camera and approximating the conversion from signal level to digital counts, the simulated values were generated in the following manner. First, the gray signal and the signal of the sample patches were generated. The value, offset and time that produces no over exposure were applied.

This approximates the average, uncorrected signals that would be generated by the camera. These signals were then normalized and multiplied by the illumination

correction factors corresponding to each average pixel value. The pixels were then replicated and used with the Matlab imnoise() function to apply Gaussian noise to the pixels according to the functional relationship previously calculated. The pixels were then unnormalized and corrected with the simulated gray card. The entire process is outlined in the flowchart shown in figure 5.4.

5-12

Generate Target and Gray Signal

Apply k, t, and O

Normalize Signal - Apply Illumination Correction

Replicate and apply imnoise()

Quantize and correct with gray card

Figure 5.4 - Camera model flowchart outlining the pixel simulation process.

Model Results: Figure 5.5 a, b, and c shows the results of generating digital counts at 1/15th of a second with the full model against the actual measured values that were obtained while imaging at the National Gallery of Art, Washington D.C. for each channel.

5-13

x 10 3.5

y=.98x+3e2

3
Average DC Simulated

2.5 2 1.5 1 0.5 0 0

0.5

1.5 2 2.5 Average DC Measured

3.5 x 10
4

a. Red
x 10 3.5
y=1.1x2.6e2
4

3
Average DC Simulated

2.5 2 1.5 1 0.5 0 0

0.5

1.5 2 2.5 Average DC Measured

3.5 x 10
4

b. Green

5-14

x 10 3.5

y=1.1x2.2e2

3
Average DC Simulated

2.5 2 1.5 1 0.5 0 0

0.5

1.5 2 2.5 Average DC Measured

3.5 x 10
4

c. Blue Figure 5.5 - a, b, and c - Average measured digital counts vs. simulated digital counts of the CCDC after being simulated with the model including flat fielding for each channel.

The figure shows that there is a relatively good correlation between the measured and predicted values. The slopes of the fit lines indicate that the measured and simulated mean digital count are very close to unity, meaning that the measured and predicted values are almost the same. While the average value for a patch was not exactly

predicted, this was expected since it is a value generated from a fit line based on average measured values. The error was computed between the measured and actual average values by equation 5.4 over all 240 samples and was found to be 2% in the red, 6% in the green, and 5% in the blue. dm , dm , ds, *100 (5.4)

5-15

As an example, Figure 5.6 a, b, and c show the cumulative distribution plot for the normalized values of patch J10 in each channel, with x representing the normalized digital count.
1 0.9 0.8 0.7 0.6
F(x)

0.5 0.4 0.3 0.2 0.1 0 0.245 0.25 0.255 0.26 x 0.265 0.27 0.275 0.28

a. Red
1 0.9 0.8 0.7 0.6
F(x)

0.5 0.4 0.3 0.2 0.1 0 0.18 0.185 0.19 0.195 0.2 x 0.205 0.21 0.215 0.22 0.225

b. Green

5-16

1 0.9 0.8 0.7 0.6

F(x)

0.5 0.4 0.3 0.2 0.1 0 0.15 0.16 0.17 0.18 x 0.19 0.2 0.21

c. Blue Figure 5.6 - a, b and c - Cumulative distribution plots of the measured (black dashed) and simulated (colored solid) digital counts (x), verifying that the pixels are distributed with an approximate Gaussian density.

These figures show that the Gaussian variance distribution among the simulated pixels holds, but is slightly altered by the inability to exactly predict the average measured value, as shown in the green and blue channel.

Conclusions: While the model cannot exactly predict the average values, it can make a reasonable approximation. As mentioned earlier, future simulations will include the use of an IR cutoff filter that will allow us to capture information corresponding to the near IR region of the spectrum. Also, an effective method of simulating pixels with noise that approximates how the camera responds and introduces differences due to illumination has been developed. It is important to note that there are differences between the

5-17

measured and simulated pixels, but they are most likely due to sources that cannot be controlled or are beyond the scope of this research, such as noise caused by electronics, cross talk between channels, and other various sources.

5-18

Chapter 6 - Filter Selection for Spectral Estimation Incorporating a Noise Model:

Purpose: In Chapter Four, a process for selecting filters using a noiseless camera model was discussed. While this model can be used as a guide, it can lead to selections that do not make intuitive sense. This chapter is concerned with performing a simulation using the camera model described in Chapter Five to incorporate noise and then to make a filter selection based on the simulation results.

Theory of Spectral Reconstruction using a direct pseudo-inverse transformation: Square areas measuring 31 x 31 pixels were simulated with the camera model for each of three channels. This implies that a direct pseudo inverse transform built from pixel data can be used for the characterization target in the a priori analysis, as mentioned in Chapter Three. The simulated pixels of the characterization target were created with the camera model and arranged into a (b x q x h) matrix, Dc, where b is the number of channels and q are the number of uniform samples from the characterization target, and h is the number of pixels from each individual sample. For example, the Esser and blues target consists of 3 x 320 x 31 x 31 or 951,360 pixels. A corresponding (n x q x h) reflectance matrix, Rc, was formed where n is the number of wavelength samples corresponding to each of the pixels. The relationship between the individual patch reflectances and digital counts was formed by Eq. 6.1: T=R c pinv(Dc ) (6.1)

6-1

where T represents the (n x b) dimensional transformation matrix that converts digital counts to spectral reflectance. The reflectance of the verification target Rv at each pixel was then estimated by equation 6.2: Rv Experimental: This experiment followed the same general procedure as the noiseless case, except now the camera model including noise was used to simulate the digital counts of the camera. The noise characteristics determined from images taken at the National Gallery of Art, Washington D.C., were used with the spectral sensitivities of the Balzers filter and TerraPix camera, shown respectively in figures 4.1 and 4.2. The simulated exposure time was 1/15th of a second, which was the longest exposure time that the model did not predict overexposure in the unfiltered simulation. All filtered simulations were also timed at 1/15th of a second. Both spectral and colorimetric errors were Tpinv(D v ) (6.2)

computed, and various curves were analyzed to select a list of 25 filter combinations representing various degrees of performance. The filter combinations evaluated with the camera model were the top 1,351 originally chosen in the noiseless case, making the assumption that filters not performing well in the noiseless case can not perform well when noise is introduced. This was done to reduce processing time and the amount of information to sort.

6-2

Results and Discussion: Figures 6.1 and 6.2 show histogram plots of the average percentage RMS spectral error and average CIEDE2000 under illuminant D65 calculated for the 1931 two degree observer for each of the 1,351 filter combinations.
160 140 120 100

Count

80 60 40 20 0 3

5 6 7 Average % RMS Spectral Error

Figure 6.1 - Average % RMS spectral error histogram calculated using the top 1,351 filter combinations from the noiseless case.

6-3

200 180 160 140 120

Count

100 80 60 40 20 0 0 2 4 6 8 Average CIEDE2000 10 12

Figure 6.2 - Average CIEDE2000 histogram calculated with the noise model and top 1,351 filter combinations from the noiseless case.

These plots show that a reasonable number of filters outperform the entire group by a significant difference in performance. An initial selection of filters was made by

choosing threshold levels and selecting from filter combinations that fall below a specific threshold. Figures 6.3 and 6.4 show the plots of the average percentage RMS spectral error and average CIEDE2000 from the case with noise versus the case without noise to look for any correlations. From the figures, it is apparent that there are none. This implies that filters can be selected with reasonable threshold levels and looking at the properties of those that fall within the threshold criteria.

6-4

11 10 9 8
Noise

7 6 5 4 3 1.8

1.9

2.0

2.1 2.2 No Noise

2.3

2.4

2.5

Figure 6.3 - Average % RMS spectral error of the noise case vs. the noiseless case plotted to look for correlations. See text for explanation of color coding.

6-5

11 10 9 8

Noise

7 6 5 4 3 2 0.4

0.6

0.8

1 1.2 No Noise

1.4

1.6

1.8

Figure 6.4 - Average CIEDE2000 from the noise case vs. the noiseless case. See text for explanation of color coding.

Several different combinations of filters were selected based on the results of the simulation with noise model. They were grouped by selecting different threshold values for different metrics and observing where they fell on the plots shown in Figures 6.3 and 6.4. This further emphasizes that many factors need to be taken into account when making filter selections. The first group, denoted by magenta circles, was seen to have performed well in both cases and clustered together on both metric plots. The second group denoted by red circles were chosen because they appeared to perform well in the noise case but comparatively worse in the noiseless case. The third group, denoted by cyan circles, was chosen based on comparatively low average CIEDE2000 in both cases, 6-6

but shown to have a higher RMS spectral error. The final set, denoted with green circles, were outliers chosen so that properties of filters that performed poorly in both cases could be examined or to show how the addition of noise eliminates filters that appear to perform well in the noiseless case. From these plots, it can also be shown that a small average spectral RMS does not necessarily mean that there will be a small CIEDE2000 and vice versa. Table 6.1 shows the selection criteria used for the first three groups, with NA meaning that the criterion was not used. The outliers were individually chosen to illustrate the case where a filter combination demonstrated good performance for both metrics in the simulation without noise but poor performance in the simulation with noise and the case where poor performance was shown for both metrics in both simulations.

Table 6.1 - Selection criteria used to define magenta, red, and cyan groups representing different types of performance.

Group 1. Magenta 2. Red 3. Cyan

With Noise % Spectral CIEDE2000 RMS error (D65/2 deg.) <4 <2.5 <4 <3 NA <3

Without Noise % Spectral CIEDE2000 RMS error (D65/2deg.) <2 <.7 >2.4 >1 NA <.5

# of Combinations 6 8 7

Table 6.2 shows the final list of filters chosen for evaluation and comparison, which was based on both the simulation and potential availability and table 6.3 shows the integrated camera signals. The simulation with noise shows several interesting results and gives some general insight as to where we might expect error to occur in actual imaging. The results also give an indication to the general properties of the best

performing filters and filter combinations.

6-7

Table 6.2 - Selected Filter Combinations sorted by average % RMS spectral error. C=cyan, M=magenta, R=red, G=green see text for definitions.

Average % RMS spectral Std. Average Std. Color Set# F1# Filter 1 F2# Filter 2 error Max Min Dev. CIEDE2000 Max Min Dev code M 771 51 WR-55 SPEC TRANS GREEN 106 NF 3.61 10.46 1.51 1.66 2.49 7.19 0.25 1.6 R 1195 81 WR-80D SPEC TRANS BLUE 101 WR-90 SPEC TRANS DK GRAY AMBER 3.65 10.45 1.44 1.77 2.94 9.39 0.11 1.92 R 1308 89 WR-82B SPEC TRANS BLUISH 101 WR-90 SPEC TRANS DK GRAY AMBER 3.67 10.35 1.48 1.76 2.96 9.5 0.19 1.92 R 1320 90 WR-82C SPEC TRANS BLUISH 101 WR-90 SPEC TRANS DK GRAY AMBER 3.69 10.27 1.48 1.74 2.87 9.27 0.06 1.86 M 760 51 WR-55 SPEC TRANS GREEN 87 WR-82 SPEC TRANS BLUISH 3.73 10.37 1.65 1.63 2.42 6.74 0.33 1.47 M 754 51 WR-55 SPEC TRANS GREEN 81 WR-80D SPEC TRANS BLUE 3.76 10.23 1.74 1.62 2.33 6.6 0.36 1.41 M 761 51 WR-55 SPEC TRANS GREEN 88 WR-82A SPEC TRANS BLUISH 3.77 10.31 1.69 1.62 2.35 6.61 0.34 1.42 C 796 52 WR-56 SPEC TRANS LIGHT GREEN 106 NF 3.78 10.34 1.54 1.82 2.97 8.32 0.56 1.89 R 1063 75 WR-78B SPEC TRANS BLUISH 101 WR-90 SPEC TRANS DK GRAY AMBER 3.79 10.24 1.64 1.71 2.85 8.96 0.28 1.85 C 597 35 WR-40 SPEC TRANS LIGHT GREEN 106 NF 3.8 10.47 1.78 1.68 2.87 7.41 0.25 1.99 M 762 51 WR-55 SPEC TRANS GREEN 89 WR-82B SPEC TRANS BLUISH 3.8 10.16 1.77 1.6 2.28 6.24 0.26 1.34 R 1178 80 WR-80C SPEC TRANS BLUE 101 WR-90 SPEC TRANS DK GRAY AMBER 3.81 10.2 1.67 1.69 2.84 8.64 0.26 1.83 M 749 51 WR-55 SPEC TRANS GREEN 76 WR-78C SPEC TRANS BLUISH 3.82 10.29 1.73 1.61 2.35 6.47 0.28 1.42 C 895 58 WR-60 SPEC TRANS GREEN 106 NF 3.84 10.99 1.64 1.8 2.95 10.59 0.36 2.21 M 763 51 WR-55 SPEC TRANS GREEN 90 WR-82C SPEC TRANS BLUISH 3.85 10.1 1.85 1.59 2.25 6.09 0.32 1.29 C 844 56 WR-59 SPEC TRANS LIGHT GREEN 106 NF 3.85 10.55 1.7 1.75 2.97 7.95 0.43 2.08 C 871 57 WR-59A SPEC TRANS LIGHT GREEN 106 NF 3.89 10.52 1.66 1.78 3.06 8.05 0.59 2.08 R 1156 79 WR-80B SPEC TRANS BLUE 101 WR-90 SPEC TRANS DK GRAY AMBER 3.98 10.13 1.86 1.63 2.78 8.07 0.25 1.73 M 748 51 WR-55 SPEC TRANS GREEN 75 WR-78B SPEC TRANS BLUISH 3.98 10.07 1.97 1.6 2.22 6.11 0.28 1.26 C 635 37 WR-44A SPEC TRANS LT BLUE-GREEN 106 NF 4.16 13.04 1.34 2.53 3.92 15.92 0.69 3.01 C 552 32 WR-38 SPEC TRANS LIGHT BLUE 58 WR-60 SPEC TRANS GREEN 4.55 10.35 2.39 1.72 2.03 5.82 0.2 1.28 G 121 3 WR-2C SPEC TRANS PALE YELLOW 106 NF 6.7 15.64 2.99 3.31 7.17 18.91 1.94 4.07 G 155 5 WR-3 SPEC TRANS LIGHT YELLOW 8 WR-6 SPEC TRANS LIGHT YELLOW 8.81 27.44 2.75 6.31 10.9 27.68 1.16 6.97 G 960 72 WR-78 SPEC TRANS BLUISH 77 WR-79 SPEC TRANS LIGHT BLUE 9.85 28.61 4.92 5.3 7.86 23 0.8 4.2 G 572 33 WR-38A SPEC TRANS BLUE 68 WR-75 SPEC TRANS DK BLUE GREEN 10.21 18.6 5.23 3.74 6.21 17.7 0.45 4.4

6-8

Table 6.3 Integrated camera signals of the selected filter combinations. Integrated Camera Signals Filter 1 WR-55 GREEN WR-80D BLUE WR-82B BLUISH WR-82C BLUISH WR-55 GREEN WR-55 GREEN WR-55 GREEN WR-56 LIGHT GREEN WR-78B BLUISH WR-40 LIGHT GREEN WR-55 GREEN WR-80C BLUE WR-55 GREEN WR-60 GREEN WR-55 GREEN WR-59 LIGHT GREEN WR-59A LIGHT GREEN WR-80B SPEC TRANS BLUE WR-55 GREEN WR-44A LT BLUE-GREEN WR-38 LIGHT BLUE WR-2C PALE YELLOW WR-3 LIGHT YELLOW WR-78 BLUISH WR-38A BLUE Filter 2 NF WR-90 DK GRAY AMBER WR-90 DK GRAY AMBER WR-90 DK GRAY AMBER WR-82 BLUISH WR-80D BLUE WR-82A BLUISH NF WR-90 DK GRAY AMBER NF WR-82B BLUISH WR-90 DK GRAY AMBER WR-78C BLUISH NF WR-82C BLUISH NF NF WR-90 DK GRAY AMBER WR-78B BLUISH NF WR-60 GREEN NF WR-6 LIGHT YELLOW WR-79 LIGHT BLUE WR-75 DK BLUE GREEN R1 3.65 11.08 10.17 8.51 3.65 3.65 3.65 5.55 7.15 3.18 3.65 6.42 3.65 1.49 3.65 3.17 3.84 4.31 3.65 0.59 2.44 16.20 16.08 0.98 0.53 G1 5.90 9.82 9.33 8.31 5.90 5.90 5.90 7.83 6.90 5.24 5.90 6.14 5.90 4.03 5.90 6.11 6.81 4.38 5.90 2.00 5.34 13.32 13.07 1.38 2.13 B1 2.04 8.76 8.27 7.68 2.04 2.04 2.04 2.42 6.86 2.65 2.04 6.61 2.04 1.38 2.04 2.22 2.49 5.62 2.04 3.29 6.06 10.12 6.68 2.70 3.91 R2 17.83 3.76 3.76 3.76 13.86 11.08 11.94 17.83 3.76 17.83 10.17 3.76 11.59 17.83 8.51 17.83 17.83 3.76 7.15 17.83 1.49 17.83 16.32 1.15 0.05 G2 14.75 2.05 2.05 2.05 11.82 9.82 10.54 14.75 2.05 14.75 9.33 2.05 9.99 14.75 8.31 14.75 14.75 2.05 6.90 14.75 4.03 14.75 13.25 1.67 0.20 B2 11.73 0.94 0.94 0.94 9.66 8.76 8.95 11.73 0.94 11.73 8.27 0.94 8.54 11.73 7.68 11.73 11.73 0.94 6.86 11.73 1.38 11.73 6.95 2.59 0.47

From Table 6.2, it can be seen that filters with lower average RMS spectral error do not necessarily have the lowest CIEDE2000. This leads to the conclusion that some filters not only perform better than others overall, but they perform differently in different regions of the spectrum. For example, the combination of the channels generated with the Wratten 55 filter and unfiltered case appear to generate the lowest RMS error, but the combination of the Wratten 38 and Wratten 60 appear to have the best colorimetric results. Table 6.3 shows that while integrated signal is an important factor, it is not the 6-9

sole determining factor. Integrated signal was considered in the noiseless case because weak signals, in reality, would give information with a poor signal to noise ratio. It was not as important a factor in the noise simulation because signals were simulated with noise qualities added in, making the simulation more realistic. The group formed from filter combinations that provided the best overall performance in both RMS and CIEDE2000 were generally composed of green filters that transmitted light in the long red region and no filter or combined with a light blue filter. Table 6.2 shows that the filter combination with the best performance was the Wratten 55 and unfiltered combination. Figures 6.5 and 6.6 show the spectral transmittance of this combination as well as the altered spectral sensitivities of the camera. Figure 6.7 shows a normalized plot of each camera signal, showing that the peaks are distributed across the spectrum.

6-10

1 0.9 0.8
Transmittance Factor
WR55 SPEC TRANS GREEN NF

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.5 Spectral Transmittance of the Wratten 55.

6-11

0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 400

Spectral Sensitivity (Quantum Efficiency)

450

500

550

600

650

700

Wavelength (nm)
Figure 6.6 Camera sensitivities resulting from the use of the Wratten 55 and unfiltered combination. The dashed lines represent the resulting filtered sensitivities.

6-12

Red 1

Green 1

Blue 1

1 0.8 0.6 0.4 0.2 0 400

730

400

730

400

730

Red 2

Green 2

Blue 2

1 0.8 0.6 0.4 0.2 0 400

730

400

730

400

730

Figure 6.7 Normalized camera signal plots resulting from the Wratten 55 and unfiltered combination. The x axis represents wavelength (nm) and y axis represents normalized spectral sensitivity.

A comparison of the spectral difference plots in Figures 6.8 and 6.9 from the noiseless and noise case show the effects that the addition of noise can have on the system, even in the best overall case.

6-13

0.5 0.4 0.3 0.2

Reflectance Factor

0.1 0 0.1 0.2 0.3 0.4 0.5 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.8 Difference plot of estimated spectra from measured spectra of the Gamblin verification target resulting from the use of data simulated with the Wratten 55 and unfiltered camera sensitivities in the noiseless case.

6-14

0.5 0.4 0.3 0.2

Reflectance Factor

0.1 0 0.1 0.2 0.3 0.4 0.5 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.9 - Difference plot of estimated spectra from measured spectra of the Gamblin verification target resulting from the use of data simulated with the Wratten 55 and unfiltered camera sensitivities in the noise case.

While both figures show that the error tends to occur in the same general regions of the spectrum, it becomes much more noticeable in the noise case. Figure 6.10 shows the estimated spectra for selected samples of the Gamblin target when this filter was used, with the red, dashed lines representing the estimate with the noise case, the green, solid, x marked line represents the noiseless case, and the blue, solid line represents the measured reflectance spectra. It gives an indication as to the types of pigments that the system will have difficulty in accurately estimating. It shows that the estimate will be particularly erroneous with the cobalt violet, which is to be expected because this pigment is not represented in the characterization data and has the most complex

6-15

spectrum in the pigment set. It also shows that this filter can be expected to perform very similarly to the noiseless case, meaning that this filter creates a combination of six channels that are accurate and is predicted to be robust to noise.

6-16

Cadmium Red Light 1

Quinacridone Red

0.5

Cobalt violet 1

Ultramarine blue

0.5

0 400

730

400

730

Figure 6.10 Individual reflectance spectra for both cases using the Wratten 55 and unfiltered camera sensitivities compared with the measured data. The x axis represents wavelength (nm) and y axis represents reflectance factor. See text for color and line codes.

6-17

The second group selected was investigated because they performed well in the noise case but comparatively worse in the noiseless case. The combinations from this set are composed of a blue filter and the Wratten 90 dark gray / amber monochrome viewing filter. For example, from this set the best performer based on spectral RMS error was the combination of the Wratten 80D and 90 filters, whose spectral transmittance and resulting camera sensitivities can be seen in figures 6.11 and 6.12, respectively. Figure 6.13 shows the normalized spectral sensitivity from each channel resulting from this combination.

1 0.9 0.8
WR80D SPEC TRANS BLUE WR90 SPEC TRANS DK GRAY AMBER

Transmittance Factor

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.11 Spectral transmittance of the Wratten 80D and 90 filters.

6-18

0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 400

Spectral Sensitivity (Quantum Efficiency)

450

500

550

600

650

700

Wavelength (nm)
Figure 6.12 Camera sensitivities with the Wratten 80D and 90 filters applied. The dashed lines represent sensitivities resulting from the use of the Wratten 90.

6-19

Red 1

Green 1

Blue 1

1 0.8 0.6 0.4 0.2 0 400

730

400

730

400

730

Red 2

Green 2

Blue 2

1 0.8 0.6 0.4 0.2 0 400

730

400

730

400

730

Figure 6.13 Normalized camera sensitivities resulting from the Wratten 80D and 90 filters. The x axis represents wavelength (nm) and y axis represents normalized spectral sensitivity.

Figure 6.14 shows the same selected estimates for the verification (Gamblin) data. It shows that this particular filter combination also has problems with the same pigments, such as cobalt violet. It also indicates that while it performs very similarly to the first case, it is slightly worse, as shown in the average metric values from table 6.1.

6-20

Cadmium Red Light 1

Quinacridone Red

0.5

Cobalt violet 1

Ultramarine blue

0.5

0 400

730

400

730

Figure 6.14 - Selected reflectance spectra for both cases using the Wratten 80D and 90 filtered camera sensitivities compared with the measured data. The x axis represents wavelength (nm) and y axis represents reflectance factor. See text for color and line codes.

6-21

The third group was chosen for evaluation because they have very low CIEDE2000, but a relatively higher spectral RMS than the first group. This group includes the filter combination composed of the Wratten 38 and Wratten 60, a blue and green filter, respectively, with their transmittances and resulting camera sensitivities shown respectively in figures 6.15 and 6.16, and the normalized camera channel sensitivities shown in figure 6.17.

1 0.9 0.8
WR38 SPEC TRANS LIGHT BLUE WR60 SPEC TRANS GREEN

Transmittance Factor

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.15 - Spectral transmittance of the Wratten 38 and 60 filters.

6-22

0.16 0.14 0.12

Spectral Sensitivity

0.1 0.08 0.06 0.04 0.02 0 400

450

500

550

600

650

700

Wavelength (nm)
Figure 6.16 - Camera sensitivities with the Wratten 38 and 60 filters applied. The dashed lines represent the sensitivities resulting from the use of the Wratten 60.

6-23

Red 1

Green 1

Blue 1

1 0.8 0.6 0.4 0.2 0 400

730

400

730

400

730

Red 2

Green 2

Blue 2

1 0.8 0.6 0.4 0.2 0 400

730

400

730

400

730

Figure 6.17 Normalized camera sensitivities resulting from the Wratten 38 and 60 filters. The x axis represents wavelength (nm) and y axis represents normalized spectral sensitivity.

The graphs show that there is significant transmittance in the blue and green region of the spectrum, but very little in the red and long red. Figure 6.18 shows a difference plot predicted by the model. Compared to the filters which provide more transmittance in the tail end, it is clear that the model predicts more error in the longer wavelength regions of the spectrum, but less in the shorter. This may be one reason that the model predicts a better CIEDE2000, but worse RMS spectral error.

6-24

0.5 0.4 0.3 0.2

Reflectance Factor

0.1 0 0.1 0.2 0.3 0.4 0.5 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.18 - Difference plot of the measured and estimated Gamblin spectra using the WR-38 and WR-60 filter combination in the noise case.

The final group composed of the outliers shows how selections from the noiseless case may no longer be valid with the addition of noise. For example, the combination of the Wratten 2C and unfiltered image data (camera sensitivity plot in figure 6.19) that was predicted to perform with approximately 1% average spectral RMS error was predicted to have roughly 6% average spectral RMS error with respect to the verification target. Figure 6.20 shows the selected reflectance spectra estimates resulting from this filter combination.

6-25

0.16 0.14 0.12

Spectral Sensitivity

0.1 0.08 0.06 0.04 0.02 0 400

450

500

550

600

650

700

Wavelength (nm)
Figure 6.19 Camera sensitivities resulting from the Wratten 2C and unfiltered combination. The dashed line represent the unfiltered sensitivities.

6-26

Cadmium Red Light 1

Quinacridone Red

0.5

Cobalt violet 1

Ultramarine blue

0.5

0 400

730

400

730

Figure 6.20 - Selected reflectance spectra for both cases using the Wratten 2C and unfiltered camera sensitivities compared with the measured data. The x axis represents wavelength (nm) and y axis represents reflectance factor. See text for color and line codes.

6-27

The red lines lack of fit across the individual samples shows how the addition of noise can impact the system. The impact of noise is also shown by the spectral difference plots in figures 6.21 and 6.22. The plot resulting from the noiseless case shows relatively little change across the spectrum, meaning that the noiseless model for this filter combination is able to estimate the individual verification samples accurately. The

second plot resulting from the addition of noise shows that the filter set is actually not a good choice as the estimates become quite different from the verification data.

0.5 0.4 0.3 0.2

Reflectance Factor

0.1 0 0.1 0.2 0.3 0.4 0.5 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.21 - Difference plot of estimated and measured spectra resulting from the noiseless simulation using the Wratten 2C and unfiltered data.

6-28

0.5 0.4 0.3 0.2

Reflectance Factor

0.1 0 0.1 0.2 0.3 0.4 0.5 400 450 500 550 600 Wavelength (nm) 650 700

Figure 6.22 - Difference plot of estimated and measured spectra resulting from the simulation including noise using the Wratten 2C and unfiltered data.

Conclusions: Modeling the noise in the camera system has shown that noise can have a significant impact on the accuracy of spectral reflectance estimation. The 1,351 top combinations from the noiseless case were used in a noise simulation, and the top 25 were chosen based on an evaluation of not only RMS spectral error, but CIEDE2000 as well. In order to avoid the problems typically associated with using singular values to describe a system, choices were made by also taking into account the resulting spectral sensitivity of the camera as well as a visual evaluation of the individual pigment spectra and resulting spectral estimate difference plots.

6-29

Chapter 7 - Imaging and Data Comparison:

Purpose: This chapter describes the process used to image the target with the filter combinations chosen based on the results of the noise simulation. First, the scene setup and exposure metering are described. Then, the experimental data is compared to the simulated data. Differences between the model and the experiment as well as physical limitations of the system are identified and discussed as possible sources of error.

Imaging: The conditions in the imaging laboratory were assumed to be similar to those found in a museum studio or other environment that would perform copy work. The camera was placed approximately 210 cm from the easel holding the target to the CCD plane. The lights were placed 210 cm away from the target at approximately 45 degrees to the cameras optical axis and aimed at the center of the target to provide even, diffuse illumination (London 1998). The area surrounding the scene was covered in black drop cloth and black cards were used to reduce optical flare, which has been shown to have an effect on spectral reflectance estimates (Day, D.C. 2003). In this phase of the

experiment, the IR cutoff filter was mounted to the camera using a custom built apparatus, shown in figure 7.1. It allowed the filter to be angled to approximately 30 degrees to minimize any error that may be introduced by inter-reflections, which has also been shown to have an effect on spectral estimates (Knig 1998). A test image was taken to ensure that no IR radiation was affecting the image by checking for any obvious blooming in the image. Figures 7.2 and 7.3 show the scene setup and dimensions. 7-1

Figure 7.1 - IR Cutoff Filter mounted on the camera.

Figure 7.2 Scene setup at MCSL.

7-2

Figure 7.3 Scene dimensions and geometry.

Of the 25 filter combinations selected from the simulation, 18 were readily available in the MCSL for testing. Table 7.1 shows the filter combinations that were tested. For each filter set, two sets of images were taken, both illuminated with 541 cd/m2, which was measured from light incident on a pressed halon disk with the PR-650 Spectroradiometer. All images were taken with the aperture set at f/8 so that errors between depth of field and exposure time would be minimized. The first set was taken at a constant exposure time of 1/30th of a second. The images were spatially corrected with an image of the unfiltered gray card for comparison to the noise simulation. This set will be referred to as the unoptimized set for the duration of this thesis, because the exposure times were not optimized to take advantage of the cameras dynamic range. In practice, this would be done if an individual was trying not to alter camera settings, but to simply change the filters in the system, a situation which would occur when imaging an in vivo subject, such as a portrait of a human face.

7-3

Table 7.1 - Tested Kodak Wratten absorption gelatin filter combinations (NF denotes no filter was used). Set # Filter 1 Filter 2 771 1308 1320 760 761 1063 597 762 1178 749 895 763 844 871 1156 748 635 121 55 82B 82C 55 55 78B 40 55 80C 55 60 55 59 59A 80B 55 44A 2C NF 90 90 82 82A 90 NF 82B 90 78C NF 82C NF NF 90 78B NF NF

The second set of images were taken so that the dynamic range of the images were maximized without clipping to improve performance. The time for each filter was determined by evaluating the histogram of the raw image and choosing a time that gave very few to no digital counts of 4095. In some cases this could not be avoided because exposure can only be controlled in whole stop increments. Once the exposure time for a given filter was chosen, they were taken in an order that would minimize contact with the camera and reduce error that would result from registration problems. Table 7.2 gives the exposure times used and the filters associated with them and Table 7.3 shows the maximum raw digital count for the optimized exposure and one stop greater exposure. Figure 7.4 a and b show examples of a properly exposed and overexposed histogram. Note the spike at 4095 signifying overexposure.

7-4

2.5

x 10

2.5

x 10

1.5

1.5

Count

Count
1 0.5 0.5 0 1000 0 1000 2000 DC 3000 4000 5000 0 1000

1000

2000 DC

3000

4000

5000

a. Optimized

b. Optimized + 1 stop

Figure 7.4 - Histogram examples used to help determine optimal exposure time. For these two, the unfiltered image was used. The optimized (a) was taken at 1/30 of a second. One stop greater (b.) taken at 1/15th of a second yields over exposure.

Table 7.2 Optimized filter times for Kodak Wratten filters.

Exposure time (seconds) Kodak Wratten Filter Number

1/30s NF 2C 6

1/15s 78C 82 82A 82B 82C

1/8s 40 55 59 59A 78B 80B 80C 90

1/4s 60

1/2s 38A 44A 79

7-5

Table 7.3 Maximum digital counts between optimized time and optimized time + 1 stop. Filter NF 2C 6 38A 40 44A 55 59 59A 60 78B 78C 79 80B 80C 82 82A 82B 82C 90 Optimized Time 2718 2688 2488 3066 2469 2398 2812 2250 2942 3880 4095 3567 2565 2425 3799 4095 3485 3122 2617 2460 Optimized time + 1 stop 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095 4095

Each target image in the second set had a corresponding gray card image taken with it for spatial correction because the exposure varied among the filters and the assumption that the unfiltered exposure had the most dynamic range was no longer valid. For both sets, the spatially corrected pixel values were calculated using equation 7.1: Dc,f ( Dr,f Dgray,f Ddark, t Ddark,t ) *(Dgray,f Ddark,t ) (7.1)

In this equation, f represents the filter and t represents the time for the filter used. Otherwise, the terms are the same as those in equation 5.1. The pixels corresponding to the target samples were then extracted by using masks for each target. The pixels were then arranged into a six row matrix, with each

7-6

row representing a channel of data created with the filters. This organized the data for comparison to the simulation. Figure 7.5 shows a flow chart of imaging pipeline, which was done for each of the filter combinations.

Exposure Meetering Exposure Time(s) Target Images Dark Current Images Gray Card images

Corrected Digital Signals

Imaging

Figure 7.5 - Imaging pipeline used for all filter combinations.

Data Comparison and Error Source Analysis: The experimental and model data was compared to help determine the reliability of the model and whether or not the experimental data should be used. Limitations of the model are discussed and the differences between the model and experimental data are identified as potential sources of error.

Model limitations: The unfiltered experimental and simulated data were compared to determine which components in the model may have caused some of the discrepancies noticed between the theory and the experimental data. Figure 7.6 a, b and c shows the plots of the average measured digital counts versus the average simulated digital counts in each

7-7

channel for the CCDC. It shows that the model is over predicting exposure in all three channels. This was caused by using data taken from images made while using an IR cutoff filter that had a transmittance range of 400 to 700 nm to create the model parameters. By using the Balzers IR cutoff filter, sensitivity was extended to 750 nm. By increasing the sensitivity, the exposure constants would have to decrease for the model to match the experimental digital counts.

7-8

2000 1800 1600

Average Experimental DC
y=.6996x186.9970, R2=.999

1400 1200 1000 800 600 400 200 0 0 500 1000 1500 2000 2500 Average Simulated DC 3000 3500

a. Red
700

600
Average Experimental DC

500
y=.6996x186.9970: R2: .9989

400

300

200

100

0 0

200

400

600 800 Average Simulated DC

1000

1200

b. Green

7-9

700

600
Average Experimental DC

500
y=.6647x168.2100: R2: .9965

400

300

200

100

0 0

200

400

600 800 Average Simulated DC

1000

1200

c. Blue Figure 7.6 - a, b, and c - Average measured digital counts versus average simulated digital counts for the unfiltered image of the CCDC.

The offset, which was first mentioned in Chapter Five, is a result of not having dark image data available for characterization. When the model parameters were

determined, images processed with the TerraPix software were used. At that time, it was thought that the software automatically dealt with the dark image data. It was later found that this was only accurate for exposure times of 1/250th of a second or less. The shortest exposure used for characterization was timed at 1/15th of a second at f/11. It was decided that the shift was a linear factor, and the offset was overlooked. The correlation between the average simulated and experimental data was determined by calculating the R2 value for each channel to decide if the data from the experiment were useable. The results indicate a very good correlation, with all three channels having an R2 value of more than .99. Similar results were found for the filtered

7-10

simulated and experimental data, as shown in Table 7.4, which gives the offset, slope, and R2 value of the simulated data and the unoptimized experimental data of the CCDC. These data can be directly compared because the exposure for the imaging and simulation were designed to be equivalent. The major error was found to be in the slope, which should ideally be one. This was not as important a factor as it is related to a scalar value that is resolved in the transformation matrix. Based on this information, the experimental data were considered to be usable.
Table 7.4 Comparison of average experimental and model values for the CCDC Red Offset NF 2C 6 38A 40 44A 55 59 59A 60 78B 78C 79 80B 80C 82 82A 82B 82C 90 -187 -196 -187 -176 -153 -124 -185 -118 -213 -217 -176 -192 -145 -149 -175 -181 -190 -190 -189 -135 Slope 0.70 0.73 0.71 0.70 0.63 0.50 0.75 0.47 0.84 0.86 0.69 0.73 0.57 0.59 0.69 0.69 0.72 0.73 0.72 0.53 R
2

Green Offset -174 -183 -176 -179 -194 -158 -209 -164 -196 -235 -172 -183 -158 -155 -174 -170 -179 -180 -176 -151 Slope 0.67 0.70 0.68 0.70 0.77 0.63 0.82 0.63 0.76 0.92 0.68 0.71 0.62 0.62 0.69 0.66 0.70 0.70 0.68 0.59 R
2

Blue Offset -168 -178 -177 -146 -161 -126 -187 -141 -191 -200 -163 -177 -144 -146 -162 -165 -175 -174 -169 -156 Slope 0.66 0.69 0.70 0.58 0.65 0.50 0.75 0.56 0.76 0.80 0.65 0.70 0.57 0.59 0.65 0.65 0.69 0.69 0.66 0.62 R2 1.00 1.00 1.00 1.00 0.99 1.00 0.99 0.99 0.99 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 0.99 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99

1.00 1.00 1.00 0.99 0.99 0.96 1.00 0.99 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99

Table 7.5 shows the same comparison for the model and optimized CCDC values. A direct comparison is possible after dividing out the exposure times used for the

7-11

simulation and experimental data, making the assumption that all the remaining exposure constants in the simulation are the same as those in the experiment.
Table 7.5 Comparison of model and experimental data with exposure times divided out. Red Offset NF 2C 6 38A 40 44A 55 59 59A 60 78B 78C 79 80B 80C 82 82A 82B 82C 90 -5884 -6270 -5644 -6493 -6184 -4736 -6991 -4513 -7620 -8118 -5899 -5870 -5805 -5543 -6213 -5722 -5681 -5890 -5949 -5130 Slope 1.46 1.60 1.45 1.74 1.67 1.27 1.89 1.22 2.03 2.17 1.56 1.52 1.56 1.49 1.64 1.48 1.48 1.53 1.54 1.38 R
2

Green Offset -5514 -5929 -5309 -6739 -7485 -5875 -7597 -5875 -7031 -8637 -5876 -5722 -6132 -5701 -6266 -5489 -5488 -5624 -5597 -5786 Slope 1.41 1.55 1.39 1.80 1.98 1.57 2.00 1.55 1.85 2.29 1.55 1.49 1.64 1.52 1.65 1.43 1.44 1.47 1.47 1.56 R
2

Blue Offset -5275 -5706 -5252 -5312 -6146 -4487 -6813 -5030 -6811 -7147 -5518 -5455 -5477 -5284 -5775 -5309 -5304 -5396 -5308 -6175 Slope 1.38 1.52 1.41 1.43 1.65 1.20 1.84 1.35 1.83 1.91 1.47 1.45 1.47 1.42 1.54 1.41 1.41 1.43 1.41 1.67 R2 1.00 1.00 1.00 1.00 0.99 1.00 0.99 0.99 0.99 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 0.99 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00 0.99 0.94 0.99 0.98 0.99 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

The difference in slopes between the data points shows that the optimized experimental data was overexposed. This was expected as exposure time for many of the filters was increased. Another factor which would explain the differences is that the model does not take into account all the properties of the filters, such as the filter factor describing the necessary increase in exposure time, which is usually given by the manufacturer for colored filters. This is more evidence that the model can be improved by increasing its complexity and decomposing the exposure constants to include terms that would account for these properties.

7-12

Figure 7.7 shows plots of the experimental variance versus the simulated variance for the unfiltered CCDC. The first subplot is an overview of all the data points. There is one outlier, which was found to be a scuff mark on the CCDC. The second subplot shows the data without the outlier. Here, the variance was overestimated in the red, almost correct in the green, and under estimated in the blue with respect to the linear fit. The R2 values for these data are particularly low as shown in the plot. The low R2 values indicating a wider spread around the linear fit may be a result of several different factors. Aside from the factors affecting the average values, low R2 values could also be a result of the gray card simulation. In the model, the gray card simulation and illumination correction were based on randomly generated values and assumed that any patch could occupy any point in the target plane. This correction was applied to the simulated gray card and target. The values used to generate the gray card and illumination correction were approximations based on the images taken at the National Gallery of Art, Washington D.C. Any differences here from the actual experimental scene introduced a certain degree of error. Another factor that may have influenced the variance was that the exposure times in the simulated and experimental conditions were different. While they were matched so the exposures would be the same theoretically (f/8 at 1/30th of a second for the experiment, f/11 at 1/15th of a second for the simulation), the difference may have caused problems with the variance. Also, the lighting and setup geometries for the data used to make the model and the experiment were different with respect to distances. Finally, the simulated variance was, again, a very simple linear model that did not simulate any spread around the signal and variance relationship it was based on.

7-13

Experimental Variance

3000 2000 1000 0 0 400

Red Green Blue

100

200

300 400 Simulated Variance

500

600

Experimental Variance

300 200 100 0 0

y=.3776x : Red y=.9549x : Green y= 1.46x : Blue R2: R:.2313 G:.5820 B:.6869

100

200

300 400 Simulated Variance

500

600

Figure 7.7 - Experimental versus simulated variance of the unfiltered CCDC, showing an inability to accurately model variance to match the experiment.

The model also could not account for texture that may be part of the image. Figure 7.8 a, b, and c shows the average measured and simulated values while Figure 7.9 shows the variance for the Esser and blues target. The outliers in Figure 7.9, marked by the red diamonds, are either blue samples which were painted and have texture or samples on the Esser target that have scratches or other abnormalities.

7-14

1800 1600 1400

Average Experimental DC
y=.6778212.73, R2=.9946

1200 1000 800 600 400 200 0 0

500

1000 1500 2000 Average Simulated DC

2500

3000

a. Red
900 800
Average Experimental DC

700 600 500 400 300 200 100 0 200

y=.6517x180.69, R2=.9831

400

600

800 1000 Average Simulated DC

1200

1400

1600

b. Green

7-15

450 400
Average Experimental DC

350 300 250 200 150 100 50 0 200

y=.7053x202.89, R2=.9682

300

400

500 600 700 Average Simulated DC

800

900

1000

c. Blue Figure 7.8 - a, b, and c - Average experimental versus simulated data for the Esser and blues data, showing good linear fit but off by a scalar for the CCDC.

2000 1800 1600

Experimental Variance
Red Green Blue

1400 1200 1000 800 600 400 200 0 0 100 200 300 400 Simulated Variance 500 600

Figure 7.9 - Experimental versus simulated variance for the Esser and blues target showing the models inability to handle texture.

7-16

The plots and experimental results suggest that the camera model and its components are, as a whole, very simplistic. As shown in figures 7.6 and 7.8, the average was approximated fairly well. The slopes of the experimental versus simulated data show that some adjustments are necessary. More importantly, the variance component of the model needs to be improved. The current method is acceptable in that it helped bring the number of filter candidates from 5,565 to 25, before performing actual imaging with selected filters. While this is an acceptable practice for this research, it would be

desirable to have a model accurate enough so that the pool of filter combinations could be reduced even further.

Equipment Limitations: The camera itself had some limitations that may have had an effect on the results of the experiment. The first was found in the difficulty to optimize exposure. The camera exposure was tied to preset intervals in whole stop increments. This means that the exposures must be based on maximizing the most sensitive channel, which in the case of the TerraPix was usually the red. This often led to the camera using approximately half its available dynamic range in the red channel and then less in the other two. It is always a case of having one exposure time for three channels as opposed to one exposure time per channel. Also, the whole stop exposure adjustments make it impossible to find an intermediate value. For example, if an exposure taken at f/8 for 1/15th of a second yields 4095 in the red channel, then f/8 at 1/30th will give approximately half that. There is no way to time the exposure to yield a maximum digital count of 3800 off of a halon

7-17

disk, which is commonly used when a camera that has a computerized shutter control is used, as done in previous experiments (www.art-si.org). A second limitation that may have caused error was that the camera settings were not electronically controlled. The user had to physically change the settings on the camera. This may have caused slight shifts in the image, introducing registration errors. Error may have also been introduced by the Wratten filters. A particular filter may not have had the same spectral transmittance properties across the filter, causing differences in the resulting camera sensitivities. Also, they may have had scratches or other abnormalities that introduced some spatial noise in the system. In fact, most of the filters used for this research were previously used. Finally, the geometry or positioning of the filter may not have been flush against the lens which would introduce geometric errors.

Conclusions: The methods used in imaging the targets to obtain the experimental data have been described. It was shown that the model cannot exactly predict experimental values due to its overall simplicity, but it can be shown that, as was mentioned in Chapter Five, it can be used as a guiding tool in making filter pre-selections for spectral estimation. It was shown that the model particularly has problems with predicting variance in textured samples, and that variance in general is very difficult to model. The limitations of the camera system were also discussed. While most of these limitations seemed fairly insignificant, the system does suffer from an inability to use its full dynamic range as a result of the exposure control being limited to whole stop

7-18

increments. This limitation was further exaggerated by the fact that three channels of information are tied to one exposure.

7-19

Chapter 8 - Filter Combination Analysis

Purpose: The comparison of the experimental data to the model data in Chapter Six showed that the experimental data were usable. While the experimental data were not the same as the model as a result of the models simplicity, they will be used independently of the model. Any type of calculations performed will be strictly related to and optimized for the experimental data. This means that the lack of fit between the slopes, as mentioned in the previous chapter, relate to a scaling factor that generally falls out in the transformation matrix generated by using a pseudo inverse operation, as outlined in Chapter Six. Also, aspects relating to the light sources and targets will be examined. Finally, the sensitivity of the pseudo inverse calculation will be discussed.

Spectral estimation and performance evaluation: After determining that the experimental data were usable and identifying areas of the model and system which may introduce error, the pixels corresponding to target a pseudo inverse matrix was then derived to minimize the error between the reflectance estimations and the measured reflectances, as was done in equations 6.1 and 6.2. This matrix was then used on the Gamblin pigment target for verification. The spectral and colorimetric errors were calculated for illuminant D65 and the CIE 1931 2 degree standard observer. It is important to note that the reflectance estimates were clipped to remain in the range of zero and one to minimize error in colorimetric calculations. For past calculations, the evaluation was performed by finding the average reflectance curves from all the pixels and then calculating the appropriate quality metrics. 8-1

This method does not give an accurate representation of the process at the pixel level. Calculations from this experiment were performed by estimating the reflectance at each pixel, calculating the appropriate quality metric at each pixel, and then separating the data by patch for comparison. Figure 8.1 outlines the processing pipeline used on the

corrected images generated from the imaging discussed in Chapter 7. Figure 8.2 details the method used to calculate the metrics at the pixel level for the evaluation portion of the processing pipeline.

Transformation - Digital Signals to Reflectances Characterization Target Signals Corrected Digital Signals

Verification Target Signals Transformation Matrix Evaluation Characterization Target Measured Reflectances Verification Target Measured Reflectances Verification Target Estimated Reflectances Spectral Estimation

Figure 8.1 Flowchart of the data processing pipeline.

8-2

Estimated Reflectance Spectra at each pixel

Measured Reflectance Spectra associated with pixel (measurement)

Data Tristimulus and CIELAB calculation - D65 2 degree observer Estimated XYZ and L*a*b* Measured XYZ and L*a*b*

RMS spectral error at each pixel

CIEDE2000 at each pixel

Average RMS spectral error at each patch

Average CIEDE 2000 at each patch

Figure 8.2 - Data processing at the evaluation level.

8-3

Results and Discussion: Table 8.1 shows the data used to give an initial indication of filter performance. It gives the average RMS spectral error and CIEDE2000 taken over all the patches of the Gamblin target for both the unoptimized and optimized exposure groups.
Table 8.1 - Average statistics from the Gamblin verification target. Average % Average Average Average RMS % RMS - CIEDE2000 CIEDE2000 Unoptimized Optimized Unoptimized Optimized 6.3 6.8 6.9 6.3 6.3 6.8 6.8 6.2 7.1 6.2 6.4 6.3 6.8 6.3 5.9 6.2 6.4 6.1 6.45 5.2 5.9 5.8 5 5.1 5.7 5.7 5 5.6 5.1 4.7 5.1 4.9 5.2 5.6 5 6.2 6.4 5.40 5.6 5.9 5.9 5.7 5.6 5.7 6.3 5.6 5.9 5.6 5.9 5.6 6 5.9 5.7 5.4 5.8 7.9 5.89 7.3 8.8 8.7 6.6 6.8 8.3 8 6.4 8 6.7 6.1 6.4 6.5 7.1 7.4 6.2 4.6 9 7.16

Filter1 55 82B 82C 55 55 78B 40 55 80C 55 60 55 59 59A 80B 55 44A 2C

Filter2 NF 90 90 82 82A 90 NF 82B 90 78C NF 82C NF NF 90 78B NF NF

Average

This table shows that the RMS spectral error is decreasing by expanding the dynamic range as would be expected, but the average CIEDE2000 is increasing for the optimized case, which is not intuitive. It was expected that as the RMS spectral error improved, the CIEDE2000 would also improve. Table 8.2 shows the same metrics when the transforms were used on the characterization data. This shows that the performance of the

transforms are, on average, behaving as expected, suggesting that there is noise in the

8-4

Gamblin data that the transform is sensitive to, such as the painted texture of the canvas. While the pseudo inverse calculation includes noise to help make the transformation matrix more robust, it still tends to cause problems in the accuracy of the spectral reflectance estimates.
Table 8.2 - Average performance for Esser and Blues characterization data. Average % Average Average Average RMS % RMS CIEDE2000 CIEDE2000 Unoptimized Optimized Unoptimized Optimized 3.4 3.6 3.7 3.4 3.5 3.5 3.3 3.5 3.5 3.5 3.6 3.6 3.4 3.5 3.7 3.5 3.9 4.6 3.59 3.2 2.9 3 3 3 2.8 3.1 3 2.8 3.1 3.1 3.1 3.2 3.3 2.8 3 3 4.6 3.11 4.7 5.1 5.3 4.9 5 5 4.7 5 5 4.9 5 5.2 4.9 5 5.4 5 5.1 5.8 5.06 4.6 3.6 3.6 4 3.9 3 4.4 4 3 4.1 4.6 4 4.5 4.9 2.9 3.6 3.3 5.8 3.99

Filter1 55 82B 82C 55 55 78B 40 55 80C 55 60 55 59 59A 80B 55 44A 2C

Filter2 NF 90 90 82 82A 90 NF 82B 90 78C NF 82C NF NF 90 78B NF NF

Average

The following three filter combinations were evaluated to obtain a more accurate idea of how the system works: the Wratten 55 and unfiltered (NF) combination because it was predicted to have the best performance with respect to verification spectral RMS and only slightly above average CIEDE2000, the Wratten 60 and unfiltered combination because it appeared to perform the best experimentally as it had the best verification

8-5

spectral RMS and below average CIEDE2000, and the 2C and unfiltered combination as an example of poor performance.

Wratten 55 and NF combination The filter combination consisting of the Wratten 55 and unfiltered combination was predicted to perform with the lowest RMS spectral error and a relatively low CIEDE2000, as shown in Table 6.2, and it showed experimentally the inverse RMS and CIEDE2000 relationship with an approximate 1 percent decrease in RMS and a 1.6 unit increase in CIEDE2000, shown in Table 8.1. The individual patches of the Gamblin target were separated into four cases: RMS decreases and CIEDE2000 increases, RMS increases and CIEDE2000 decreases, both metrics increasing and both metrics decreasing. The most important case to evaluate is where the CIEDE2000 increases while the RMS spectral error decreases, as this is counterintuitive. Figure 8.3 shows plots of the spectra that fall into this category. The blue line represents the measured spectra, the red, dashed line represents the estimation from the unoptimized exposures, and the green represents the estimations from the optimized exposure set. Figure 8.4 shows the reflectance difference from the measured with the blue, dashed line representing the unoptimized and the red, solid representing the optimized set. These figures show that the optimized estimation has a better fit than the unoptimized. The difference plots show that error tends to occur in the blue region of the spectrum in the optimized case while the error occurs mostly in the red for the unoptimized case.

8-6

Cadmium Red Light 1

Burnnt Sienna

Cadmium Yellow Light

Cadmium Orange

Quinacridone Red

Cadmium Red Light

Burnnt Umber

0.5

0 Cadmium Yellow Medium 1 Cadmium Orange Quinacridone Red Red medium Burnnt Umber Yellow Medium Black

0.5

0 Red medium 1 Raw Sienna Black Yellow Medium Prussian Blue RawUmber Blank

0.5

0 Raw Sienna 1 Hansa Yellow Medium Raw umber Venetian red Blank Indian Yellow Indian red

0.5

0 Yellow ochre 1 Venetian red Blank Indian Yellow Burnt Sienna Cadmium yellow light Indian red

0.5

0 400 Yellow ochre 1 Earth Yellow 730 400 730 400 730 400 730 400 730

0.5

0 400

730

400

730

Figure 8.3 - Estimated spectra of samples where RMS spectral error decreased and CIEDE2000 increased for the 55 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor.

8-7

Cadmium Red Light 0.2

Burnnt Sienna

Cadmium Yellow Light

Cadmium Orange

Quinacridone Red

Cadmium Red Light

Burnnt Umber

0.2 Cadmium Yellow Medium 0.2 Cadmium Orange Quinacridone Red Red medium Burnnt Umber Yellow Medium Black

0.2 Red medium 0.2 Raw Sienna Black Yellow Medium Prussian Blue RawUmber Blank

0.2 Raw Sienna 0.2 Hansa Yellow Medium Raw umber Venetian red Blank Indian Yellow Indian red

0.2 Yellow ochre 0.2 Venetian red Blank Indian Yellow Burnt Sienna Cadmium yellow light Indian red

0.2 400 Yellow ochre 0.2 Earth Yellow 730 400 730 400 730 400 730 400 730

0.2 400

730

400

730

Figure 8.4 - Difference of estimated reflectance spectra from measured reflectance spectra for both optimized and unoptimized cases for the 55 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference.

8-8

Several factors contributed to the inverse RMS and CIEDE2000 relationship. The first is that the transformation matrix minimizes the RMS spectral error. In the

unoptimized exposure, the matrix was able to create estimates which were accurate in the blue region but much less accurate in the red region. The optimized exposures appear to be estimating the opposite, pushing the overall curve down and causing error in the blue areas of the spectrum. Since CIEDE2000 is calculated between 400 and 700 nm, this measure increases in the optimized exposure for these samples, but the RMS spectral error is decreasing as it takes the entire curve into account. A second factor that should be noted is that CIEDE2000 was designed to include factors related to human perception (Luo 2001). Figure 8.5 shows the estimated spectra of blank patch 37 on the Gamblin target. It shows the optimized curve that has a much better RMS spectral and worse CIEDE2000 than the unoptimized curve. This means that theoretically, the unoptimized curve should appear visually closer to the measured than the optimized. Figures 8.6 and 8.7 a and b show histogram plots of the RMS spectral error and CIEDE2000 after being calculated for each individual pixel. The histograms show how the distribution of values shifts between the RMS spectral errors, and while the CIEDE2000 are close, the distribution is still wider for the optimized exposure.

8-9

1 0.9 0.8 0.7 Reflectance Factor 0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500
1/30 DE:4.96 %RMS:10.98 OPT. DE:5.39 %RMS:7.38

Measured 1/30 Optimized

550 600 Wavelength (nm)

650

700

Figure 8.5 - Reflectance of blank patch 37 for the optimized and unoptimized case compared with measured spectra.
250

OPT UOPT

200

Opt. Mean:0.074 Uopt. Mean:0.110 Opt. Max:0.221 Uopt. Max:0.192 Opt. Min:0.038 Uopt. Min:0.051 Opt. Std:0.017 Uopt. Std:0.025

150

Count
100 50 0 0

10 15 % RMS Spectral Error

20

25

Figure 8.6 Histogram of individual pixel %RMS spectral error for blank patch 37 - both optimized and unoptimized case.

8-10

1200

1200

1000

Mean:4.957 Max:19.068 Min:1.032 Std:1.627

1000

Mean:5.393 Max:21.723 Min:1.003 Std:2.372

800

800

Count

600

Count

600

400

400

200

200

0 0

10 15 CIEDE 2000

20

25

0 0

10 15 CIEDE 2000

20

25

a. unoptimized

b. optimized

Figure 8.7 a and b - Histogram of individual pixel CIEDE2000 for blank patch 37 optimized and unoptimized case.

The values for the components of the CIEDE2000 calculation are given in table 8.3. It shows that the L value decreases as expected and that the major component contributing to the difference in the optimized exposure is the change in C, which is significantly greater than the unoptimized set. Table 8.3 also shows the same measures calculated for the yellow patches in this group. It confirms that the major contributing factor for the increase in CIEDE2000 is a greater shift in C in the optimized exposures.

8-11

Table 8.3 - CIEDE2000 Components for the 55 and NF combination, showing an increase in C. Patch L L a' a b b C C H H # Uopt. Opt. Uopt. Opt. Uopt. Opt. Uopt. Opt. Uopt. Opt. Blank Cadmium Yellow Light Cadmium Yellow light Cadmium Yellow Medium Earth Yellow Hansa Yellow Medium Indian Yellow Indian Yellow Yellow Medium Yellow Medium Yellow ochre Yellow ochre 37 4 59 13 63 42 47 56 22 33 53 62 2.37 1.48 2.6 1.22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.65 1.19 1.89 3.55 1.65 1.19 -0.85 0.17 0.51 4.66 -0.85 0.17 -0.93 -0.49 0.05 5.56 -0.93 -0.49 2.37 4.09 1.61 6.43 2.37 4.09 -0.98 4.5 4.4 1.53 8.52 -0.98 4.5 4.4 5.09

2.41 1.17 2.06 0.21 1.09 -2.04 2.09 0.9 0.5 -1 0.43 -1.82 2.12 0.04 2.06 0.45 2.08 -0.55 2.04 -1.19

0.49 1.99 1.89 5.02 0.49 1.99 5.09 2.42 5.76 7 3.97 5.33 2.46 3.97 5.33

2.47 3.72 1.48 5.01 2.47 3.72 0.03 1.97 1.87 6.78 0.03 1.97 0.97 5.41 0.2 5.56 0.97 5.41 1.44 6.65 0.09 7.61 1.44 6.65

The remaining patches behaved as expected. In the group of samples where both measures increased, there was more error in the blue region resulting from the downward shift in the estimates. The patches where both measures decreased showed that the downward shifting trend in the blue region of the spectrum was what improved the estimates. Figures 8.8 and 8.9 respectively show the spectral estimates and calculated reflectance differences for the case where both RMS spectral error and CIEDE2000 decreased.

8-12

Cobalt Green 1

Phtalo Blue

Viridian

Cobalt Green

Viridian

Cobalt Violet

Ultramarine blue

0.5

0 Cobalt violet 1 Oxide green Ultramarine blue Chromium Oxide Green Managnese blue hue Phthalo Green Cobalt blue

0.5

0 400 Phtalo green 1 Phtalo blue 730 400 730 400 730 400 730 400 730

0.5

0 400

730

400

730

Figure 8.8 - Estimated reflectance spectra of samples where both RMS spectral error and CIEDE2000 decrease for the 55 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor.

8-13

Cobalt Green 0.2

Phtalo Blue

Viridian

Cobalt Green

Viridian

Cobalt Violet

Ultramarine blue

0.2 Cobalt violet 0.2 Oxide green Ultramarine blue Chromium Oxide Green Managnese blue hue Phthalo Green Cobalt blue

0.2 400 Phtalo green 0.2 Phtalo blue 730 400 730 400 730 400 730 400 730

0.2 400

730

400

730

Figure 8.9 - Difference plots for estimated spectra samples where both RMS spectral error and CIEDE2000 decrease for the 55 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference.

Arrow plots representing changes in the XYZ tristimulus values and L*a*b* colorimetric coordinates between the two exposure sets were used to give some indication as to which channels may be producing the errors seen. In these figures, the base of the arrow indicates the measured values and the tip indicates the estimate. The X vs. Y and Z vs. Y plots in Figure 8.10 a-d show an increase in X, Y, and Z for the unoptimized exposure while the optimized exposure shows decreases in X, Y, and Z. This is consistent with the error in the estimated blue regions of the spectrum as X, Y, and Z are a non-linear transformation involving the integration of reflectance spectra with the color matching functions and source illuminant. This then translates to the errors

8-14

seen in the L*a*b* plots in Figure 8.11 showing decreases in L* and significant increases in b* for yellow and red samples.

120

120

100

100

80

80

60

60

40

Y
20 40 60 X 80 100 120

40

20

20

0 0

0 0

20

40

60 X

80

100

120

a. Y vs. X - Unoptimized
120 120

b. Y vs. X - Optimized

100

100

80

80

60

60

40

Y
20 40 60 Z 80 100 120

40

20

20

0 0

0 0

20

40

60 Z

80

100

120

c. Y vs. Z Unoptimized

d. Y vs. Z - Optimized

Figure 8.10 - XYZ arrow plots for 55 and NF combination comparing the unoptimized and optimized exposures.

8-15

100

100

50

50

b*

b*
100 50 0 a* 50 100

50

50

100

100 100 50 0 a* 50 100

a. b* vs. a* - Unoptimized

b. b* vs. a* - Optimized.

100

100

50

50

L*

L*
100 50 0 a* 50 100

50

50

100

100 100 50 0 a* 50 100

c. L* vs. a* - Unoptimized

d. L* vs. a* - Optimized

100

100

50

50

L*

L*
100 50 0 b* 50 100

50

50

100

100 100 50 0 b* 50 100

e. L* vs. b* - Unoptimized

f. L* vs. b* - Optimized

Figure 8.11 - L*a*b* arrow plots for the Wratten 55 and NF combination.

8-16

Figure 8.12 shows plots of each channel of the transformation matrix generated by each set with the time scaling factor divided out. The blue solid line represents the unoptimized channels and the green dashed represents the optimized. The left three represent the filtered red, green, and blue channels and the right represents the unfiltered. The behavior of this matrix is what makes the spectral estimation possible. As can be seen, the channels together are selective at various points in the spectrum. It also appears that all but one of the channels behave similarly between the optimized and unoptimized exposure. It is this difference that is causing the downward shift noticed in the optimized spectral reconstructions.

8-17

x 10

x 10

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

a. Red Filtered
x 10
4

b. Red - Unfiltered
x 10
4

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

c. Green - Filtered
x 10
4

d. Green - Unfiltered
x 10
4

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

e. Blue Filtered

f. Blue - Unfiltered

Figure 8.12 -55 and NF transformation matrix channels with the blue line representing the channel from the unoptimized matrix and the green dashed line representing the optimized matrix channels.

8-18

Although this filter did not perform the best, it did perform reasonably well, and would make a sensible selection for use in this particular application. It demonstrated the following desirable characteristics: the transform generated showed a reasonable degree of selectivity, when compared to the other filters it performs slightly better than the average, it did a reasonable job of estimating the long wavelength region of the spectrum in many cases, and in most cases the RMS spectral error decreased by optimizing the exposure time. While it may not be the best for colorimetric accuracy, it would also be useful in analysis for pigments with tails that extend into the long wavelength region of the spectrum as shown by the optimized estimates.

Wratten 60 and NF filter combination The Wratten 60 and NF combination performed best experimentally. By

optimizing the exposure time, RMS spectral error decreased 1.6% giving it the lowest average RMS spectral error across patches and showed only a .18 increase in CIEDE2000, giving it the second lowest average amongst the filter combinations. In separating the patches into the various categories of performance, they all fell into either one of the following two: either both CIEDE2000 and RMS spectral error decreased or the RMS spectral error decreased and the CIEDE2000 increased. It has already been shown that these two measures do not have a direct correlation meaning that for all the patches, optimizing the time reduced the RMS spectral error without significantly increasing the colorimetric measure. Figure 8.13 shows the reflectance estimates for the case where the CIEDE2000 increased, but RMS spectral error decreased. Figure 8.14 shows the reflectance differences as a function of wavelength for

8-19

the same case. Figures 8.15 and 8.16 show the same data for the case where both measures decrease.
Burnnt Sienna 1 Cadmium Yellow Light Cadmium Orange Earth Yellow Burnnt Umber Cadmium Yellow Medium Cadmium Orange

0.5

0 Dioxazine Purple 1 Trans Earth Red Burnnt Umber Yellow Medium Black Diozine Purple Prussian Blue

0.5

0 Trans Earth Red 1 Raw Sienna Black Yellow Medium Prussian Blue RawUmber Raw Sienna

0.5

0 Hansa Yellow Medium 1 Venetian red Indian Yellow Spinel Yellow ochre Venetian red Indian Yellow

0.5

0 400 Cadmium yellow light 1 Yellow ochre Earth Yellow 730 400 730 400 730 400 730

0.5

0 400

730

400

730

400

730

Figure 8.13 - Estimated reflectance spectra of samples where RMS spectral error decreases and CIEDE2000 increases for the 60 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor.

8-20

Burnnt Sienna 0.2

Cadmium Yellow Light

Cadmium Orange

Earth Yellow

Burnnt Umber

Cadmium Yellow Medium

Cadmium Orange

0.2 Dioxazine Purple 0.2 Trans Earth Red Burnnt Umber Yellow Medium Black Diozine Purple Prussian Blue

0.2 Trans Earth Red 0.2 Raw Sienna Black Yellow Medium Prussian Blue RawUmber Raw Sienna

0.2 Hansa Yellow Medium 0.2 Venetian red Indian Yellow Spinel Yellow ochre Venetian red Indian Yellow

0.2 400 Cadmium yellow light 0.2 Yellow ochre Earth Yellow 730 400 730 400 730 400 730

0.2 400

730

400

730

400

730

Figure 8.14 - Difference of estimated reflectance spectra of samples where RMS spectral error decreases and CIEDE2000 increases for the 60 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference.

8-21

Cadmium Red Light 1

Cobalt Green

Phtalo Blue

Quinacridone Red

Viridian

Cadmium Red Light

Cobalt Green

0.5

0 Quinacridone Red 1 Viridian Red medium Cobalt Violet Ultramarine blue Red medium Cobalt violet

0.5

0 Oxide green 1 Ultramarine blue Blank Black Spinel Chromium Oxide Green Managnese blue hue Phthalo Green

0.5

0 Raw umber 1 Blank Cobalt blue Managnese blue hue Indian red Phtalo green Blank

0.5

0 400 Burnt Sienna 1 Cobalt blue Indian red Phtalo blue 730 400 730 400 730

0.5

0 400

730

400

730

400

730

400

730

Figure 8.15 - Estimated reflectance spectra of samples where both RMS spectral error and CIEDE2000 decrease for the 60 and NF combination. The red, dashed line represents the unoptimized estimate, the green is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor.

8-22

Cadmium Red Light 0.2

Cobalt Green

Phtalo Blue

Quinacridone Red

Viridian

Cadmium Red Light

Cobalt Green

0.2 Quinacridone Red 0.2 Viridian Red medium Cobalt Violet Ultramarine blue Red medium Cobalt violet

0.2 Oxide green 0.2 Ultramarine blue Blank Black Spinel Chromium Oxide Green Managnese blue hue Phthalo Green

0.2 Raw umber 0.2 Blank Cobalt blue Managnese blue hue Indian red Phtalo green Blank

0.2 400 Burnt Sienna 0.2 Cobalt blue Indian red Phtalo blue 730 400 730 400 730

0.2 400

730

400

730

400

730

400

730

Figure 8.16 - Difference plots for estimated spectra samples where both RMS spectral error and CIEDE2000 decrease for the 60 and NF combination. The blue, dashed line represents the unoptimized estimate, the red, solid is optimized. The x axis is wavelength; the y axis is reflectance factor difference.

In the samples that CIEDE2000 behave similarly to the Wratten 55 and NF combination, the matrix is causing the optimized estimate to shift down, making estimates in the long wavelength region better at the expense of error in the blue

8-23

introducing colorimetric error. In the samples where both cases decrease, both the blue and the red region are more accurate and the cost to each region has been balanced. Table 8.4 shows these measures for a selection of yellow patches from the case where RMS decreases and CIEDE2000 increases. The table shows that the largest contribution to the increase in CIEDE2000 again comes from the C shift.
Table 8.4 - CIEDE2000 components for the 60/NF filter combination showing increase in C. Patch # Blank Cadmium Yellow Light Cadmium Yellow light Cadmium Yellow Medium Earth Yellow Hansa Yellow Medium Indian Yellow Indian Yellow Yellow Medium Yellow Medium Yellow ochre Yellow ochre 37 4 59 13 63 42 47 56 22 33 53 62 L Uopt. 2.27 2.58 2.35 1.96 0.95 2 0.34 0.85 2.06 1.91 1.97 1.94 L Opt. 1.28 1.25 1.22 0.44 -0.94 0.65 -1.16 -0.61 0.44 0.6 0.1 -0.21 a' Uopt. 0 0 0 0 0 0 0 0 0 0 0 0 a Opt. 0 0 0 0 0 0 0 0 0 0 0 0 b Uopt. 1.87 -0.65 -0.44 2.51 -0.09 0.56 4.81 4.23 2.48 0.12 1.87 2.03 b Opt. -0.36 -0.73 -0.91 3.07 3.69 0.96 4.81 4.72 2.97 1 4.61 5.66 C Uopt. 2.98 0.67 0.32 2.25 1.97 2.32 2.77 2.86 2.05 2.61 0.48 0.66 C Opt. 2.9 5.18 5.76 7.06 6.4 5.68 5.73 7.16 5.52 7.28 4.53 5.67 H Uopt. 1.87 -0.65 -0.44 2.51 -0.09 0.56 4.81 4.23 2.48 0.12 1.87 2.03 H Opt. -0.36 -0.73 -0.91 3.07 3.69 0.96 4.81 4.72 2.97 1 4.61 5.66

Tristimulus and L*a*b* plots for this filter combination show the same trends noticed in the Wratten 55, but are generally less pronounced as shown in figures 8.17 and 8.18. This is consistent with the spectral graphs.

8-24

120

120

100

100

80

80

60

60

40

Y
20 40 60 X 80 100 120

40

20

20

0 0

0 0

20

40

60 X

80

100

120

a. Y vs. X - Unoptimized
120 120

b. Y vs. X - Optimized

100

100

80

80

60

60

40

Y
20 40 60 Z 80 100 120

40

20

20

0 0

0 0

20

40

60 Z

80

100

120

c. Y vs. Z - Unoptimized

d. Y vs. Z - Optimized

Figure 8.17 - XYZ arrow plots for the Wratten 60 and NF filter combination.

8-25

100

100

50

50

b*

b*
100 50 0 a* 50 100

50

50

100

100 100 50 0 a* 50 100

a. b* vs. a* - unoptimized

b. b* vs. a* - optimized

100

100

50

50

L*

L*
100 50 0 a* 50 100

50

50

100

100 100 50 0 a* 50 100

c. L* vs. a* - unoptimized

d. L* vs. a* - optimized

100

100

50

50

L*

L*
100 50 0 b* 50 100

50

50

100

100 100 50 0 b* 50 100

e. L* vs. b* - Unoptimized

f. L* vs. b* - Optimized

Figure 8.18 - L*a*b* arrow plots for the Wratten 60 and NF combination.

8-26

Plots of the individual transformation matrix channels in figure 8.19 show that this filter is also spectrally selective at various points in the spectrum, but also shows that in addition to the large difference in the filtered blue channel, there is a much steeper slope in the long red region of the filtered red channel, which points to some of the increased error noticed in the long red region of the optimized estimates.

8-27

x 10

x 10

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

a. Red Filtered
x 10
4

b. Red - Unfiltered
x 10
4

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

c. Green Filtered
x 10
4

d. Green - Unfiltered
x 10
4

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

e. Blue Filtered

f. Blue - Unfiltered

Figure 8.19 - Wratten 60 and NF transformation matrix channels with the blue line representing the channel from the unoptimized matrix and the green dashed line representing the optimized matrix channels.

8-28

If colorimetric accuracy is one of the imaging goals, than this filter makes a good choice because in the optimized image, the RMS spectral error is reduced, but the red is more inaccurate and balanced by more accuracy in the blue region of the spectrum where CIEDE2000 is calculated from.

Wratten 2C and NF combination This filter combination performed the worst in both the experiment and the simulation. Although the optimized time for this filter combination was 1/30th of a second, the images were taken with both sets to try and show how slight changes in the environment may affect performance. The set taken in the unoptimized group was flat fielded with the same image, while the optimized was flat fielded with a corresponding filtered gray card image. Also, the target had to be moved between exposure groups, which may have introduced differences and errors which would cascade through the system. Figure 8.20 shows selected spectral estimates generated with this filter combination. This combination has difficulty in dealing with both the blue and red regions of the spectrum and appears to only perform marginally well in the green.

8-29

Cadmium Red Light 1

Quinacridone Red

0.5

Cobalt violet 1

Ultramarine blue

0.5

0 400

730

400

730

Figure 8.20 Selected spectra estimates for the Wratten 2C and NF combination. The red, dashed line represents the unoptimized estimate, the green x line is optimized, and the blue is measured. The x axis is wavelength and the y axis is reflectance factor.

8-30

Figure 8.21 shows the plots of the transformation matrix generated with this data. The transform is not very selective over the different regions of the spectrum, meaning it has more difficulty in accurately fitting the data.

8-31

x 10

x 10

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

a. Red Filtered
x 10
4

b. Red - Unfiltered
x 10
4

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

c. Green Filtered
x 10
4

d. Green - Unfiltered
x 10
4

Reflectance Factor/DC

Reflectance Factor/DC
450 500 550 600 Wavelength (nm) 650 700

400

400

450

500

550 600 Wavelength (nm)

650

700

e. Blue Filtered

f. Blue - Unfiltered

Figure 8.21 - Transformation matrix channels for the Wratten 2C and NF combination with the blue line representing the channel from the unoptimized matrix and the green dashed line representing the optimized matrix channels.

8-32

Sensitivity of the pseudo-inverse transformation matrix: While the pseudo inverse has shown to give acceptable results, it is very sensitive to noise in the data. Figure 8.22 shows the progression of the estimates from the noiseless model to the optimized experimental data exploiting the dynamic range of the camera for the cobalt blue (patch 49) of the Gamblin paint target. The curves show that the noiseless case has the best estimate but it must be kept in mind that it is only good for average values, not pixel based measurements that include noise. It also shows that the simulated noise estimate is close to the unoptimized estimate resulting from images taken with the exposure time set to 1/30th of a second, particularly in the long wavelength region. This would be expected as the simulation was designed for the unoptimized case. The curves in the figure also show that the optimized case gives the best results with the a priori analysis, which is the recommended case to use for practical imaging.

8-33

1 0.9 0.8 0.7

Measured Noiseless simulation Noise simulation Imaging: Unoptimized Imaging: Optimized

Reflectance Factor

0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500 550 600 650 700

Wavelength (nm)
Figure 8.22 - Progression of estimated reflectance spectra through the simulations and imaging cases for cobalt blue (49) on the Gamblin paint target.

Figure 8.23 shows the estimate from the average pixel values and five estimates resulting from five randomly chosen pixel values of the same cobalt Blue patch for the optimized case. Table 8.5 gives the pixel values used to create the curves and their quality metrics.

8-34

1 0.9 0.8 0.7

Reflectance Factor
Measured Average 1 2 3 4 5

0.6 0.5 0.4 0.3 0.2 0.1 0 400 450 500 550 600 Wavelength (nm) 650 700

Figure 8.23 - Plots of curves generated from individual pixels from cobalt blue demonstrating the sensitivity of the pseudo-inverse transform.

Table 8.5 - Average and individual digital counts from cobalt blue patch with resulting quality metrics. Pixel Average R1 G1 Channel B1 R2 G2 B2 % Spectral RMS error CIEDE200 (D65/2 degree observer) 1009 1192 493 916 409 331 3.8 1 1013 1177 475 929 419 344 4.3 2 1023 1211 508 918 401 328 3.6 3 1003 1180 498 891 392 325 4.0 4 799 990 320 866 382 305 10.2 5 1117 1320 579 971 461 364 3.7

Quality Metric

1.4

3.2

0.7

1.3

9.9

2.7

8-35

Table 8.5 and Figure 8.23 show that small changes in pixel values can have a significant effect on the RMS spectral error and the resulting CIEDE2000. It supports the idea that the CIEDE2000 calculation does not directly correspond with RMS spectral error, and that multiple metrics, the actual reflectance curves, and the objective of the spectral estimation must be considered when making filter choices.

Target and lighting analysis: The evaluation of the filter combinations, particularly the Wratten 55 and NF and Wratten 60 and NF combinations has shown that the system is deficient in the short wavelength regions of the spectrum. It is clear that the Z tristimulus value is being underpredicted and that the CIELAB b* value is increasing, particularly in the yellowish samples. While the plots in Chapter Three of the Esser and blues target suggest a well balanced target, this may not be the case. Figure 8.24 shows a plot of the average reflectance spectra as a function of wavelength for the Esser and blues characterization target. It shows that the weighting of the target increases in the longer wavelength region. There is a slight boost in the short wavelength region resulting from the addition of the blues target to the characterization data.

8-36

0.6

Average Average + Std. Dev. Average Std. Dev.

0.5
Reflectance Factor

0.4

0.3

0.2

0.1

0 400

450

500

550 600 Wavelength (nm)

650

700

Figure 8.24 Average reflectance over wavelength for the Esser and blues characterization target.

While Figure 8.24 gives some insight into the properties of the selected characterization target, another factor that has yet to be considered is the light sources used, whose spectral power distribution is given in Figure 3.4. This type of light gives relatively low output in the short wavelength region. This can have a dramatic effect on the target reflectance in the blue region of the spectrum. Figure 8.25 shows the average target reflectance multiplied by the spectral power distribution of the Scanlites used.

8-37

0.8 0.7 0.6

Reflectance Factor
Average EB * Source Average EB * Source + Std. Dev. Average EB * Source Std. Dev.

0.5 0.4 0.3 0.2 0.1 0 400

450

500

550 600 Wavelength (nm)

650

700

Figure 8.25 Average Esser and blues target reflectance multiplied by source spectral power distribution over wavelength.

The light source appears to be severely hurting the blue region of the spectrum. This will clearly have an effect on the unfiltered camera signals and even more severe effect on filtered camera signals. Figure 8.26 shows the curve resulting from the average Esser and blues target reflectance, the light source, and the camera sensitivities, or the curve resulting from Eq. 8.1 where q( ) is the product of the curves over wavelength. q ( )=p( )r ( )s ( ) (8.1)

8-38

0.04 0.035 0.03

Relative Camera Signal

0.025 0.02 0.015 0.01 0.005 0 400

450

500

550 600 Wavelength (nm)

650

700

Figure 8.26 Resulting camera signals before integration over wavelength.

Figure 8.26 is showing that the blue signal, relative to the green and red, has been greatly reduced. This suggests that both the selection of both the target and the lighting may play a very important role in the ability of a given system to perform spectral reflectance estimation.

Conclusions: The data from three different filter combinations from the experiment discussed in Chapter Seven were evaluated. Different aspects of the combinations resulting estimates were observed. It was found that for a particular filter to work well for spectral

estimation with six channels, the transform must exhibit spectral selectivity across the

8-39

spectrum over the channels being used. For this camera, it appears that a good filter choice is generally a green filter with a tail in the long wavelength region of the spectrum for the first three channels and then no filter which provides the information for the visible region of the spectrum. From the colorimetric calculations and visual evaluation of the spectra, it has been shown that trying to maximize dynamic range as much as possible will decrease RMS spectral error but the CIEDE2000 does not necessarily correlate with RMS spectral error. The CIEDE2000 measure includes weightings which take the human visual system into account. This causes the CIEDE2000 to have a dependency on the particular region of the spectrum where error is occurring, and may cause it to increase although RMS is decreasing. Also, by maximizing dynamic range, the transformation matrix may

rebalance how it uses each channel and while it may reduce overall average RMS, it may shift error into the region that CIEDE2000 is calculated in. Finally, it was found that this system will generally have problems estimating the blue region of the spectrum, affecting both the X and Z tristimulus values and causing a significant b* (yellowness) increase in the yellowish verification samples. It was also found that a matrix formed from a green filter will tend to shift the curve down, lowering Y and L*. The sensitivity of the pseudo inverse transform was shown across unequal pixels. Small changes in pixel values can cause large changes in the spectral estimation and resulting metrics. Finally, an analysis of the properties of the target and lighting was done. The results show that target and lighting selection may also play a critical role in the system performance.

8-40

Chapter 9 - Conclusions and Future Research:

Conclusions: Over the course of this research, two simulations and an imaging experiment were performed using a trichromatic digital camera to develop methods to help select a combination of filters from a larger, readily available set that would provide the best performance with respect to spectral and colorimetric quality metrics. Based on the results of a target reconstruction using eigenvector analysis, six channels were chosen to be used for the simulations. The first simulation was based on a simple linear model while the second was designed to be more complex and include noise properties of the camera system. Chapter Four outlined the noiseless simulation. PCA was used on the

characterization target and the scalar eigenvalues were related to the simulated camera response. It was found that the results were too similar for a large portion of the combinations performing well, making decisions very difficult. A second result showed that the simulation predicted that certain filters that did not significantly affect the cameras spectral sensitivity and did not make intuitive sense were predicted to perform well. It was determined that this method, while commonly used in many previous simulations, did not include noise and that a camera model that included noise needed to be developed. Chapter Five outlined how the noise characteristics were approximated and added to the camera model. After analyzing data taken at a previous imaging session, it was found that the variance of the pixel data corresponding to a target patch could be approximated by creating a linear relationship between the average signal and variance in

9-1

each channel. This relationship was then applied by generating a group of pixels having the same average value corresponding to a target sample and using the signal to variance relationship as input to an available Matlab function that would redistribute the pixels in a Gaussian manner so that they would have the same average value and variance specified by the model. This data was then used to simulate spectral reflectance estimates which were analyzed by the appropriate quality metrics. While the model was more complex than the one used in the noiseless case, it was still very simple in that it remained based on linear relationships that were created from least squares fits to known camera data, but was found to be acceptable for adding noise to the system and creating more realistic results. Chapter Six illustrated the new simulation process. By generating many

individual pixels as opposed to average values, a large overdetermined system could be created to which a pseudo inverse operation could be applied to find a least squares fit between simulated pixel data and measured target reflectance. This method showed that the addition of noise had a serious impact on the system and affected the quality of the spectral estimations. It was also found that this model helped solve the problem of predicting filters that did not make sense. Thus, this simulation gave a much better initial insight as to which Wratten filters would work best for this system. Chapter Seven described the imaging experiment, which was performed in a setting which closely approximated imaging at an art gallery or other environment that copy work would be performed in. Based on the results of the noise simulation, several combinations of Kodak Wratten filters were chosen for testing. Two sets of images were taken for each combination. The first were all taken at the same exposure for all filters to

9-2

match the model. The reasoning for this was that an individual using this system may not take the time to rebalance the exposure for different filters, but would simply add or change the filters in the system. The second method optimized the exposure for each filter by finding the exposure time that would maximize the use of the dynamic range of the camera without clipping the image. This data was then compared with the simulated data and differences between the model and experiment were identified. It was shown that the model still had difficulties as it predicted exposure levels that were either to high or to low, which was a result of the model not taking aspects such as filter factor into the equation. These errors were determined to be linear, and were not considered to

invalidate the model as these factors would only affect scaling and would fall out in any linear operations used to estimate spectral reflectance. The model also was unable to exactly model variance, which has shown to be a result of texture in both the characterization and verification target. It was also found that the system suffered from some physical limitations, the most important being that exposure control was in whole stop increments and that three channels were tied to one exposure. Chapter Eight discussed the results of the spectral reflectance estimates resulting from the experimental data. It showed that expanding the dynamic range decreased the spectral RMS error in the optimized verification target, but could increase the CIEDE2000. This was found to be a result of several factors including the fact that CIEDE2000 is a weighted metric that takes several aspects of human vision into account giving it a dependency on which region in the spectrum the error was occurring in for the reflectance curve or curves in question. It was also found that a significant source of error was found in the short wavelength regions of the spectrum, confirmed by the poor

9-3

estimation of the Z tristimulus value and CIELAB b* coordinate. This chapter also briefly mentioned the suitability of the lighting and characterization target being used. This resulted in finding that the spectral power distribution of the light sources may be seriously affecting the data collected in the short wavelength region of the spectrum. It also leads to the point that great care must be used in selecting a target for system characterization. Finally, the sensitivity of the pseudo inverse transform was

demonstrated. It was shown that small variations in the data can have a large effect on the resulting estimation and quality metrics. Based on the results of this research, the following general conclusions were reached. First, the concept of using a mathematical model as a guide was proven. In general, the filter combination that works best within the confines of this system based on the TerraPix camera would be a green filter with a tail passing information in the long red regions of the spectrum. If the only information used to select a filter is the simulation, than the combination of data from the Wratten 55 and unfiltered data would be recommended. Since there was the ability to experimentally test different combinations, it was found that the combination of the Wratten 60 and unfiltered data produces the best results for spectral estimation. It can also be concluded that the choice of filters used depends heavily on the purpose of the imaging work being done and that combinations of metrics should be taken into consideration. This result is very typical and has been found by others using CCD cameras to perform spectral image acquisition (Berns 2003; Imai 2000a).

9-4

Future Improvements and Research: This research has shown that a mathematical simulation based on the system characteristics can be used as a guide to selecting a combination of filters from a larger set of readily available filters by reducing the number of combinations from an extremely large set (greater than 1000) to a much smaller set (less than 30 and often using similar filters within combinations) for experimental testing and then selecting the best performing combination. However, improvements can be made as it is more desirable to be able to make choices based solely on the simulation. The first suggestion is to find a way to improve the simulated data. This could be done including texture in the variance model and improving the overall performance of the simulation by trying to make it more closely match the actual system and by including a component that would take filter factors into account and maximize the dynamic range in simulated exposures as was done for the optimized data set. A second area for improvement may be to look at the methods used to perform the spectral reflectance reconstruction. While the pseudo inverse has proven to be very useful, it is sensitive to variations in the data. Other methods have been suggested, and it may prove beneficial to find ways to modify the pseudo inverse or alter other methods to be used with individual, pixel level data. Currently, the specifications of a camera system to be used at the National Gallery of Art, Washington D.C. and The Museum of Modern Art in New York City are being determined. Already, some of the issues addressed by this thesis are being worked on. Improvements in lighting and modifying the transform used to relate digital counts and spectral reflectance are already being considered. It is hoped that the major results of this research are that doors to new areas of research have been opened and that a useful

9-5

methodology for simulating and evaluating a spectral imaging system has been made and is available to be openly used and improved upon.

9-6

Chapter 10 - References:
Berns, R. S. (1999). Challenges for Colour Science in Multimedia Imaging. Colour Imaging: Vision and Technology. L. W. MacDonald and M. R. Luo, John Wiley & Sons Ltd.: (99-127). Berns, R. S. (2000). Billmeyer and Saltzman's Principles of Color Technology. New York, NY, John Wiley & Sons, Inc.: (27-29). Berns, R. S., L. A. Taplin, F. H. Imai, E. A. Day and D. C. Day (2003). Spectral Imaging of Matisses Pot of Geraniums: A Case Study. IS&T and SID 11th Color Imaging Conference, IS&T, Springfield, VA 2003 (In press). Burns, P. D. and R. S. Berns (1998). Image Noise and Colorimetric Precision in Multispectral Image Capture. The Sixth Color Imaging Conference: Color Science, Systems, and Applications, Scottsdale, Arizona, IS&T, Springfield, VA 1998: (83-85). Burns, P. D. and R. S. Berns (1999). Quantization in Multispectral Color Image Acquisition. The Seventh Color Imaging Conference: Color Science, Systems, and Applications, Scottsdale, Arizona, IS&T, Springfield, VA 1999: (32-35). Day, D. C. (2003). Evaluation of Optical Flare and Its Effects On Spectral Estimation Accuracy. Rochester, NY, MCSL Technical Report. www.art-si.org. Day, E. A., F. H. Imai, L. A. Taplin and S. Quan (2002). Characterization of a Roper Scientific Quantix monochrome camera. Rochester, N.Y., MCSL Technical Report. www.art-si.org. Day, E. A. (2003). The Effects of Multi-channel Visible Spectrum Imaging on Perceived Spatial Image Quality and Color Reproduction Accuracy, Rochester Institute of Technology: (1, 3) www.art-si.org. Eastman Kodak Company (1994). Application Note: Solid State Image Sensors Terminology. Rochester, N.Y.: (1-54). Fairchild, M. D., M. R. Rosen and G. M. Johnson (2001). Spectral and Metameric Color Imaging. Rochester, NY, MCSL Internal Report. www.art-si.org. Haneishi, H., T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura and Y. Miyake (2000). "System design for accurately estimating the spectral reflectance of art paintings." Applied Optics 39(35): (6621-6632). Hardeberg, J. Y., F. Schmitt, H. Brettel, J. P. Crettez and H. Matre (1999). Multispectral image acquisition and simulation of illuminant changes. Colour Imaging: Vision and Technology. L. W. MacDonald and M. R. Luo, John Wiley & Sons Ltd.: (145-164). Hardeberg, J. Y. (2003). Filter Selection for Multispectral Color Image Acquisition. PICS Conference, Rochester, NY, IS&T, Springfield, VA 2003: (177-181). Imai, F. H. (1998a). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part II - Technical Approach. Rochester, NY, MCSL Technical Report. www.artsi.org. Imai, F. H. (1998b). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters. Rochester, NY, MCSL Technical Report. www.art-si.org.

10-1

Imai, F. H. (1998c). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part III - Samples and Spaces. Rochester, NY, MCSL Technical Report. www.artsi.org. Imai, F. H. (1998d). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part IV - Target imaging and digital counts processing. Rochester, NY, MCSL Technical Report. www.art-si.org. Imai, F. H. (1998e). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part Va - Spectral reconstruction in reflectance space: Target I. Rochester, NY, MCSL Technical Report. www.art-si.org. Imai, F. H. (1998f). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part Vb - Spectral reconstruction in reflectance space: Targets II and III. Rochester, NY, MCSL Technical Report. www.art-si.org. Imai, F. H. (1998g). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part VI - Spectral reconstruction in Kubelka-Munk space. Rochester, NY, MCSL Technical Report. www.art-si.org. Imai, F. H. (1998h). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part VII - Spectral reconstruction in the new empirical space. Rochester, NY, MCSL Technical Report. www.art-si.org. Imai, F. H. (1998i). Multi-spectral Image Acquisition and Spectral Reconstruction using a Trichromatic Digital Camera System Associated with Absorption Filters: Part VIII - General Discussion. Rochester, NY, MCSL Technical Report. www.artsi.org. Imai, F. H. and R. S. Berns (1999). Spectral Estimation Using Trichromatic Digital Cameras. International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives, Chiba, Japan: (42-49). Imai, F. H., R. S. Berns and D.-Y. Tzeng (2000a). "A Comparative Analysis of Spectral Reflectance Estimated in Various Spaces Using a Trichromatic Camera System." The Journal of Imaging Science and Technology 44(4): (280-287). Imai, F. H., M. R. Rosen and R. S. Berns (2000b). Comparison of spectrally narrow-band capture versus wide-band with a priori sample analysis for spectral reflectance estimation. The Eighth Color Imaging Conference: Color Science, Systems, and Applications, Scottsdale, Arizona, IS&T, Springfield, VA 2000: (234-241). Imai, F. H., M. R. Rosen and R. S. Berns (2002). Comparative study of metrics for spectral match quality. First European Conference on Color in Graphics, CGIV 2002, Imaging and Vision, IS&T, Springfield, VA 2002: (492-496). Imai, F. H., L. A. Taplin and E. A. Day (2003). Comparative study of spectral reflectance estimation based on broad-band Imaging systems. Rochester, N.Y., MCSL Technical Report. www.art-si.org. Johnson, G. M. and M. D. Fairchild (1998). Computer Synthesis of Spectroradiometric Images for Color Imaging Systems Analysis. The Sixth Color Imaging

10-2

Conference: Color Science, Systems, and Applications, Scottsdale, Arizona, IS&T, Springfield, VA, 1998: (150-153). Knig, F. and W. Praefcke (1998). The Practice of Multispectral Image Acquisition. EUROPTO Conference on Electronic Imaging: Processing, Printing, and Publishing in Color, Zurich, Switzerland, SPIE: (34-41). Knig, F. and W. Praefcke (1999). A multispectral scanner. Colour Imaging: Vision and Technology. L. W. MacDonald and M. R. Luo, John Wiley & Sons Ltd.: (129143). London, B. and J. Upton (1998). Photography. New York, Addison-Wesley: 174. Luo, M. R., G. Cui and B. Rigg (2001). "The Development of the CIE 2000 ColourDifference Formula: CIEDE2000." Color Research and Application 26(5): (340350). Quan, S., N. Ohta and N. Katoh (2000). Optimization of Camera Spectral Sensitivities. The Eighth Color Imaging Conference: Color Science, Systems, and Applications, Scottsdale, AZ, IS&T, Springfield, VA 2000: (273-278). Rosen, M. R. and X. Jiang (1999). Lippmann 2000: A spectral image database under construction. International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives, Chiba University, Chiba, Japan: (117-122).

10-3

A.

Matlab Programs

generate_dc: Used to generate the digital counts based on equation 3.6

function camera_dc = generate_dc(source,reflectance,sensitivity,filter,dl) % GENERATE_DC Generates the digital counts of the camera before any % corrections % GENERATE_DC(source,reflectance,sensitivity,filter,dl) % source - required - power distribution of light source % reflectance - required - reflectance data of samples to calibrate to % sensitivity - required - RGB camera sensitivities % filter - optional - default is 1 (no filter), use to add in effects of % filter for multi-channel data (more than 3 channels) % dl - default 10 (nm) - data sampling interval %this implementation neglects dl as it is a scalar value which comes out of %matrix if nargin < 5, dl = 10; end; if nargin < 4, filter = 1; end; size_data = size(reflectance); %get the number of samples for reflectance data for i=1:size_data(2), %number of samples for j=1:3, %R,G,B camera_dc(i,j) = sum(source.*sensitivity(:,j).*reflectance(:,i).*filter); end; end;

A-1

combinations_a.m used to calculate data for a noiseless simulation

%add paths with data files addpath c:\matlab6p5\work\Sinar_fun addpath c:\matlab6p5\work\Sinar_fun\data %addpath /pd2/lat3977/django_crt/munsell %set the wavelengths to use wl_s = 400:10:730; %spectral wavelengths wl_c = 400:10:700; %colorimetric wavelengths %load the targets load esser_blues.mat; load gamblin.mat; load ccdc.mat; load cc.mat; %indexes to use for spectral and colorimetric calculations %spectral - 400:10:730, colorimetric - 400:10:700 idx_s = 5:38; idx_c = 5:35; %number of channels and eigen vectors num_channels = 6; num_evectors = 6; %load the camera sensitivity and interpolate it, and add %the balzers filter load terrapix_QE.mat; load balzers.mat; camera = interp1(terrapix_QE(:,1),terrapix_QE(:,2:4),wl_s); camera = diag(balzers(idx_s))*camera; %load the scanlite = range is from 400-750 nm load scanlite.mat; source = interp1(scanlite(:,1),scanlite(:,2),wl_s); source = (source./max(source))'; %colorimetric stuff cmf = load('CIE1931_2deg.txt'); illuminant = illD(6500); xyz_ccdc = xyz(ccdc(idx_c,:),cmf,illuminant); xyz_esser = xyz(esser_blues(idx_c,:),cmf,illuminant); xyz_gamblin = xyz(gamblin(idx_c,:),cmf,illuminant); xyz_cc = xyz(cc(idx_c,:),cmf,illuminant); xyzn = xyz(ones(31,1),cmf,illuminant); lab_ccdc = lab(xyz_ccdc,xyzn); lab_esser = lab(xyz_esser,xyzn); lab_gamblin = lab(xyz_gamblin,xyzn); lab_cc = lab(xyz_cc,xyzn); %perfect white - for digital counts perfect_white = ones(length(idx_s),1); %choose filter set here - set it up

A-2

%uncomment the following for wratten old = load('wratten_old.mat'); filter_set.filters = interp1(old.spectra.wl,old.spectra.data,wl_s); filter_set.names = old.spectra.patchnames; filter_set.filters = [filter_set.filters,ones(size(filter_set.filters,1),1)]; filter_set.count = size(filter_set.filters,2); filter_set.names{end+1} = 'NF'; clear old; if size(filter_set.filters,2)<100 %if filter set is schott, etc %add no filter filter_set.filters = interp1(filter_set.wl,filter_set.filters,wl_s); filter_set.filters = [filter_set.filters,ones(size(filter_set.filters,1),1)]; filter_set.names{end+1}='NF'; filter_set.count = size(filter_set.filters,2); %build list of sets fprintf('Building filters...\n'); filter_combos = nchoosek(1:size(filter_set.filters,2),2); %multiply combinations filter_set.filters = prod(reshape(filter_set.filters(:,filter_combos),length(idx_s),820,2),3 ); %add in no filter to filter set filter_set.filters = [filter_set.filters,ones(size(filter_set.filters,1),1)]; filter_set.count = size(filter_set.filters,2); end %Build digital counts for the target/filter combinations fprintf('Building images...\n'); for i=1:filter_set.count; sdesser.images(:,i,:) = generate_dc(source,esser_blues(idx_s,:),camera,filter_set.filters(:,i)) ; sdgamblin.images(:,i,:) = generate_dc(source,gamblin(idx_s,:),camera,filter_set.filters(:,i)); sdccdc.images(:,i,:) = generate_dc(source,ccdc(idx_s,:),camera,filter_set.filters(:,i)); sdcc.images(:,i,:) = generate_dc(source,cc(idx_s,:),camera,filter_set.filters(:,i)); white_signal.images(:,i,:) = generate_dc(perfect_white,perfect_white,camera,filter_set.filters(:,i)) ; end; sdesser.count = filter_set.count; sdgamblin.count = filter_set.count; sdccdc.count = filter_set.count; sdcc.count = filter_set.count; white_signal.count = filter_set.count; sdesser.images = permute(sdesser.images,[2,3,1]); sdgamblin.images = permute(sdgamblin.images,[2,3,1]);

A-3

sdccdc.images = permute(sdccdc.images,[2,3,1]); sdcc.images = permute(sdcc.images,[2,3,1]); white_signal.images = permute(white_signal.images,[2,3,1]); fprintf('Big set...\n'); image_combos = nchoosek(1:size(filter_set.filters,2),(num_channels/3)); fprintf('Making transform constant...\n'); [v,e,c_v] = pca(esser_blues(idx_s,:),size(idx_s,2)); v = v(:,size(v,2):-1:size(v,2)-num_evectors+1); T = v*(v\esser_blues(idx_s,:)); %GO LOOPS!!!! fprintf('Hurry up and wait...\n'); for i=1:size(image_combos,1) if mod(i,1000)==0 fprintf('We are at set %i\n',i); end; %PCA esserDC = reshape(sdesser.images(image_combos(i,:),:,:),num_channels,320); ccdcDC = reshape(sdccdc.images(image_combos(i,:),:,:),num_channels,240); gamblinDC = reshape(sdgamblin.images(image_combos(i,:),:,:),num_channels,63); ccDC = reshape(sdcc.images(image_combos(i,:),:,:),num_channels,24); %white signal - sum of camera sens*filter with theroetical equal illuminant %and perfect reflectance ws = reshape(white_signal.images(image_combos(i,:),:),num_channels,1); T2 = T*pinv(esserDC); predS_esser = T2*esserDC; predS_ccdc = T2*ccdcDC; predS_gamblin = T2*gamblinDC; predS_cc = T2*ccDC; diff_esser = (esser_blues(idx_s,:)-predS_esser).^2; diff_ccdc = (ccdc(idx_s,:)-predS_ccdc).^2; diff_gamblin = (gamblin(idx_s,:)-predS_gamblin).^2; diff_cc = (cc(idx_s,:)-predS_cc).^2; RMSmax_esser(i) = max(sqrt(mean((diff_esser),1))); RMSmax_ccdc(i) = max(sqrt(mean((diff_ccdc),1))); RMSmax_gamblin(i) = max(sqrt(mean((diff_gamblin),1))); RMSmax_cc(i) = max(sqrt(mean((diff_cc),1))); RMSstd_esser(i) = std(sqrt(mean((diff_esser),1))); RMSstd_ccdc(i) = std(sqrt(mean((diff_ccdc),1))); RMSstd_gamblin(i) = std(sqrt(mean((diff_gamblin),1))); RMSstd_cc(i) = std(sqrt(mean((diff_cc),1))); RMS_esser(i) = mean(sqrt(mean((diff_esser),1))); RMS_ccdc(i) = mean(sqrt(mean((diff_ccdc),1))); RMS_gamblin(i) = mean(sqrt(mean((diff_gamblin),1))); RMS_cc(i) = mean(sqrt(mean((diff_cc),1)));

A-4

%min and sum of white signals ws_min(i) = min(ws); ws_sum(i) = sum(ws); %colorimetric vals predxyz_esser = xyz(predS_esser(1:31,:),cmf,illuminant(1:31,:)); predxyz_ccdc = xyz(predS_ccdc(1:31,:),cmf,illuminant(1:31,:)); predxyz_gamblin = xyz(predS_gamblin(1:31,:),cmf,illuminant(1:31,:)); predxyz_cc = xyz(predS_cc(1:31,:),cmf,illuminant(1:31,:)); predlab_ccdc = lab(predxyz_ccdc,xyzn); predlab_esser = lab(predxyz_esser,xyzn); predlab_gamblin = lab(predxyz_gamblin,xyzn); predlab_cc = lab(predxyz_cc,xyzn); DE00_esser = deltaE00(predlab_esser,lab_esser); DE00_ccdc = deltaE00(predlab_ccdc,lab_ccdc); DE00_gamblin = deltaE00(predlab_gamblin,lab_gamblin); DE00_cc = deltaE00(predlab_cc,lab_cc); meanDE00_esser(i) = mean(DE00_esser); meanDE00_ccdc(i) = mean(DE00_ccdc); meanDE00_cc(i) = mean(DE00_cc); meanDE00_gamblin(i) = mean(DE00_gamblin); maxDE00_esser(i) = max(DE00_esser); maxDE00_ccdc(i) = max(DE00_ccdc); maxDE00_cc(i) = max(DE00_cc); maxDE00_gamblin(i) = max(DE00_gamblin); stdDE00_esser(i) = std(DE00_esser); stdDE00_ccdc(i) = std(DE00_ccdc); stdDE00_cc(i) = std(DE00_cc); stdDE00_gamblin(i) = std(DE00_gamblin); end; idx=(1:size(image_combos,1))'; sort_gamblin = [idx,image_combos,RMS_gamblin',RMSmax_gamblin',RMSstd_gamblin',meanDE00 _gamblin',maxDE00_gamblin',stdDE00_gamblin',ws_min',ws_sum']; sort_ccdc = [idx,image_combos,RMS_ccdc',RMSmax_ccdc',RMSstd_ccdc',meanDE00_ccdc',ma xDE00_ccdc',stdDE00_ccdc',ws_min',ws_sum']; sort_cc = [idx,image_combos,RMS_cc',RMSmax_cc',RMSstd_cc',meanDE00_cc',maxDE00_cc ',stdDE00_cc',ws_min',ws_sum']; sort_esser = [idx,image_combos,RMS_esser',RMSmax_esser',RMSstd_esser',meanDE00_esser ',maxDE00_esser',stdDE00_esser',ws_min',ws_sum'];

A-5

simulate_noise_il2_pyth simulates pixels and noise using Matlab imnoise() function

%add paths with data files % addpath c:\matlab6p5\work\Sinar_fun addpath % c:\matlab6p5\work\Sinar_fun\data addpath /pd2/lat3977/django_crt/munsell addpath /pd2/dcd7078/work/Sinar_fun/data %set the wavelengths to use wl_s = 400:10:730; %spectral wavelengths wl_c = 400:10:700; %colorimetric wavelengths %indexes to use for spectral and colorimetric calculations spectral %400:10:730, colorimetric - 400:10:700 idx_s = 5:38; idx_c = 5:35; old = load('wratten_old.mat'); filter_set.filters = interp1(old.spectra.wl,old.spectra.data,wl_s); filter_set.names = old.spectra.patchnames; filter_set.filters = [filter_set.filters,ones(size(filter_set.filters,1),1)]; filter_set.count = size(filter_set.filters,2); filter_set.names{end+1} = 'NF' clear old; %load the targets load esser_blues.mat; load gamblin.mat; load ccdc.mat; load cc.mat; load whitecard.mat; %assign calibration and verification targets c_target = esser_blues(idx_s,:); v_target = gamblin(idx_s,:); %number of channels and eigen vectors for PCA num_channels = 6; num_evectors = 6; %load the camera sensitivity and interpolate it, and add the balzers filter load terrapix_QE.mat; load balzers.mat; camera = interp1(terrapix_QE(:,1),terrapix_QE(:,2:4),wl_s); camera = diag(balzers(idx_s))*camera; %load the scanlite = range is from 400-750 nm load scanlite.mat; source = interp1(scanlite(:,1),scanlite(:,2),wl_s); source = (source./max(source))'; %step 1: form variance/intensity relationship for each channel

%intensity values should be normalized between 0-1 intensity_range =[0:.01:1];

A-6

%relationships determined based on plots of variance as a function of %average noralized digital count - Power Function had best R^2 value for %all channels %The following is for the variance modeled as a power function % R_multiplier = 6e-05; R_power = 1.4385; % % G_multiplier = 8e-05; G_power = 1.509; % % B_multiplier = 8e-05; B_power = 1.5578; % % %create variance vectors for input to imnoise var_R = % (intensity_range.^R_power)*R_multiplier; var_G = % (intensity_range.^G_power)*G_multiplier; var_B = % (intensity_range.^B_power)*B_multiplier; %The following is for a linear model of variance determined by the %robustfit algorithm in matlab - lowers weight of outliers Rv_slope = .5080e-4; Rv_offset = -.0312e-4; Gv_slope = .3991e-4; Gv_offset = -.0180e-4; Bv_slope = .3109e-4; Bv_offset = -.0120e-4; % %create variance vectors for input to imnoise var_R = (intensity_range.*Rv_slope)+Rv_offset; var_G = (intensity_range.*Gv_slope)+Gv_offset; var_B = (intensity_range.*Bv_slope)+Bv_offset; var_R(find(var_R<0))=0; var_G(find(var_G<0))=0; var_B(find(var_B<0))=0;

%exposure time that is being simulated t=1/15; %start loop here - do for each filter %slopes are the k values which were calculated using the pseudo inverse and %measured gray corrected data from all availible exposures that was not %clipped. r_slope = 5.4123e5; r_offset = .4e4; g_slope = 4.8335e5; g_offset = .4e4; b_slope = 5.1767e5; b_offset = .4e4; %illumination correction c_ic = normrnd(1,.001,320,1);

A-7

v_ic = normrnd(1,.001,63,1);

for k=1:size(filter_set.filters,2) %gray signal gs = generate_dc(source,whitecard(idx_s),camera,filter_set.filters(:,k)); %signal of target c_signal = generate_dc(source,c_target,camera,filter_set.filters(:,k)); v_signal = generate_dc(source,v_target,camera,filter_set.filters(:,k)); gs = t*(gs*diag([r_slope;g_slope;b_slope]))+([r_offset,g_offset,b_offset]); c_signal(:,1) = (t*c_signal(:,1).*(r_slope))+r_offset; c_signal(:,2) = (t*c_signal(:,2).*(g_slope))+g_offset; c_signal(:,3) = (t*c_signal(:,3).*(b_slope))+b_offset; v_signal(:,1) = (t*v_signal(:,1).*(r_slope))+r_offset; v_signal(:,2) = (t*v_signal(:,2).*(g_slope))+g_offset; v_signal(:,3) = (t*v_signal(:,3).*(b_slope))+b_offset; gs(find(gs>65535))=65535; c_signal_means = c_signal; c_signal = c_signal/65535; v_signal_means = v_signal; v_signal = v_signal/65535; gs = gs/65535;

%create an %simulated c_ic_all = v_ic_all =

array to correspond to the illumination correction at each pixel repmat(c_ic,1,3); repmat(v_ic,1,3);

%create signal arrays which will create all the simulated pixels c_signal = repmat(c_signal.*c_ic_all,[1 1 961]); v_signal = repmat(v_signal.*v_ic_all,[1 1 961]); c_gray_card = repmat(c_ic_all*diag(gs),[1 1 961]); v_gray_card = repmat(v_ic_all*diag(gs),[1 1 961]); %apply noise using the matlab imnoise algorithm c_signal(:,1,:) = imnoise(c_signal(:,1,:),'localvar',intensity_range,var_R); c_signal(:,2,:) = imnoise(c_signal(:,2,:),'localvar',intensity_range,var_G); c_signal(:,3,:) = imnoise(c_signal(:,3,:),'localvar',intensity_range,var_B); v_signal(:,1,:) = imnoise(v_signal(:,1,:),'localvar',intensity_range,var_R); v_signal(:,2,:) = imnoise(v_signal(:,2,:),'localvar',intensity_range,var_G); v_signal(:,3,:) = imnoise(v_signal(:,3,:),'localvar',intensity_range,var_B);

A-8

c_ic_final(:,1,:) = imnoise(c_gray_card(:,1,:),'localvar',intensity_range,var_R); c_ic_final(:,2,:) = imnoise(c_gray_card(:,2,:),'localvar',intensity_range,var_G); c_ic_final(:,3,:) = imnoise(c_gray_card(:,3,:),'localvar',intensity_range,var_B); v_ic_final(:,1,:) = imnoise(v_gray_card(:,1,:),'localvar',intensity_range,var_R); v_ic_final(:,2,:) = imnoise(v_gray_card(:,2,:),'localvar',intensity_range,var_G); v_ic_final(:,3,:) = imnoise(v_gray_card(:,3,:),'localvar',intensity_range,var_B); %quantization error c_signal = floor(c_signal*65535); v_signal = floor(v_signal*65535); c_gray_card = floor(c_gray_card*65535); v_gray_card = floor(v_gray_card*65535); gs = floor(gs*65535); %final signal - rounded to simulate quantization error c_corrected_patches = (c_signal./c_gray_card); v_corrected_patches = (v_signal./v_gray_card); for counter = 1:3 c_corrected_patches(:,counter,:) = c_corrected_patches(:,counter,:)*gs(counter); v_corrected_patches(:,counter,:) = v_corrected_patches(:,counter,:)*gs(counter); end fprintf('Filter %i\n',k); c_corrected_patches = reshape(permute(c_corrected_patches,[2,1,3]),3,320*961); v_corrected_patches = reshape(permute(v_corrected_patches,[2,1,3]),3,63*961); save(sprintf('/pd2/dcd7078/work/Sinar_fun/sim_filter_data_esser/filter_ %i',k),'c_corrected_patches','v_corrected_patches'); end; %save simulated_filter_data.mat c_corrected_batch v_corrected_batch fprintf('Done\n'); %use the following lines if you want statistical data on the simulated %patches % for counter = 1:240 % %clip pixels if greater than 65535 mean_rgb_sim(:,counter) = % mean(squeeze(corrected_patches(counter,:,:)),2);

A-9

% % % % % % % % end

max_rgb_sim(:,counter) = max(squeeze(corrected_patches(counter,:,:)),[],2); min_rgb_sim(:,counter) = min(squeeze(corrected_patches(counter,:,:)),[],2); std_rgb_sim(:,counter) = std(squeeze(corrected_patches(counter,:,:)),0,2); var_rgb_sim(:,counter) = std_rgb_sim(:,counter).^2;

%mean_rgb_sim = mean_rgb_sim'; max_rgb_sim = max_rgb_sim'; min_rgb_sim = %min_rgb_sim'; std_rgb_sim = std_rgb_sim'; var_rgb_sim = var_rgb_sim'; %save simulated_data_ccdc_visnf.mat corrected_patches mean_rgb_sim %max_rgb_sim min_rgb_sim std_rgb_sim var_rgb_sim; save %simulated_data_ccdc_visnf.txt mean_rgb_sim max_rgb_sim min_rgb_sim %std_rgb_sim var_rgb_sim -double -ascii;

A-10

make_transforms_pyth.m computes transforms from simulated data

%add paths with data files % addpath c:\matlab6p5\work\Sinar_fun % addpath c:\matlab6p5\work\Sinar_fun\data addpath /pd2/lat3977/django_crt/munsell addpath /pd2/dcd7078/work/Sinar_fun/data %set the wavelengths to use wl_s = 400:10:730; %spectral wavelengths wl_c = 400:10:700; %colorimetric wavelengths %load the list of filters to use load image_combos; %load the targets load esser_blues.mat; load gamblin.mat; load ccdc.mat; load cc.mat; %indexes to use for spectral and colorimetric calculations %spectral - 400:10:730, colorimetric - 400:10:700 idx_s = 5:38; idx_c = 5:35; %number of channels and eigen vectors num_channels = 6; num_evectors = 6; %load the camera sensitivity and interpolate it, and add %the balzers filter load terrapix_QE.mat; load balzers.mat; camera = interp1(terrapix_QE(:,1),terrapix_QE(:,2:4),wl_s); camera = diag(balzers(idx_s))*camera; %camera = terrapix_vis; %load the scanlite = range is from 400-750 nm load scanlite.mat; source = interp1(scanlite(:,1),scanlite(:,2),wl_s); source = (source./max(source))'; old = load('wratten_old.mat'); filter_set.filters = interp1(old.spectra.wl,old.spectra.data,wl_s); filter_set.names = old.spectra.patchnames; filter_set.filters = [filter_set.filters,ones(size(filter_set.filters,1),1)]; filter_set.count = size(filter_set.filters,2); filter_set.names{end+1} = 'NF' clear old; c_target = esser_blues(idx_s,:); Rc = reshape(repmat(c_target,[1,1,961]),size(c_target,1),size(c_target,2)*96 1);

A-11

%Build digital counts for the target/filter combinations fprintf('Building images for no noise...\n'); for i=1:filter_set.count; sdesser.images(:,i,:) = generate_dc(source,esser_blues(idx_s,:),camera,filter_set.filters(:,i)) ; sdgamblin.images(:,i,:) = generate_dc(source,gamblin(idx_s,:),camera,filter_set.filters(:,i)); end; sdesser.count = filter_set.count; sdgamblin.count = filter_set.count; sdesser.images = permute(sdesser.images,[2,3,1]); sdgamblin.images = permute(sdgamblin.images,[2,3,1]); fprintf('Making transform constant...\n'); [v,e,c_v] = pca(esser_blues(idx_s,:),size(idx_s,2)); v = v(:,size(v,2):-1:size(v,2)-num_evectors+1); C = v*(v\esser_blues(idx_s,:)); %GO LOOPS!!!! fprintf('Hurry up and wait...\n'); for i=1:size(image_combos,1) if mod(i,1000)==0 fprintf('We are at set %i\n',i); end; %PCA esserDC = reshape(sdesser.images(image_combos(i,:),:,:),num_channels,320); T(1,:,:,i) = C*pinv(esserDC); %Direct Pinv filter1 = load(sprintf('/pd2/dcd7078/work/Sinar_fun/sim_filter_data_esser/filter_ %i.mat',image_combos(i,1))); filter2 = load(sprintf('/pd2/dcd7078/work/Sinar_fun/sim_filter_data_esser/filter_ %i.mat',image_combos(i,2))); DC_c = [filter1.c_corrected_patches;filter2.c_corrected_patches]; T(2,:,:,i) = Rc*pinv(DC_c); end; save T1.mat T image_combos;

A-12

reflectance_estimates_from_noise_pyth.m creates reflectance estimates from simulated noise

%load in the list of filters to check %load in precomputed transforms % addpath c:\matlab6p5\work\Sinar_fun % addpath c:\matlab6p5\work\Sinar_fun\data addpath /pd2/lat3977/django_crt/munsell addpath /pd2/dcd7078/work/Sinar_fun/data load T1.mat %setup stuff %indexes for colorimetric and spectral data wl_s = 400:10:730; %spectral wavelengths wl_c = 400:10:700; %colorimetric wavelengths %indexes to use for spectral and colorimetric calculations %spectral - 400:10:730, colorimetric - 400:10:700 idx_s = 5:38; idx_c = 5:35; load esser_blues; load gamblin; c_target = esser_blues(idx_s,:); v_target = gamblin(idx_s,:); Rc = reshape(repmat(c_target,[1,1,961]),size(c_target,1),size(c_target,2)*96 1); Rv = reshape(repmat(v_target,[1,1,961]),size(v_target,1),size(v_target,2)*96 1); %colorimetric stuff cmf = load('CIE1931_2deg.txt'); illuminant = illD(6500); xyz_Rc = xyz(Rc(1:31,:),cmf,illuminant); xyz_Rv = xyz(Rv(1:31,:),cmf,illuminant); xyz_esser = xyz(esser_blues(idx_c,:),cmf,illuminant); xyz_gamblin = xyz(gamblin(idx_c,:),cmf,illuminant); xyzn = xyz(ones(length(idx_c),1),cmf,illuminant); lab_Rc = lab(xyz_Rc,xyzn); lab_Rv = lab(xyz_Rv,xyzn); lab_esser = lab(xyz_esser,xyzn); lab_gamblin = lab(xyz_gamblin,xyzn); %compute reflectance estimates fprintf('Big set...\n'); for i = 1:size(image_combos,1) if mod(i-1,100)==0 fprintf('At %d of %d.\n',i,size(image_combos,1)); end %tic

A-13

filter1 = load(sprintf('/pd2/dcd7078/work/Sinar_fun/sim_filter_data_esser/filter_ %i.mat',image_combos(i,1))); filter2 = load(sprintf('/pd2/dcd7078/work/Sinar_fun/sim_filter_data_esser/filter_ %i.mat',image_combos(i,2))); %toc DC_c = [filter1.c_corrected_patches;filter2.c_corrected_patches]; DC_v = [filter1.v_corrected_patches;filter2.v_corrected_patches]; Rc_pred = squeeze(T(2,:,:,i))*DC_c; Rv_pred = squeeze(T(2,:,:,i))*DC_v; %compute RMS statistics Rc_diff_sq = (Rc_pred-Rc).^2; Rv_diff_sq = (Rv_pred-Rv).^2; Rc_pixel_rms = sqrt(mean(Rc_diff_sq,1)); Rv_pixel_rms = sqrt(mean(Rv_diff_sq,1)); Rc_pixel_rms = reshape(Rc_pixel_rms,320,961); Rv_pixel_rms = reshape(Rv_pixel_rms,63,961); %pixel stats rms_pixel_stats_Rc.mean(i,:) = mean(Rc_pixel_rms,2); rms_pixel_stats_Rc.max(i,:) = max(Rc_pixel_rms,[],2); rms_pixel_stats_Rc.min(i,:) = min(Rc_pixel_rms,[],2); rms_pixel_stats_Rc.std(i,:) = std(Rc_pixel_rms,0,2); rms_pixel_stats_Rv.mean(i,:) = mean(Rv_pixel_rms,2); rms_pixel_stats_Rv.max(i,:) = max(Rv_pixel_rms,[],2); rms_pixel_stats_Rv.min(i,:) = min(Rv_pixel_rms,[],2); rms_pixel_stats_Rv.std(i,:) = std(Rv_pixel_rms,0,2); %target_pixel_stats rms_target_stats_Rc(i,1) rms_target_stats_Rc(i,2) rms_target_stats_Rc(i,3) rms_target_stats_Rc(i,4) rms_target_stats_Rv(i,1) rms_target_stats_Rv(i,2) rms_target_stats_Rv(i,3) rms_target_stats_Rv(i,4) = = = = = = = = mean(rms_pixel_stats_Rc.mean(i,:)); max(rms_pixel_stats_Rc.mean(i,:)); min(rms_pixel_stats_Rc.mean(i,:)); std(rms_pixel_stats_Rc.mean(i,:),0,2); mean(rms_pixel_stats_Rv.mean(i,:)); max(rms_pixel_stats_Rv.mean(i,:)); min(rms_pixel_stats_Rv.mean(i,:)); std(rms_pixel_stats_Rv.mean(i,:),0,2);

%Patch Avg Ref Stats Rc_pred = reshape(Rc_pred,34,320,961); Rv_pred = reshape(Rv_pred,34,63,961); Rc_avg_ref = mean(Rc_pred,3);%do clorimetry on these (34x320) Rv_avg_ref = mean(Rv_pred,3);%convert to XYZ and then do colorimetry Rc_diff_rms(i,:) = sqrt(mean((Rc_avg_ref-c_target).^2)); % (1x320) Rv_diff_rms(i,:) = sqrt(mean((Rv_avg_ref-v_target).^2)); % (1x63) avg_Rctarget_rms(i,1) avg_Rctarget_rms(i,2) avg_Rctarget_rms(i,3) avg_Rctarget_rms(i,4) = = = = mean(Rc_diff_rms(i,:)); max(Rc_diff_rms(i,:)); min(Rc_diff_rms(i,:)); std(Rc_diff_rms(i,:),0,2);

avg_Rvtarget_rms(i,1) = mean(Rv_diff_rms(i,:)); avg_Rvtarget_rms(i,2) = max(Rv_diff_rms(i,:));

A-14

avg_Rvtarget_rms(i,3) = min(Rv_diff_rms(i,:)); avg_Rvtarget_rms(i,4) = std(Rv_diff_rms(i,:),0,2); %colorimetry time %the following lines performs colorimetry on the average reflectance %values Rc_xyz_avg = xyz(Rc_avg_ref(1:31,:),cmf,illuminant(1:31,:)); Rv_xyz_avg = xyz(Rv_avg_ref(1:31,:),cmf,illuminant(1:31,:)); Rc_lab_avg = lab(Rc_xyz_avg,xyzn); Rv_lab_avg = lab(Rv_xyz_avg,xyzn); DE00_Rc_avg = deltaE00(Rc_lab_avg,lab_esser); DE00_Rv_avg = deltaE00(Rv_lab_avg,lab_gamblin); DERc_stats(i,1) DERc_stats(i,2) DERc_stats(i,3) DERc_stats(i,4) DERv_stats(i,1) DERv_stats(i,2) DERv_stats(i,3) DERv_stats(i,4) end; = = = = = = = = mean(DE00_Rc_avg); max(DE00_Rc_avg); min(DE00_Rc_avg); std(DE00_Rc_avg,0,2); mean(DE00_Rv_avg); max(DE00_Rv_avg); min(DE00_Rv_avg); std(DE00_Rv_avg,0,2);

A-15

pullDC_30.m takes digital counts from 1/30th second images

%code to pull DC from the 1/30 second images - first set clear;close all; %setup list of filters filters={'nf','2c','6','38a','40','44a','55','59','59a',... '60','78b','78c','79','80b','80c','82', '82a','82b','82c','90'}; %setup masks for each target eb_mask = imread('data/images_30_masks/eb_mask.tif'); gamblin_mask = imread('data/images_30_masks/gamblin_mask.tif'); ccdc_mask = imread('data/images_30_masks/ccdc_mask.tif'); cc_mask = imread('data/images_30_masks/cc_mask.tif'); kodakgs_mask = imread('data/images_30_masks/kodakgs_mask.tif'); halon_mask = imread('data/images_30_masks/halon_mask.tif'); %indices to patch pixels eb_pixIdx = find(eb_mask); gamblin_pixIdx = find(gamblin_mask); ccdc_pixIdx = find(ccdc_mask); cc_pixIdx = find(cc_mask); kodakgs_pixIdx = find(kodakgs_mask); halon_pixIdx = find(halon_mask); %patch number for each pixel - for each target eb_pixNum = eb_mask(eb_pixIdx); gamblin_pixNum = gamblin_mask(gamblin_pixIdx); ccdc_pixNum = ccdc_mask(ccdc_pixIdx); cc_pixNum = cc_mask(cc_pixIdx); kodakgs_pixNum = kodakgs_mask(kodakgs_pixIdx); halon_pixNum = halon_mask(halon_pixIdx); %load in the dark reference image load images/darkimages_30/average_dark_30.mat; %deal with gray card %load in mask to compute average value gray_mask = imread('data/images_30_masks/gray_mask.tif'); gray_pixIdx = find(gray_mask); %load in the gray card image,subtract the dark,demosiac and flip it fprintf('Loading gray card...\n'); gray_card = readTP('images/whitecard/white_30.raw'); gray_card = gray_card - average_dark; fprintf('Demosaicing gray card...\n'); gray_card = demosaic(gray_card); fprintf('Flipping gray card...\n'); for i=1:3 gray_card(:,:,i)=flipud(gray_card(:,:,i)); end; %perform bluring on gray image fprintf('Bluring Images...');

A-16

[yDim xDim, c] = size(gray_card); [x y] = meshgrid(-xDim/2+1:xDim/2,-yDim/2+1:yDim/2); w = 50; % filter width G = fftshift(exp(-(x.^2 + y.^2)/(w^2))); for j=1:3 fprintf('%d ',j); gray_card(:,:,j) = real(ifft2(fft2(gray_card(:,:,j)).*G)); end fprintf('...done\n'); %reshape gray to list of pixels gray_card = reshape(gray_card,size(gray_card,1)*size(gray_card,2),3); mean_gray = diag(mean(gray_card(gray_pixIdx,:))); %for each filter (image), pull pixels for each target and save to file for i=1:length(filters) %load the image file filename = sprintf('images/images_30/wr%s.raw',filters{i}); fprintf('Loading "%s"\n',filename); img = readTP(filename); img = img-average_dark; fprintf('Demosaicing "%s"\n',filename); img = demosaic(img); fprintf('Flipping image...\n'); for j=1:3 img(:,:,j)=flipud(img(:,:,j)); end; img = reshape(img,size(img,1)*size(img,2),3); %compute corrected DC for each target outputFilename = sprintf('images/images_30/wr%s_DC.mat',filters{i}); eb_DC = (img(eb_pixIdx,:)./gray_card(eb_pixIdx,:)*mean_gray); gamblin_DC = (img(gamblin_pixIdx,:)./gray_card(gamblin_pixIdx,:)*mean_gray); ccdc_DC = (img(ccdc_pixIdx,:)./gray_card(ccdc_pixIdx,:)*mean_gray); cc_DC = (img(cc_pixIdx,:)./gray_card(cc_pixIdx,:)*mean_gray); kodakgs_DC = (img(kodakgs_pixIdx,:)./gray_card(kodakgs_pixIdx,:)*mean_gray); halon_DC = (img(halon_pixIdx,:)./gray_card(halon_pixIdx,:)*mean_gray); fprintf('Saving %s\n',outputFilename); save(outputFilename,'eb_DC','gamblin_DC','ccdc_DC','cc_DC','kodak gs_DC','halon_DC','-mat'); end save('images/images_30/pixNums.mat','eb_pixNum','gamblin_pixNum', 'ccdc_pixNum',... 'cc_pixNum','kodakgs_pixNum','halon_pixNum','-mat');

A-17

pullDC.m pull digital counts from optimized images

%pull Digital counts from images %because times were altered to maximize the signal in this set, %a seperater whitecard will be used. %This will mean SLOW processing time %directories - setup where info is mask_dir = 'data/target_images2_masks'; %directory of mask files image_dir = 'images/target_images2'; %directory of images white_dir = 'images/white_images2'; %directory of whitecard images %dark_dir = 'images/dark' %filters - comment and uncomment appropriate lines for each time setting %because dark current data will be different for each filters = {'nf','2c','6'}; %1/30 second %filters = {'nf_2','2c_2','6_2','78c','82','82a','82b','82c'};%1/15 %filters = {'78c_2','82_2','82a_2','82b_2','82c_2','40','55','59','59a','78b','80b ','80c','90'};%1/8 %filters = {'40_2','55_2','59_2','59a_2','78b_2','80b_2','80c_2','90_2','60'};%1/4 %filters = {'60_2','38a','44a','79'};%1/2 %filters = {'38a_2','44a_2','79_2'};%1s %setup masks fprintf('Reading masks...\n'); eb_mask = imread(sprintf('%s/eb_mask.tif',mask_dir)); gamblin_mask = imread(sprintf('%s/gamblin_mask.tif',mask_dir)); ccdc_mask = imread(sprintf('%s/ccdc_mask.tif',mask_dir)); cc_mask = imread(sprintf('%s/cc_mask.tif',mask_dir)); kodakgs_mask = imread(sprintf('%s/kodakgs_mask.tif',mask_dir)); halon_mask = imread(sprintf('%s/halon_mask.tif',mask_dir)); gray_mask = imread(sprintf('%s/gray_mask.tif',mask_dir)); %indices to patch pixels fprintf('Getting pixel indices...\n'); eb_pixIdx = find(eb_mask); gamblin_pixIdx = find(gamblin_mask); ccdc_pixIdx = find(ccdc_mask); cc_pixIdx = find(cc_mask); kodakgs_pixIdx = find(kodakgs_mask); halon_pixIdx = find(halon_mask); gray_pixIdx = find(gray_mask); %patch number for each pixel - for each target fprintf('Getting patch numbers...\n'); eb_pixNum = eb_mask(eb_pixIdx); gamblin_pixNum = gamblin_mask(gamblin_pixIdx); ccdc_pixNum = ccdc_mask(ccdc_pixIdx); cc_pixNum = cc_mask(cc_pixIdx); kodakgs_pixNum = kodakgs_mask(kodakgs_pixIdx); halon_pixNum = halon_mask(halon_pixIdx);

A-18

%load the dark image - use one for appropriate time load images/dark/average_dark_r30.mat; for i=1:length(filters); %load in the gray card image,subtract the dark,demosiac and flip it fprintf('Loading gray card...\n'); gray_card = readTP(sprintf('%s/wr%s_w.raw',white_dir,filters{i})); gray_card = gray_card - average_dark; fprintf('Demosaicing gray card...\n'); gray_card = demosaic(gray_card); fprintf('Flipping gray card...\n'); for k=1:3 gray_card(:,:,k)=flipud(gray_card(:,:,k)); end; %perform bluring on gray image fprintf('Bluring Images...'); [yDim xDim, c] = size(gray_card); [x y] = meshgrid(-xDim/2+1:xDim/2,-yDim/2+1:yDim/2); w = 50; % filter width G = fftshift(exp(-(x.^2 + y.^2)/(w^2))); for j=1:3 fprintf('%d ',j); gray_card(:,:,j) = real(ifft2(fft2(gray_card(:,:,j)).*G)); end fprintf('...done\n'); %reshape gray to list of pixels gray_card = reshape(gray_card,size(gray_card,1)*size(gray_card,2),3); mean_gray = diag(mean(gray_card(gray_pixIdx,:))); filename = sprintf('%s/wr%s.raw',image_dir,filters{i}); fprintf('Loading "%s"\n',filename); img = readTP(filename); img = img-average_dark; fprintf('Demosaicing "%s"\n',filename); img = demosaic(img); fprintf('Flipping image...\n'); for j=1:3 img(:,:,j)=flipud(img(:,:,j)); end; img = reshape(img,size(img,1)*size(img,2),3); %compute corrected DC for each target outputFilename = sprintf('%s/wr%s_DC.mat',image_dir,filters{i}); eb_DC = (img(eb_pixIdx,:)./gray_card(eb_pixIdx,:)*mean_gray); gamblin_DC = (img(gamblin_pixIdx,:)./gray_card(gamblin_pixIdx,:)*mean_gray); ccdc_DC = (img(ccdc_pixIdx,:)./gray_card(ccdc_pixIdx,:)*mean_gray); cc_DC = (img(cc_pixIdx,:)./gray_card(cc_pixIdx,:)*mean_gray); kodakgs_DC = (img(kodakgs_pixIdx,:)./gray_card(kodakgs_pixIdx,:)*mean_gray); halon_DC = (img(halon_pixIdx,:)./gray_card(halon_pixIdx,:)*mean_gray); fprintf('Saving %s\n',outputFilename);

A-19

save(outputFilename,'eb_DC','gamblin_DC','ccdc_DC','cc_DC','kodak gs_DC','halon_DC','-mat'); end save('images/target_images2/pixNums.mat','eb_pixNum','gamblin_pixNum',' ccdc_pixNum',... 'cc_pixNum','kodakgs_pixNum','halon_pixNum','-mat');

A-20

make_pixel_transforms.m calculates the transforms based on experimental data

%computes the transformation matrices for the experimental data %remember to change the directories when necessary addpath /pd2/dcd7078/work/Sinar_fun/data filters={'nf','2c','6','38a','40','44a','55','59','59a',... '60','78b','78c','79','80b','80c','82', '82a','82b','82c','90'}; combinations=[[7,1];[18,20];[19,20];[7,16];[7,17];[11,20];... [5,1];[7,18];[15,20];[7,12];[10,1];[7,19];... [8,1];[9,1];[14,20];[7,11];[6,1];[2,1]]; idx_s=5:38; %spectral wavelength 400-730nm transform_directory = 'images/images_30/transforms'; directory = 'images/images_30'; load(sprintf('%s/pixNums.mat',directory)); load esser_blues.mat; Rc_pixref = esser_blues(idx_s,eb_pixNum); for i=1:size(combinations,1) fprintf('Transform %i...\n',i); pixDC1 = load(sprintf('%s/wr%s_DC.mat',directory,filters{combinations(i,1)})); pixDC2 = load(sprintf('%s/wr%s_DC.mat',directory,filters{combinations(i,2)})); pixDCa = [pixDC1.eb_DC,pixDC2.eb_DC]'; T_pix(:,:,i) = Rc_pixref*pinv(pixDCa); end; save(sprintf('%s/T_pix.mat',transform_directory),'T_pix','-mat')

A-21

r_final_2.m calculates the results of the experimental data

%program to loop through and calculate all statistics from the reflectance %estimates %setup data that only needs to be loaded or calculated once %load in the directories directories = {'images/images_30','images/target_images2'}; %load in the list of filters filters={'nf','2c','6','38a','40','44a','55','59','59a',... '60','78b','78c','79','80b','80c','82', '82a','82b','82c','90'}; %set up combinations combinations =[[7,1];[18,20];[19,20];[7,16];[7,17];[11,20];... [5,1];[7,18];[15,20];[7,12];[10,1];[7,19];... [8,1];[9,1];[14,20];[7,11];[6,1];[2,1]]; %create target list targets = {'cc','ccdc','eb','gamblin','kodakgs'}; %add in data paths addpath /pd2/lat3977/django_crt/munsell; addpath /pd2/dcd7078/work/Sinar_fun/data; %setup indices for colorimetric and spectral data wl_s = 400:10:730; %spectral wavelengths wl_c = 400:10:700; %colorimeric wavelengths idx_s = 5:38; idx_c = 5:35; %colorimetric data cmf_2 = load('CIE1931_2deg.txt'); %2 deg observer cmf_10 = load('CIE1964_10deg.txt'); %10 deg observer illuminant_D65 = illD(6500); illuminant_A = load('CIE_illA.txt'); white = ones(length(idx_c),1); % XYZn xyznD65 = xyz(white,cmf_2,illuminant_D65); xyznA = xyz(white,cmf_2,illuminant_A); %weighting matrix for wRMS with diagonal R %D65 / 2deg A1 = diag(illuminant_D65)*cmf_2; diagR1 = diag(A1*inv(A1'*A1)*A1'); %A / 2deg A2 = diag(illuminant_A)*cmf_2; diagR2 = diag(A2*inv(A2'*A2)*A2'); %begin processing

A-22

for i=1:length(directories) %load the Transforms and pixel indices load(sprintf('%s/transforms/T_pix.mat',directories{i})); pixNums = load(sprintf('%s/pixNums.mat',directories{i})); for j=1:length(targets) %load the measured spectral reflectances target = load(sprintf('%s.mat',targets{j})); target = target.(sprintf('%s',targets{j}))(idx_s,:); %pixel index pix_idx = pixNums.(sprintf('%s_pixNum',targets{j})); R_measured = target(:,pix_idx); %target XYZ matrices xyz_targetD65 = xyz(R_measured(1:31,:),cmf_2,illuminant_D65); xyz_targetA = xyz(R_measured(1:31,:),cmf_2,illuminant_A); xyz_t = xyz(target(1:31,:),cmf_2,illuminant_D65); %Target LAB values lab_targetD65 = lab(xyz_targetD65,xyznD65); lab_targetA = lab(xyz_targetA,xyznA); lab_t=lab(xyz_t,xyznD65); fprintf('%s Target: %s\nCombination:\n',directories{i},targets{j}); %do for each combination for k = 1:size(combinations,1) fprintf('%i\n',k); pixDC1 = load(sprintf('%s/wr%s_DC.mat',directories{i},filters{combinations(k,1)} )); pixDC2 = load(sprintf('%s/wr%s_DC.mat',directories{i},filters{combinations(k,2)} )); pixDCa = [pixDC1.(sprintf('%s_DC',targets{j})),pixDC2.(sprintf('%s_DC',targets{j }))]'; %compute estimated reflectances and clip 0->1 R_pred = max(min(T_pix(:,:,k)*pixDCa,1),0); D65.xyz = xyz(R_pred(1:31,:),cmf_2,illuminant_D65); D65.lab = lab(D65.xyz,xyznD65); D65.DE00 = deltaE00(D65.lab,lab_targetD65); D65.DE94 = deltaE94(D65.lab,lab_targetD65); D65.DEab = deltaEab(D65.lab,lab_targetD65); A.xyz= xyz(R_pred(1:31,:),cmf_2,illuminant_A); A.lab = lab(A.xyz,xyznA); A.DE00 = deltaE00(A.lab,lab_targetA); A.DE94 = deltaE94(A.lab,lab_targetA); A.DEab = deltaEab(A.lab,lab_targetA);

A-23

pixel over

%processing path 1: %Pixel Reflectance -> RMS error -> XYZ at pixel -> LAB at %Then find metrics for pixels not seperated by patch %Then find metrics per patch -> Target metrics (metrics %patches. Calculate average reflectance curve %arc = average reflectance curve

pixeldata.rms=sqrt(mean((R_pred-R_measured).^2,1)); pixeldata.WRMSD65 = sqrt(mean(((R_pred(1:31,:)R_measured(1:31,:)).^2).*repmat(diagR1,[1 size(R_pred(1:31,:),2)]),1)); pixeldata.WRMSA = sqrt(mean(((R_pred(1:31,:)R_measured(1:31,:)).^2).*repmat(diagR2,[1 size(R_pred(1:31,:),2)]),1)); pixeldata.MID65toA = meta_idx00(R_pred(1:31,:),R_measured(1:31,:),white,cmf_2,illuminant_D65 ,illuminant_A); pixeldata.MIAtoD65 = meta_idx00(R_pred(1:31,:),R_measured(1:31,:),white,cmf_2,illuminant_A,i lluminant_D65); for l=1:double(max(pix_idx)); patchdata.reflectance(:,l,k)=mean(R_pred(:,find(pix_idx==l)),2); patchdata.rms(k,l,1)=mean(pixeldata.rms(find(pix_idx==l))); patchdata.WRMSD65(k,l,1) = mean(pixeldata.WRMSD65(find(pix_idx==l))); patchdata.WRMSA(k,l,1)=mean(pixeldata.WRMSA(find(pix_idx==l))); patchdata.MID65toA(k,l,1)=mean(pixeldata.MID65toA(find(pix_idx==l))); patchdata.MIAtoD65(k,l,1)=mean(pixeldata.MIAtoD65(find(pix_idx==l))); %components of DE00 calculated from average Lab values avg_lab=mean(D65.lab(:,find(pix_idx==l)),2); patchdata.DEcomp(k,l,:)=deltaE00_comp(avg_lab,lab_t(:,l));

patchdata.rms(k,l,2)=max(pixeldata.rms(find(pix_idx==l))); patchdata.WRMSD65(k,l,2) = max(pixeldata.WRMSD65(find(pix_idx==l))); patchdata.WRMSA(k,l,2)=max(pixeldata.WRMSA(find(pix_idx==l))); patchdata.MID65toA(k,l,2)=max(pixeldata.MID65toA(find(pix_idx==l))); patchdata.MIAtoD65(k,l,2)=max(pixeldata.MIAtoD65(find(pix_idx==l))); patchdata.rms(k,l,3)=min(pixeldata.rms(find(pix_idx==l))); patchdata.WRMSD65(k,l,3) = min(pixeldata.WRMSD65(find(pix_idx==l)));

A-24

patchdata.WRMSA(k,l,3)=min(pixeldata.WRMSA(find(pix_idx==l))); patchdata.MID65toA(k,l,3)=min(pixeldata.MID65toA(find(pix_idx==l))); patchdata.MIAtoD65(k,l,3)=min(pixeldata.MIAtoD65(find(pix_idx==l))); patchdata.rms(k,l,4)=std(pixeldata.rms(find(pix_idx==l))); patchdata.WRMSD65(k,l,4)=std(pixeldata.WRMSD65(find(pix_idx==l))); patchdata.WRMSA(k,l,4)=std(pixeldata.WRMSA(find(pix_idx==l))); patchdata.MID65toA(k,l,4)=std(pixeldata.MID65toA(find(pix_idx==l))); patchdata.MIAtoD65(k,l,4)=std(pixeldata.MIAtoD65(find(pix_idx==l))); patchdata.rms(k,l,5)=patchdata.rms(k,l,4).^2; patchdata.WRMSD65(k,l,5)=patchdata.WRMSD65(k,l,4).^2; patchdata.WRMSA(k,l,5)=patchdata.WRMSA(k,l,4).^2; patchdata.MID65toA(k,l,5)=patchdata.MID65toA(k,l,4).^2; patchdata.MIAtoD65(k,l,5)=patchdata.MIAtoD65(k,l,4).^2; patchdata.DE00_65(k,l,1)=mean(D65.DE00(find(pix_idx==l))); patchdata.DE94_65(k,l,1)=mean(D65.DE94(find(pix_idx==l))); patchdata.DEab_65(k,l,1)=mean(D65.DEab(find(pix_idx==l))); patchdata.DE00_A(k,l,1)=mean(A.DE00(find(pix_idx==l))); patchdata.DE94_A(k,l,1)=mean(A.DE94(find(pix_idx==l))); patchdata.DEab_A(k,l,1)=mean(A.DEab(find(pix_idx==l))); patchdata.DE00_65(k,l,2)=max(D65.DE00(find(pix_idx==l))); patchdata.DE94_65(k,l,2)=max(D65.DE94(find(pix_idx==l))); patchdata.DEab_65(k,l,2)=max(D65.DEab(find(pix_idx==l))); patchdata.DE00_A(k,l,2)=max(A.DE00(find(pix_idx==l))); patchdata.DE94_A(k,l,2)=max(A.DE94(find(pix_idx==l))); patchdata.DEab_A(k,l,2)=max(A.DEab(find(pix_idx==l))); patchdata.DE00_65(k,l,3)=min(D65.DE00(find(pix_idx==l))); patchdata.DE94_65(k,l,3)=min(D65.DE94(find(pix_idx==l))); patchdata.DEab_65(k,l,3)=min(D65.DEab(find(pix_idx==l))); patchdata.DE00_A(k,l,3)=min(A.DE00(find(pix_idx==l))); patchdata.DE94_A(k,l,3)=min(A.DE94(find(pix_idx==l))); patchdata.DEab_A(k,l,3)=min(A.DEab(find(pix_idx==l)));

A-25

patchdata.DE00_65(k,l,4)=std(D65.DE00(find(pix_idx==l))); patchdata.DE94_65(k,l,4)=std(D65.DE94(find(pix_idx==l))); patchdata.DEab_65(k,l,4)=std(D65.DEab(find(pix_idx==l))); patchdata.DE00_A(k,l,4)=std(A.DE00(find(pix_idx==l))); patchdata.DE94_A(k,l,4)=std(A.DE94(find(pix_idx==l))); patchdata.DEab_A(k,l,4)=std(A.DEab(find(pix_idx==l))); patchdata.DE00_65(k,l,5)=patchdata.DE00_65(k,l,4).^2; patchdata.DE94_65(k,l,5)=patchdata.DE94_65(k,l,4).^2; patchdata.DEab_65(k,l,5)=patchdata.DEab_65(k,l,4).^2; patchdata.DE00_A(k,l,5)=patchdata.DE00_A(k,l,4).^2; patchdata.DE94_A(k,l,5)=patchdata.DE94_A(k,l,4).^2; patchdata.DEab_A(k,l,5)=patchdata.DEab_A(k,l,4).^2; end target_data.rms(k,1)=mean(patchdata.rms(k,:,1)); target_data.rms(k,2)=max(patchdata.rms(k,:,1)); target_data.rms(k,3)=min(patchdata.rms(k,:,1)); target_data.rms(k,4)=std(patchdata.rms(k,:,1)); target_data.rms(k,5)=target_data.rms(k,4).^2; target_data.WRMSD65(k,1)=mean(patchdata.WRMSD65(k,:,1)); target_data.WRMSD65(k,2)=max(patchdata.WRMSD65(k,:,1)); target_data.WRMSD65(k,3)=min(patchdata.WRMSD65(k,:,1)); target_data.WRMSD65(k,4)=std(patchdata.WRMSD65(k,:,1)); target_data.WRMSD65(k,5)=target_data.WRMSD65(k,4).^2; target_data.WRMSA(k,1)=mean(patchdata.WRMSA(k,:,1)); target_data.WRMSA(k,2)=max(patchdata.WRMSA(k,:,1)); target_data.WRMSA(k,3)=min(patchdata.WRMSA(k,:,1)); target_data.WRMSA(k,4)=std(patchdata.WRMSA(k,:,1)); target_data.WRMSA(k,5)=target_data.WRMSA(k,4).^2; target_data.MIAtoD65(k,1)=mean(patchdata.MIAtoD65(k,:,1)); target_data.MIAtoD65(k,2)=max(patchdata.MIAtoD65(k,:,1)); target_data.MIAtoD65(k,3)=min(patchdata.MIAtoD65(k,:,1)); target_data.MIAtoD65(k,4)=std(patchdata.MIAtoD65(k,:,1)); target_data.MIAtoD65(k,5)=target_data.MIAtoD65(k,4).^2; target_data.MID65toA(k,1)=mean(patchdata.MID65toA(k,:,1)); target_data.MID65toA(k,2)=max(patchdata.MID65toA(k,:,1)); target_data.MID65toA(k,3)=min(patchdata.MID65toA(k,:,1)); target_data.MID65toA(k,4)=std(patchdata.MID65toA(k,:,1)); target_data.MID65toA(k,5)=target_data.MID65toA(k,4).^2; target_data.DE00_A(k,1)=mean(patchdata.DE00_A(k,:,1)); target_data.DE00_A(k,2)=max(patchdata.DE00_A(k,:,1)); target_data.DE00_A(k,3)=min(patchdata.DE00_A(k,:,1)); target_data.DE00_A(k,4)=std(patchdata.DE00_A(k,:,1)); target_data.DE00_A(k,5)=target_data.DE00_A(k,4).^2; target_data.DE94_A(k,1)=mean(patchdata.DE94_A(k,:,1));

A-26

target_data.DE94_A(k,2)=max(patchdata.DE94_A(k,:,1)); target_data.DE94_A(k,3)=min(patchdata.DE94_A(k,:,1)); target_data.DE94_A(k,4)=std(patchdata.DE94_A(k,:,1)); target_data.DE94_A(k,5)=target_data.DE94_A(k,4).^2; target_data.DEab_A(k,1)=mean(patchdata.DEab_A(k,:,1)); target_data.DEab_A(k,2)=max(patchdata.DEab_A(k,:,1)); target_data.DEab_A(k,3)=min(patchdata.DEab_A(k,:,1)); target_data.DEab_A(k,4)=std(patchdata.DEab_A(k,:,1)); target_data.DEab_A(k,5)=target_data.DEab_A(k,4).^2; target_data.DE00_65(k,1)=mean(patchdata.DE00_65(k,:,1)); target_data.DE00_65(k,2)=max(patchdata.DE00_65(k,:,1)); target_data.DE00_65(k,3)=min(patchdata.DE00_65(k,:,1)); target_data.DE00_65(k,4)=std(patchdata.DE00_65(k,:,1)); target_data.DE00_65(k,5)=target_data.DE00_65(k,4).^2; target_data.DE94_65(k,1)=mean(patchdata.DE94_65(k,:,1)); target_data.DE94_65(k,2)=max(patchdata.DE94_65(k,:,1)); target_data.DE94_65(k,3)=min(patchdata.DE94_65(k,:,1)); target_data.DE94_65(k,4)=std(patchdata.DE94_65(k,:,1)); target_data.DE94_65(k,5)=target_data.DE94_65(k,4).^2; target_data.DEab_65(k,1)=mean(patchdata.DEab_65(k,:,1)); target_data.DEab_65(k,2)=max(patchdata.DEab_65(k,:,1)); target_data.DEab_65(k,3)=min(patchdata.DEab_65(k,:,1)); target_data.DEab_65(k,4)=std(patchdata.DEab_65(k,:,1)); target_data.DEab_65(k,5)=target_data.DEab_65(k,4).^2;

end save(sprintf('%s/%s_stats_2.mat',directories{i},targets{j}),'patchdata' ,'target_data','xyz_targetA',... 'xyz_targetD65','lab_targetD65','lab_targetA'); clear target pix_idx R_measured xyz_targetD65 xyz_targetA lab_targetD65 lab_targetA... pixDC1 pixDC2 pixDCa R_pred D65 A pixeldata patchdata target_data; end clear T_pix pixNums; end

A-27