Unlock the Unknown with

Multispectral Imaging
Multispectral Imaging
Moves into the
Mainstream

Valerie C. Coffey

Advances in sensors,
filters and apertures
are driving the
evolution of
multispectral imaging
from expensive one-off
systems primarily for
military and defense
apps to affordable,
practical, commercial
systems for use in
everything from medical
imaging to satellite
remote sensing.

A multispectral image
from the DMCii satellite
identifies fresh
vegetation in Myanmar
(red), 2011. The UK-DMC-2
22-m obtained the image
from orbit using three
spectral bands: red,
green and NIR.

Courtesy of DMCii

1 | Ocean Optics E-book

 Simultaneous imaging in multiple
wavelength bands across the spectrum
enables us to “see” details far beyond the
capa-bilities of the human eye. Combined
with telescopes, multispectral imaging
(MSI) exposes the secrets of the uni-verse;
combined with high-resolution micro-
scopes, it identifies hidden diseas-es. As
recently as 15 years ago, MSI was largely
limited to very expensive, bulky and cus-
tom-built systems for laboratory and gov-
ernment applications. Systems in Earth
orbit required public funding because of
the high cost associated with building
and launching them. Recent advances in
shared apertures, sensors, filters and de-
sign have helped MSI evolve to more ac-
cessible, affordable, compact systems for
commercial use.
Multispectral imaging combines two
to five spectral imaging bands of relative-
ly large bandwidth into a single optical
system. A multispectral system usually
provides a combination of visible (0.4 to
0.7 μm), near infrared (NIR; 0.7 to 1 μm),
short-wave infrared (SWIR; 1 to 1.7 μm),
mid-wave infra-red (MWIR; 3.5 to 5 μm)
or long-wave infrared (LWIR; 8 to 12 μm)
bands into a single system.
Many electro-optic/infrared multi-spec-
tral systems for intelligence, surveillance
and reconnaissance (ISR) applications
combine at least one reflec-tive-band
(EO) sensor with one thermal-band (IR)
sensor. Visible, NIR and SWIR are usually
considered EO bands and form images
using reflected light from a target. MWIR
and LWIR are thermal IR bands, and they
directly image the blackbody radiation
from a target.
Multi- vs. shared aperture
One way to categorize multispectral
systems is by aperture type, according
to Craig Olson, senior optical engineer at
L-3 Communications (Santa Rosa, Calif.,
U.S.A.). Multi-aperture systems have one
window for each sensor, and sensors gen-
erally do not share common optics; they
have relatively uncomplicat-ed sensor de-
signs but difficult packaging constraints.
Shared-aperture systems, in contrast,
combine as much of the opti-cal path as
possible among the various focal planes
to minimize the size and weight of the to-
tal system. Shared-aper-ture systems can
2 | Ocean Optics E-book
[A fused multispectral image]
Mission capability and operational
envelopes can be extended by
performing imaging tasks later into
the evening, for example, or from a
longer range.
Courtesy of Sensors Unlimited/Goodrich
ISR
[The WESCAM 474HD]
The WESCAM 474HD is a large
shared-aperture SWIR and MWIR
system recently developed at L-3
Communications / Wescam-Sonoma.
It uses a combination of reflective and
refractive optics. An 11-in. shared-
aperture front end supports both
spectral bands. The spectral light
paths are split, and more traditional
zooming (multiple field-of-view)
refractive optics complete the imaging
system.
Courtesy of Sensors Unlimited/Goodrich
ISR
support much larger optical apertures for
the same overall system size.
According to Olson, spectral systems
incorporating the 2- to 3.5-μm band are
less common because both the atmo-
sphere and common optical glasses ab-
sorb heavily in that region. Systems for
scientific or industrial process control ap-
plications might need these wavelengths,
but combining that spe-cific band into a
multispectral system with another band
poses significant engineering challenges.
Applications between 2 and 3.5 μm are
usually quite specialized. For example,
the detection of explosives and chemicals
falls into this category.
Standard definition imagers at 640 ×
512 pixels are the most common configu-
rations for non-visible bands. Focal planes
with high-definition format (1,280 × 720
or larger) are recent developments in sen-
sors other than visible charge-coupled
devices (CCD) or complementary metal
oxide semiconductors (CMOS). Such im-
agers can exploit large apertures since
the large pixel size no longer limits spa-
tial resolution. “Maintaining sensitivity in
small pixels is an ongoing challenge for
focal plane manufacturers, as is incre-
as-ing the dynamic range or well depth,”
says Olson.
Multispectral shared-aperture systems
provide a lot of benefit when they exploit
disparate physical phenomena between
reflected and thermally emitted target
signatures. Moreover, such systems also
maximize spatial resolution. With a larger
aperture, the sensor optics can support a
much longer focal length with true reso-
lution (i.e., not empty magnifi-cation) at
the low f-numbers required for extreme-
ly sensitive detection and iden-tification
tasks common in ISR applica-tions. Mis-
sion capability and operational envelopes
can be extended by perform-ing imaging
tasks later into the evening, for example,
or from a longer range.
Mission capability and
operational envelopes can
be extended by performing
imaging tasks later into the
evening, for example, or
from a longer range.
Image fusion
MSI systems have an advantage in im-
age fusion, says Olson, a capability that
has become accessible to many systems
in recent years. Th ey are compatible by
design with image fusion and other ap-
plications that require close align-ment
of spatial information from two or more
spectral bands. While not provid-ing as
dense a dataset or “image cube” as a high-
end hyperspectral system, a multispectral
system combines just two panchromatic
data sets with diff erent insights into tar-
get characteristics to augment the infor-
mation contained in a snapshot or video
sequence.
For example, when observing a parked
automobile, someone using thermal in-
frared can easily determine how long the
car has been parked, if the engine is on,
and whether the air conditioner is run-
ning. However, since most common glass
is opaque to the thermal infrared, such
imagers cannot detect if anyone is still
inside the car. A visible or SWIR imager
will determine the presence of people in-
side the car but will yield no information
about the ther-mal history of the car. A
multispectral tactical sensor providing
fused imagery is a great asset in that case,
when both high spatial resolution and de-
tection of multiple target phenomena are
necessary.
Airborne/ground tactical sensors
An emerging group of commercial
MSI sensors combines SWIR and MWIR.
Both spectral bands typically require low
f-numbers, have similarly matched pixel
sizes available commercially, and can ex-
ploit well-known and mature glass in their
optical systems; consequently, these two
bands are well-suited for large shared-ap-
erture systems.
Both SWIR and MWIR are com-mon
bands in which the penetration of atmo-
spheric haze, smoke and/or clouds are
important. Airborne surveillance, tactical
law enforcement, strategic ISR and aircraft
monitoring applications can all benefi t
from these bands when com-bined into
multispectral systems.
The development of specialized SWIR
lenses has advanced a lot in the past four
years, according to Olson. Although astro-
nomical systems have used NIR and SWIR
imaging almost since the discovery of
the infrared band by William Herschel in
1800, only with the availability of sensors
has the SWIR band been considered cost-
eff ective for industrial inspection systems
and ISR applications. SWIR is now viable
for both single and multispectral systems
3 | Ocean Optics E-book
[The WESCAM 474HD]
Uni
que iimaging
Unique ma ging regions
regions
(Top) Teledyne DALSA’s monolithic
mul-tispectral imager can combine
different imaging areas for varying
bandwidths on one chip with “push-
broom” linear and TDI sensors to
12,000+ pixels. (Bottom) Teledyne
DALSA’s four-band CCD sensor +
pan (or mono) band enables Earth-
orbit multispectral imag-ing like that
of the DMCii constellation. The fi lters
are visible as multicolored panels on
the sensor surface.
Courtesy of Teledyne DALSA
thanks to the introduction of new opti-cal
glasses, improvements in the tech-niques
for designing lenses using these materials
and the increased availability of SWIR im-
aging sensors.
In addition, newly available dual-band
infrared detectors can image simultane-
An enhanced image from the
Worldview satellite shows
the oil spill and clean-up
effort in the Gulf of Mexico
on 15 June 2010. Various
sensor bands highlight the
oil and dispersant
ously in both MWIR and LWIR using the
same focal plane and camera engine. With
properly designed optical systems, multi-
spectral imagers using these focal planes
can dramatically reduce size, weight and
power for sys-tems that require both in-
frared bands. “Th is technology is quite
recent and is an enabling technology for
true MSI systems,” says Olson.
Earth-observing satellites
One of the most fascinating appli-
ca-tions of MSI is the remote sensing of
Earth from orbit. In the past 15 years, MSI
instruments installed on satellites such
as NASA’s Terra and DigitalGlobe’s Worl-
dview-2 have become powerful tools for
observing Earth, particularly for agricul-
tural research and mapping of man-made
and natural disasters.
In spite of a name that derives from its
Disaster Monitoring Constellation of fi ve
satellites, DMC International Imaging, Ltd.
(DMCii; Guildford, Sur-rey, England) does
more than disaster monitoring. DMCii
coordinates multiple satellites to provide
rapid-response imag-ing for disasters,
precision agriculture, land-cover map-
ping, forest monitoring and other appli-
cations. DMCii currently owns and op-
erates two satellites: the UK-DMC with a
32-m ground sample distance (GSD), and
the UK-DMC-2 with a 22-m GSD. It also
coordinates several other satellites in the
constella-tion owned and operated by
different nations. Each can image a 650-
km swath of the Earth simultaneously in
red, green and NIR spectral bands—ide-
al for vegetation observations. The three
spectral bands are similar to those of the
long-established Landsat satellite con-
stellation, which serves as a standard for
cross-calibrating satellites from dif-ferent
nations.
“The imagers are cross calibrated in or-
bit to within 1 percent of Landsat,” says
Paul Stephens, director of sales and mar-
keting at DMCii, “enabling a high image
quality that is crucial to our end-users,
such as the U.S. Department of Agricul-
ture, which relied on DMC 22-m imagery
in 2011 for crop statistics in the U.S.A.”
The choice to design new satellites with
the same key spectral bands as Landsat
allows for cross compatibility and data
continuity. The DMC constel-lation repli-
cates the core data supply of the Landsat
program, while dramati-cally improv-
ing Landsat’s 16-day revisit capability.
“Our image swath is 650 km, rather than
Landsat’s 185 km, thanks to a sensor de-
velopment,” says Stephens. “With only
four spacecraft, we achieve the daily re-
visit required to view rapidly changing
phenomena.” This is some-thing that the
Landsat community had high on their
wish list, according to Ste-phens, but the
cost of a single Landsat precludes having
multiple units in orbit. Using commercial
off-the-shelf technol-ogy has changed
the economics of space, he says, enabling
spacecraft to be built at a much lower
price. “Owning and operating spacecraft
profitably, without a government subsi-
dy, points the way to sustainable Earth
observation.”

Satellite calibration
Sensor sensibility
Spatial resolution is the resolving power of an instrument to discriminate fea-
The improved technology behind
tures at a distance. It is based on detector size, focal length and sensor alti-tude.
Earth-orbiting MSI satellites is the de-
Measures of spatial resolution include GSD and instantaneous field of view (IFOV).
velop-ment in the past two years of ad-
The IFOV, or pixel size, is the area of terrain or ocean covered by the field of view of
vanced sensors, says David Cochrane,
a single detector.
director of technology marketing at Tele-
Landsat 7, named for its seven spectral bands, is one of the most accurately cal-
dyne DALSA (Waterloo, Ontario, Cana-
ibrated Earth-observing satellites, meaning its measurements are extremely accu-
da). “The latest advance in these sensors
rate compared to the same measurements made on the ground. Thus, Land-sat 7
is twofold,” says Cochrane. “All the color
data is often used as an in-orbit standard to cross-calibrate other Earth-observing
lines are combined on one chip, and ad-
missions.
vanced dichroic filters provide high trans-
According to Steve Mackin, chief scientist at DMCii, the UK-DMC satel-lites are
mission of light in the required color band
two-foot cubes that provide images up to several thousand kilometers long. The
only. It’s a great combination for our cus-
DMC satellites are cross-calibrated radiometrically against Landsat 7 ETM+ in the
tomers’ satellites.”
three spectral bands using two sites on Earth with distinct features: the Libya 4 site
Modern CCD and CMOS fabrica-tion
featuring sand dunes
techniques combined with advanced di-
(left), and the Dome-C
chroic filters have resulted in sensors that
site in Antarctica fea-
are more cost-effective while main-tain-
turing snow (right).
ing the high performance needed in re-
The first has no instru-
mote-sensing applications. By bond-ing
ments on the ground;
advanced dichroic filters onto the cover
the second is well in-
glass directly in the imaging path, a single
strumented.
device can be tailored to image numerous
The Libya 4 site is
visible and IR bandwidths in a cost-effec-
used all year to trend
tive and reliable package. The advanced
against Landsat. The Dome-C site is mostly used in December and January. “Us-
technology filter approach enables up to
ing two sites and different methodolo-gies allows us to observe biases and remove
12,000 linear pixel arrays, while individu-
them,” says Mackin. “Ideally we want to characterize each site, so we are working
al elements are based on high-resolution
with NASA Goddard on proposals to more effectively model the interactions of the
time delay and integra-tion (TDI) technol-
surfaces we observed to try and reduce uncertainty. The long-term trend for uncer-
ogy to maximize sensitivity and through-
tainties in our data over Libya 4 is within about 0.3 percent of the long-term trend
put.
of Landsat 7 ETM+, which is rounded to 1 percent, just to be safe.”
For multispectral sensors, a unique
Flying in constellation with other nations provides daily repeat imaging any-
Teledyne DALSA process combines the
where in the world. Starting in 2014, the next DMCii constellation will provide daily
multispectral filter with a multisegment-
repeat imaging with 1-m GSD panchromatic and 5-m GSD four-band multispectral
ed linear CCD in a single package. The
imaging. As of early 2012, the imaging capabilities of Landsat 7 have been ham-
mul-tispectral filters, developed in coop-
pered by an on-board scan-corrector failure, so DMC satellites are needed even
eration with experts in optical and metal-
more. However, even with a fully functioning Landsat, additional satellites en-
lic thin-film coatings and dichroic filters,
able more frequent imaging in between Landsat’s cycle of every 16 days, which is
are mul-tilayered thin-film dielectric in-
nec-essary in vegetation studies and to account for cloud cover.
terference filters that optimize transmis-

4 | Ocean Optics E-book

sion and bandwidth selectivity. The five-
band filter obtains an average in-band
transmission of greater than 90 percent
and out-of-band transmission is typically
much less than 1 percent. High-transmis-
sion filters and TDI devices are useful in
remote-sensing applications where sat-
ellites are typically in high orbit and light
signals can be weak.
Another feature of these filters is their
precise alignment to the individual sens-
ing bands of the CCD using align-ment
marks on both the filter array and the mul-
tisegment CCD. In this design, four bands
were designed for R, G and B signals; one
NIR band; and a fifth broad panchromatic
passband encompassing all four narrow
bands. These bands cor-responded to
five high-resolution TDI segments on the
single CCD chip. The exact resolution is
proprietary. However, typical resolutions
in this application range from 4,000 to
12,000 pixels with pixel sizes from 5 to 33
μm. For aerospace remote-sensing appli-
cations, the CCD and package have also
been designed for high reliability and ra-
diation tolerance.
Whereas IR binoculars
alone make it difficult to
recognize a familiar person,
the SWIR/LWIR prototype
enables recognition of
facial features that overlaps
with a broad-brush IR
image.
(Left) The Recon BN binocular system from FLIR features true binocular vision, allowing the user’s eyes to merge separate imag-
es of the SWIR (center) and LWIR (right) bands, used in general surveillance and situational awareness. Grayscale is preferred for
optimizing contrast and easily discriminating hot from cold. With false color, the user must remember whether cyan is hotter
or colder than orange, for example.
Courtesy of FLIR
5 | Ocean Optics E-book
Prototype SWIR/LWIR
binoculars
In 2010, the Advanced Development
Group at FLIR (Wilsonville, Ore., U.S.A.)
created the first pair of infrared binoculars
using two uncooled LWIR microbolom-
eter sensors (8 to 14 μm). “Holding two
thermal weapon sights side by side, we
realized that depth per-ception was pos-
sible,” says Noel Jolivet, project manager.
(That’s him in the images below.) “No one
else had done this because of the ex-
pense, size, weight and power associated
with running two cameras simultaneous-
ly. Others had built so-called “bioculars:”
one camera with the video going to both
eyes, but that doesn’t provide depth per-
ception. With true binoculars (as opposed
to bioculars), the human vision system
comes into play, turning horizontal and
vertical disparity information from two
sources into perception of depth.”
The team proceeded to make a set of
SWIR binoculars (using InGaAS arrays of 1
to 1.7 μm), and finally, a pair of bin-ocu-
lars with an LWIR sensor for one eye and
an SWIR sensor for the other. “The beauty
of this multispectral system,” says Jolivet,
“is that the human vision system does
the fusing, as long as you match the FOVs
[fields of vision] of the optics, can register
the images to some degree, and can ad-
just the brightness of the display to each
eye.”
Whereas IR binoculars alone make it
difficult to recognize a familiar person,
the SWIR/LWIR prototype enables recog-
nition of facial features that overlaps with
[Infrared binoculars]
a broad-brush IR image. “While frankly
the effect is a bit creepy, SWIR enables
identification via facial features through
glass, while LWIR provides thermal info.”
Road signage is another good target for
this technol-ogy, says Jolivet, “since now
you can read the sign, not just see that it
is there. With a portable pair of LWIR/SWIR
binoculars—the only ones in the world—
we’ll be seeing new things in all kinds of
applications.”
Biomedical applications
The way that light interacts with
bio-logical tissue varies considerably with
wavelength, making spectral imaging a
powerful tool for biomedical and chemi-
cal applications, says Randel Mercer, vice
president of business development at
Ocean Thin Films (Golden, Colo., U.S.A.).
For example, imaging in the NIR enables
depth measurement in tissue and blood
chromophores such as oxy-hemoglobin,
deoxy-hemoglobin and bilirubin, when
compared to the visible image. Because
spectral imaging is non-invasive, it’s also
useful in the assessment of burns and skin
inflammation.
Unfortunately, such MSI systems are not
readily available commercially. Research-
ers typically build their own custom ma-
Because spectral imaging
is noninvasive, it’s also
useful in the assessment
of burns and skin
inflammation.
chines using expensive off-the-shelf com-
ponents—a cumbersome endeavor that
costs from tens of thou-sands to millions
of dollars. Ocean Thin Films recently de-
veloped a new MSI technology, the Spec-
troCam, which fea-tures a proprietary
rotating filter wheel (RFW) with custom
dichroic filters in either segmented or
monolithic wheels that can be cost-effec-
tively produced in mass volume. The seg-
mented RFW can be custom configured
with up to eight interchangeable optical
filters.
While filter wheels and CCD cameras ar-
en’t novel, what is new is the high-speed
integration of filter assem-blies that op-
erate with a rotation rate of 1,800 RPM,
a rate that far exceeds the requirements
necessary for most imaging applications.
Combined with long-life motors and a sci-
entific-grade CCD array, this filter design
enables a portable, con-figurable, high-
speed MSI camera.
To put together a system you can use
to customize, once you’ve done analysis
in spectral space, and you know the four
to eight wavelengths you need, you can
specify a low-cost custom camera for
$25,000, whereas use of the filter with
other types of multispectral imaging
technologies like acousto-optic or liquid
tunable filters might cost around $50,000,”
says Jason Eichen-holz, chief technology
officer at sister company Ocean Optics,
manufacturer of low-cost commercial
spectroscopy systems. SpectroCam can
be used for 2-D spectroscopy research in
a variety of fields, including food safety
and water-quality measurement, product
screen-ing, machine vision, medical imag-
ing, surveillance and authentication. “The
goal is to make spectral imaging compact
and affordable,” says Eichenholz. ♦
Valerie C. Coffey (stellaredit@gmail.com)
is a freelance science and technology
writer and editor based in Boxborough,
Mass., U.S.A.
[References and Resources]
»» R.A. Schowengerdt. Remote
sensing: Models and methods
for image process-ing, Academic
Press, 3rd ed., (2007).
»» J. Eichenholz et al. “Sequential
Filter Wheel Multispectral Imaging
Systems,” Proc. OSA/AIO (2010).
»» J. Eichenholz et al. “Real
Time Megapixel Multispectral
Bioimaging,” Proc. SPIE BIOS Vol.
7568 (2010).
»» J. Miller et al. “Hyperspectral and
Multi-spectral Sensors for Remote
Sensing,” DALSA Teledyne white
paper (2012).

[Tuburculosis imaged with SpectroCam MSI system and standard light microscope]

(Left) The infected lung tissue is imaged in eight bands in the visible and NIR with the SpectroCam MSI system and a standard
light microscope.(Right) The SpectroCam has an internal eight-filter assembly, the wavelengths of which are custom defined
by the user between 350 and 1,000 nm; end users can change filters in about one minute.
Courtesy of Ocean Thin Films

6 | Ocean Optics E-book

Ultracompact Fully
Integrated MegaPixel
MultiSpectral Imager
Jason M. Eichenholz
John Dougherty
Abstract
Multispectral imaging or imaging spectroscopy obtains spec-
tral content of an object by dividing the image data, pixel by pix-
el, into wavelength (color) bands. The resulting 3D data cube (x,
y, λ) allows materials to be identified by their pixel spectral con-
tent at multiple wavelengths in addition to their spatial charac-
teristics. A new class of multispectral imaging systems are being
developed that utilizes lithographically patterned dichroic filter
arrays integrated with standard CCD and CMOS detector arrays.
These new imagers offer the unique advantage of scalability
to tens of Megapixel resolutions, compact size, and no moving
parts. Our multispectral imagers are much simpler to manufac-
ture in volume because the complexity is in the lithographically
patterned dichroics rather than in the bulk optical system. The
patterned dichroic filter arrays are fabricated utilizing standard
microlithography techniques and can incorporate up to 10 dif-
ferent wavelength bands deposited onto a single substrate.
Each channel is selectively patterned on the substrate with the
dichroic filter coating applied using standard thin film coating
techniques. The technique is repeated for all of the wavelength
bands and then the final filter array is directly attached and
aligned onto the CCD.
Keywords: Multispectral, Hyperspectral, Imaging, Spectral, Spec-
tra, Patterned Dichroic, Filter Array, Bayer Pattern
1 Introduction
Multispectral imaging involves capturing images of a scene
or object over multiple discrete wavelength bands and ex-
tracting spectral content from that data. By leveraging known
spectral absorption or emission features to identify materials,
the technique can be used for everything from mapping rock
types in geological formations to identifying blood oxygen-
ation or cancer cells. The problem is that multispectral imagers
have historically been large, expensive, sophisticated airborne
or satellite-mounted instruments. Because each scene is cap-
tured in three-dimensions (x, y, λ), the resultant data cubes can
be gigabytes in size, while only a fraction of the data is useful.
Even though multispectral imaging would be a beneficial tool
for a range of low-cost, real-time, limited-wavelength applica-
tions like anticounterfeiting measures or medical diagnostics,
the complexity of today’s offering makes it impossible. This new
7 | Ocean Optics E-book
lithographically patterned dichroic filter array approach pres-
ents a way to change that.
1.1 Traditional Multispectral and Hyperspectral
Systems
There a very few commercially available multispectral imagers.
Most multispectral systems tend to be one-off or research-based
put together using off the shelf parts and cameras and can cost
anywhere from tens of thousands to millions of dollars per sys-
tem, depending on the application, wavelength bands, and res-
olution requirements. A typical multispectral imager essentially
consists of either rotating filter wheels, mechanically diced thin-
film dichroic filters mounted in front of an image sensor, or mul-
tiple cameras with bulk dichroic filters.
Even for those touted as commercial systems, there is no real
volume production pathway with significant price or reduced
complexity enhancements at even as few as tens or hundreds
of units.
A majority of devices that yield multispectral data are in fact
hyperspectral imagers that are set to a few different wavelength
bands. In its most basic form, a hyperspectral imager consists of
a wavelength dispersing element coupled to an image sensor.
Most hyperspectral systems fall into one of the following cate-
gories; Whiskbroom/Pushbroom1, or Tunable Filter2,3. In addi-
tion there are many different techniques being developed for
multispectral imaging including lens arrays coupled to individu-
al spectral filters4, multiple bandpass filters on top of traditional
color cameras5 and Computed Tomography Imaging Spectrom-
eters6,7.
The most common, whiskbroom or pushbroom systems re-
quire the system to be flown over the landscape from above
such as in an airplane or satellite to capture images of the swath
of land below or to slide the object of interest beneath the sys-
tem on a scanner such as a conveyer belt. The tunable filter
system typically uses a 2-D CCD array and utilizes either liquid
crystals or acousto-optical tunable filters (AOTFs) to scan the
wavelength. All of these hyperspectral techniques have the ad-
vantage that they are able to image the scene at hundreds to
thousands of spectral channels, each only several nanometers
wide. To do so, they sacrifice either image spatial resolution or
frame rate as they require increased acquisition time to scan ei-
ther the scene or the spectrum. This increased acquisition time
compromises and in some cases excludes using these systems
when imaging dynamic phenomena or moving objects because
they are not able to simultaneously capture the entire scene and
all of the spectral content at once.
In addition even if the scene is not moving at all such as in
a pathology slide, a hyperspectral imaging system generates a
copious amount of digital data requiring significant acquisition
and storage capabilities. Getting an answer from this data cube
is computationally intensive and in many cases the complete
hyperspectral data cube provides little additional information
compared to just a few (typically 3-8) multispectral imaging
bands. We have found that there are many applications where
just a few wavelengths are “good enough.”
Using our lithographically patterned dichroic filter arrays, we
can build multispectral imagers that capture a scene over a small
number of spectral channels. More important, such filters can be
fabricated quickly, economically, and reliably using well-estab-
lished high volume batch-processing techniques.
1.2 The Bayer filter
The most commonly utilized approach to spectral imaging
is an interlaced technique using three different colored filters
placed on top of a 2-D detector array to produce a “color camera”
via a technique called Bayer filtering.8 In a Bayer filter mosaic,
each quartet of pixels utilizes absorptive gel filters for color se-
lection. To match the human eye response one pixel filter trans-
mits only red light, one pixel filter transmits only blue light, and
two transmit only green light (see Figure 1). This filter overlays
the image sensor pixels with one-to-one correlation; in other
words, when the detector captures an image, 25% of the pixels
capture red wavelengths, 25% capture blue wavelengths, and
50% capture green wavelengths. These RGB filters are typically
absorptive gels and have extremely wide absorption bands with
significant spectral cross talk between the red, green and blue
channels. The resultant data is processed using color-space in-
terpolation algorithms to create a color image.
We can apply this same architecture to our patterned multi-
layer dichroic optical filters. We begin with the Bayer filter con-
cept but instead of using broad absorptive gels to capture data
at red, green, and blue wavelengths, we custom design the di-
chroic filters so that the image sensor operates at λ1, λ2, λ3, and
so on.
Figure 1. Bayer filter mosaics are a typical approach to color imaging in
digital cameras.
2 Patterned Dichroic Filters
The production of patterned filter array technology is based
on the intersection of traditional thin film coating technology
and standard semiconductor photolithography. It is common
for optical processing technologies to utilize techniques and
equipment from the domain of microelectronic fabrication.
Until recently, however, optical coatings have been one area
that existed primarily in the macro realm; entire optical surfaces
could be coated quite easily with a bandpass filter, for instance,
but precise deposition of patterned optical filter coatings was
limited by the use of metal masking. This masking could easily
8 | Ocean Optics E-book
withstand the in-vacuum process heat needed to produce hard,
durable dielectric multilayers, but was expensive to manufac-
ture, difficult to align with the substrate, and unable to produce
a deposited pattern that could be cleanly aligned “edge to edge”
to existing patterns without gaps or overlapping. Similarly, the
dicing and bonding of individual filters together to form an as-
sembly is a tedious and expensive process at best, with minia-
turization limited by handling and dicing constraints.
The dichroic filter array production technique combines mod-
ern optical thin film deposition techniques and standard micro-
lithographic procedures, which enables the precision placement
and patterning of optical thin film coatings on a single substrate.
This process allows for multiple patterned arrays of different op-
tical filters for the patterned sensors and multispectral imaging
applications. It also enables application in biophotonics, and use
in such applications as dense wavelength division multiplexers
(DWDM), micro optical electro mechanical systems (MEOMS)
and optical waveguide-based devices.
A wide variety of optical coatings can be patterned, including
all dielectric multilayer reflectors, bandpass filters, dichroic edge
filters, infrared blockers, or broad band antireflection coatings.
In addition, enhanced metal reflectors, low reflectivity opaque
metals, and electrically conductive transparent patterns can also
be deposited by this technique.
The production of a patterned optical multilayer coated ele-
ment begins with the preparation of the substrate by polishing
and cleaning, and then application of a photoresist layer to the
surface. Generation of a pattern is performed via mask align-
ment, photoresist exposure and development, creating a resist
pattern on the surface to be coated. The prepared substrates are
then placed into a vacuum chamber for deposition of the de-
sired multilayer filter coating. Substrates may undergo exposure
to a plasma source or a beam of energetic ions before coating
deposition to better prepare the surface on an atomic scale, and
during film deposition to enhance film quality and durability.
Substrates also may be heated before or during coating depo-
sition, in certain applications. Processes such as electron-beam
evaporation, and ion-beam as well as magnetron sputtering
can be used to create the multilayer filters. After deposition of
the filter, the patterned coating is rinsed in a suitable solvent,
removing the multilayer film and the resist from the unwanted
areas, by means of a lift-off process. Finally, this sequence can be
repeated as desired, allowing multiple filters to be deposited in
arrays or other patterns. Additional details of the processes used
to make these dichroic filter arrays have been well described
elsewhere9-11.
An example of such a patterned substrate with multiple wave-
lengths is shown below in Figure 2. The large horizontal lines are
approximately 100 microns and narrow stripes are approximate-
ly 20 microns.
There are a number of advantages that can be realized with
the use of this process to generate patterned optical multilayer
coatings. Because the process relies on precision microlithogra-
phy instead of cut metal masks to pattern the deposited coat-
ings, features (coated areas) as small as 10 microns can be pro-
duced today, with spatial registration to adjacent coated areas
within 1 micron. Work is underway to drive to features as small

Figure 2. Three-wavelength band Patterned Dichroic Filter showing
various spatial variations.
as 5 microns. Undesired “shadowing” or thickness drop-off of
the coating at the pattern edges, unavoidable with cut masks,
is eliminated due to the clean pattern edge break produced by
the liftoff process. Intricate coating patterns of any shape or size
can be manufactured without the limits imposed by practical
machining limits of the metal masks. In addition, the cost of
lithographic tooling does not increase greatly with pattern com-
plexity, and has a longer usable life than cut metal masks placed
on top of a substrate, which must be cleaned often to remove
coating deposits, and can be easily damaged during loading,
unloading, and storage.
The patterned dichroic filter coatings exhibit optical and phys-
ical properties similar to that of their traditional, non-patterned
counterparts. The patterned films have a very high inherent re-
sistance to environmental conditions such as humidity and tem-
perature. Since the optical filter coatings are directly applied to
the substrate/device, mechanical resistance to shock and vibra-
tion is improved over bonded discrete filter windows. This is a
major advantage for these multispectral imaging applications.
In addition, as photonic technologies are miniaturized and
integrated with microelectronic and micromechanical systems,
the ability to selectively deposit multilayer optical structures
directly onto the component enables the design engineer to
include wavelength selective filtering structures early in the de-
sign process. It is no longer necessary to make a physical transi-
tion from micron sized on-chip structures to macroscopic diced
and bonded optical filter elements and back again for wave-
length selective integrated optoelectronic or optomechanical
devices for imaging application.
One concern in this process is cleanliness and defect densi-
ty. Using this methodology and refining it over time, we have
produced hundreds of thousands of sharp-edged optical filter
structures for a variety of applications, including spectral sens-
ing and imaging, LCD and DLP displays, and entertainment
lighting. We have recently enhanced our capabilities in this area
by placing this entire process in a class 1000 clean room manu-
facturing space and adding magnetron sputtering to our plasma
enhanced ion sputtering capability. Additional enhancements
are also expected from new photolithography tools, including a
semiconductor grade mask aligner that will help align the adja-
cent pixels with micron scale precision and repeatability.
9 | Ocean Optics E-book
3 Patterned Photodiodes
At the most basic level a photodiode is a type of photodetec-
tor capable of converting light into either current or voltage, de-
pending upon the mode of operation. Photodiodes are similar
to regular semiconductor diodes except that they may be either
exposed (to detect vacuum UV or X-rays) or packaged with a
window or optical fiber connection to allow light to reach the
sensitive part of the device. When a photon of sufficient ener-
gy strikes the diode, it excites an electron, thereby creating a
mobile electron and a positively charged electron hole. If the
absorption occurs in the junction’s depletion region, or one
diffusion length away from it, these carriers are swept from the
junction by the built-in field of the depletion region. Thus holes
move toward the anode, and electrons toward the cathode, and
a photocurrent is produced. Materials commonly used to pro-
duce photodiodes include:
Table 1. Photodiode Detector materials and operating wavelength
bands
Material Wevelength range (nm)
Silicon 190-1100
Germanium 400-1700
Indium gallium arsenide 800-2600
Lead sulfide <1000-3500
The material used to make a photodiode is critical to defining
its properties, because only photons with sufficient energy to ex-
cite electrons across the material’s bandgap will produce signifi-
cant photocurrents. Due to their greater bandgap, silicon-based
photodiodes generate less noise than germanium-based pho-
todiodes. However if you want to detect any wavelengths great-
er than ~1 micron, germanium or InGaAs photodiodes must be
used. This is an area for future investigations.
Complementary metal–oxide–semiconductor (CMOS) refers
to both a particular style of digital circuitry design, and the fam-
ily of processes used to implement that circuitry on integrated
circuits (chips). CMOS circuitry dissipates less power when stat-
ic, and is denser than other implementations having the same
functionality. As this advantage has grown and become more
important, CMOS processes and variants have come to domi-
nate, so that the vast majority of modern integrated circuit man-
ufacturing is on CMOS processes. CMOS technology is used to
produce most of today’s microprocessors, microcontrollers, stat-
ic RAM, and other digital logic circuits. CMOS technology can
also used for a wide variety of analog circuits such as image sen-
sors in cell phone, data converters, and highly integrated trans-
ceivers for many types of communication.
The ability to utilize CMOS to make electrical devices and
photodetectors on a single chip is enabling a major revolution
in imagers and active pixel sensors. The single chip solution of
a photodiode combined with the basic electrical function and
ASICs to make a higher functioning device with active amplifier
and integrated analog to digital converters will open up a pleth-
ora of new applications.
 An active pixel sensor (APS), is an image sensor consisting of
an integrated circuit containing an array of pixel sensors, each
pixel containing a photodetector and an active amplifier. There
are many types of active pixel sensors including the CMOS APS
used most commonly in cell phone cameras and web cameras.
Such an image sensor is produced by a CMOS process (and is
hence also known as a CMOS sensor), and has emerged as an
alternative to charge-coupled device (CCD) imager sensors. The
term active pixel sensor is also used to refer to the individual pix-
el sensor itself, as opposed to the image sensor. In that case the
image sensor is sometimes called an active pixel sensor imager,
active-pixel image sensor, or active-pixel-sensor (APS) imager.
An example of a 2 x 8 array of active pixels with integrated
analog circuits and an integrated Analog to Digital converter is
shown below in Figure 3. The device (Taos TCS-3412) is config-
ured as a color sensor for with four Red, Green, Blue and Clear
color sensors as well as a broadband IR blocker deposited with
the pattered dichroic filter technology discussed above. In ad-
dition each of the four color channel’s electrical gain can be in-
dependently adjusted, balancing the spectral response due to
the wavelength variation of the sensor’s quantum efficiency.
This device can be coated at the wafer level with thousands of
sensors per wafer. The photolithography process has an added
benefit in that the bond pad of the detector left uncoated.
syndrome: A researcher views his or her process at 100 distinct
wavelengths, but in the end, only three to five wavelengths
show anything of interest. These applications can all benefit
from higher spatial resolution and significantly lower spectral
resolution, which allows us to offer an economical alternative.
Using our lithographically patterned dichroic filters, we can
build multispectral imagers that capture a scene over a small
number of spectral channels.
The coatings consist of multilayer stacks of high- and low-in-
dex materials fabricated via plasma-assisted deposition or mag-
netron sputtering. Adjusting the layer thickness and number of
pairs tunes the spectral characteristics of the resultant coating
to reflect or transmit over specific bands. As described above the
lithographic process allows us to pattern an entire wafer in pix-
els as small as 10 μm on a side. The thickness of the filter directly
limits how small the features can be. The thickness is defined by
the filter design and the desired spectral properties of the fil-
ter. Features smaller than 10μm have been fabricated utilizing
different pattern geometries and for less demanding spectral
filtering. Metallic lines smaller than 1 micron are also routinely
deposited using the same techniques.
By alternating between deposition and lithographic pattern-
ing steps, we can form Bayer-filter-like structures corresponding
to the wavelengths of interest for a specific application. We de-
posit the layers for the λ1 filter, lift off the patterning, re-pattern
the wafer to apply the λ2 filter, and so on. In experiments, we
have iterated through 11 successive depositions and patterns in
a matter of a few days. The technology allows us to make a cus-
tom multispectral imager based on what the customer or appli-
cation needs, with a direct pathway to high-volume production.
A schematic of a 4-color, 6-color (including clear) and 8-color
(including clear) dichroic filter array is shown below in Figure 4.

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2
3 4 3 4 3 4 3 4 5 3 4 5 3 4 5 3 4 5 3 4 5
1 2 1 2 1 2 1 2 1 2 6 7 6 7 6 7
3 4 3 4 3 4 5 3 4 5 3 4 1 2 1 2 1 2
1 2 1 2 1 2 1 2 1 2 3 4 5 3 4 5 3 4 5
Table 3. TAOS TCS-3414 Chip with integrated A/D, three color 3 4 3 4 3 4 3 4 5 3 4 5 6 7 6 7 6 7
plus clear channels and integrated IR blocker 1 2 1 2 1 2 1 2 1 2
5 3 4 5 3 4 3 4 5 3 4 5 3 4 5
6 7 6 7 6 7
Photodiodes and active pixel sensors are often used for ac-
curate measurement of light intensity in science and industrial Figure 4. Examples of three different Bayer type dichroic filter arrays
applications. They are also widely used in various medical appli- for multispectral imaging
cations, such as detectors for computed tomography (coupled
with scintillators) or instruments to analyze biological samples
(immunoassay). They are also used in blood gas monitors. All We have complete control of the location of each filter limited
of these applications could be enhanced with these patterned only by the capabilities of the photolithography and our coating
sensors, where each individual pixel has a different wavelength process.
passband. One of the big challenges in traditional hyperspectral
and multispectral imaging is how to extract useful infor-
4 Multispectral Imagers mation from the images. The computational burden imposed
by the sheer size of the data cube can push the processing time
As discussed above, while some military or scientific studies to hours or days or more. That’s fine for scientific research
may need hundreds of spectral channels, there are many more projects but few commercial applications can tolerate such
applications that require data over only a few well-defined delays. This is where the targeted wavelength approach of-
wavelengths of interest. In our experience, this is a common fers significant benefits. A product developer working on a

10 | Ocean Optics E-book

specific application typically already knows the absorption or
emission characteristics of his materials of interest. The often
have already performed hyperspectral analysis to determine
the required wavelength bands. Once they have isolated
their three to eight wavelength bands, we then tailor the dichro-
ic filter to control the imaging wavelength bands for each pixel
of the detector. One set of pixels can be completely transparent
to provide a baseline reference value, for example. With this in-
formation, the software can mathematically compare the results
of two or more other wavelengths to that reference wavelength.
Figure 5. Three wavelength band square Patterned Dichroic Filters.
Sharp-edged microscopic filter elements are routinely produced using
our lithographic patterned coating process. The square filter elements
are approximately 10μm in size.
(a)
(b)
Figure 6 Microphotograph images of sparsely populated color filters.
(a) is a filter designed to provide color images in low light level
scenes and the (b) is designed to provide an additional IR filter. The
rectangular elements are 19x33μm.
11 | Ocean Optics E-book
Figure 6a above is a pattern specifically designed to make a
unique color camera that allows the addition of color with a very
small impact on low light level performance and negligible
impact on limiting resolution. The approach, which includes
the NIR portion of the spectrum along with the visible is enabled
by placing this filter directly on the surface of the surface of an
EMCCD. It renders the correct hue in a real time, video rate im-
age with negligible latency. Figure 6b shows the same pattern
in reflection mode and has an additional filter for IR blocking12.
Depending on the volumes needed for some applications
the filters are coated on thin glass substrates and mounted
directly onto the CCD or CMOS detector. For higher vol-
umes the coating can be applied directly onto the wafer or
standard wafer level packaging techniques. Utilizing wafer
level packaging has an advantage that the patterned filter
array coated glass wafer is merged and bonded to the sili-
con wafer before dicing. Some of the applications require
a fixed spacer between the two wafers. We can utilize a black
chrome netting between the layers for light beam shaping and
to prevent spatial cross talk.
The custom Bayer filter approach involves very few tradeoffs.
The image is captured in a single shot, so all wavelength data is
acquired simultaneously. The method doesn’t sacrifice much
resolution just because you have a multispectral imager
operating over 8 channels with an 8 megapixel detector. This
is done in most off the shelf digital cameras sold today. They
utilize advanced recursive algorithms to interpolate spatial
content among different-wavelength pixels to form the im-
ages13. It requires some computation but it is a process
performed by small FPGAs in nearly every digital camera,
compared to the laborious task of sorting through gigabyte data
cubes searching for features of interest.
A new custom Bayer-type filter can require non-recurring
engineering charges for the design and build, but after
that point the economies of scale offered by high-volume
batch processing kick in. With the exception of the filters,
our multispectral imager incorporates commercial off-the-
shelf products, including detectors and optics. Once the
customized filters are fabricated, they’re placed atop detector
arrays in a chip-level integration process. The approach is prob-
ably not practical for five systems unless the user needs simulta-
neous imaging and spectral content or a very small size. How-
ever, if someone wants to produce 500 or even 5,000 systems,
dichroic filter arrays provide a pathway to get there at a fraction
of the price of today’s lowest cost multispectral or hyperspectral
systems.
The dichroic filter array technology is also very scalable. Since
we utilize photolithography to produce the arrays, it is just as
easy to produce a 640 x 480 array as it is to produce an ar-
ray for a 10 megapixel image sensor because the complexity
lies not in the technique for implementing the camera itself
but in the lithographically patterned dichroic filter. That sim-
plicity is what makes the dichroic filter array based system very
easy to build and easy to use.
5 Applications
The key advantage of this new type of camera and imager is
the ability to provide a compact system that is scalable to high
volumes. These new imagers offer the unique advantage
of scalability to tens of Megapixel resolutions, with compact
size and no moving parts. Classic static operations like filter
wheels on microscopes become dynamic, allowing the user
to view the scene in multiple wavelengths simultaneously.
Surface mapping of vegetation can be streamlined and units
can be added to aircraft without regard to flight path or speed.
New markets like 3-D imaging and 3-D animation, which to this
point have used multiple CCD arrays, can now target end con-
sumer use.
The applications go well beyond entertainment and secu-
rity. In agriculture, vehicle-mounted systems can be used to
monitor crops, detecting pests and diseases, assessing crop
and soil nutrient levels, and helping forecast yield. Lower
cost multispectral imagers can help ensure food safety by
scanning produce for everything from fungi and dirt to
bacteria, providing a means for preventing issues like the
recent e-coli outbreaks in tomatoes, beef, and spinach or
salmonella contamination in peanut butter. While there are
hyperspectral systems for these applications, they are too
bulky and complicated to be deployed to the supermarket, the
field or the farm.
The technology has applications in environmental monitoring;
for example, tracking pollutants from hazardous spills or moni-
toring smokestack output. In medicine, the devices could be
used in fluorescence imaging spectroscopy to detect tumors
in vivo or to monitor tissue oxygenation. The economies of scale
of the coating process make it easy to field a scanner affordable
enough to be placed in all surgical suites or even the bedside, on
an average farmer’s tractor, or at a packing house.
6 Summary
The described patterned dichroic filter array approach
takes a different path for multispectral imaging, from a sci-
ence experiment to a commercial product. It is a lower-cost,
higher-volume solution that is the perfect fit for a range of
applications. It can’t produce spectra at a thousand wave-
lengths but the target applications do not need that many
wavelengths. In fact, most applications require between 3 and 6
wavelength bands plus a clear channel. In most cases the target
customers have already performed hyperspectral imaging
using traditional systems and they have already determined
the wavelength bands they need. In addition, they really
need an answer to if the all of tumor has been removed or if
that leaf is contaminated, not a spectra. This technique, with the
unique ability to tailor an image and a spectral response for a
12 | Ocean Optics E-book
particular application, may open the doorway to a new class
of valuable photonic solutions not addressable by today’s
current product offerings. They need a multispectral imaging
product they can utilize at an affordable price, and litho-
graphically patterned dichroic filter arrays provide the answer.
References
Sterk, A. A., “Multispectral pushbroom array imager”, Sensor
Systems for the 80’s Conference, Technical Papers. (A81-
13351 03-35) 109-112, (1980).
[2] Vo-Dinh, T, Moreau, and F, Hueber, D.M. “Multispectral
imaging system using acousto-optic tunable filter (AOTF)”,
Proc. SPIE Vol. 2819, 268-273 (1996).
[3] Stevenson, B. P.; Kendall, W. B.; Stellman, Christopher M.;
and Olchowski, F. M., “PHIRST light: a liquid crystal tunable filter
hyperspectral sensor” Proc SPIE, 5093, 104-113 (2003).
[4] Mathews, S.A., ‘Design and fabrication of a low-cost,
multispectral imaging system”, Applied Optics, 47, F71
(2008).
[5] Themelis, G., Yoo, J.S., and Ntziachristos, V. “Multispectral
imaging using multiple-bandpass filters” Optics Letters, 33,
1023 (2008).
[6] Ford, B., Descour, M., and Lynch, R. “Large-image-format
computed tomography imaging spectrometer for fluores-
cence microscopy”, Optics Express, 9, 444-453 (2001).
[7] Volin, C.E. Ford, B.K., Descour, M.R., Garcia, J.P., Wilson, D.W.,
Maker, P.D., and Bearman, G.H., “High-speed spectral imager for
imaging transient fluorescence phenomena”, Applied Optics,
37, 8112 (1998).
[8] Bayer B.E., “Color Imaging Array”, US Patent 3,971,065, July
1976.
[9] P. Buchsbaum “Method of Making Dichroic Filter Arrays”, US
Patent 5,711,889, January 1998.
[10] P. Buchsbaum “Dichroic Filter Detector Arrays for Spectro-
scopic Imaging” US Patent 6,638,668, October 2003.
[11] P. Buchsbaum “Tunable Variable Bandpass Optical Filter” US
Patent 6,700,690 March 2004.
[12] Heim, G.B., Burkepile, J. and Frame W.W., “Low-Light-Level
EMCCD Color Camera”, Proc SPIE 6209, 62090F (2006).
[13] Brady, D.J., [Optical Imaging and Spectroscopy] Wiley Inter-
science Publishers p295-300, (2009).
High-contrast
subcutaneous
vein detection and
localization using
multispectral imaging
Fengtao Wang, a Ali behrooz, a
Michael Morris, b Ali Adibi, a
aGeorgia Institute of Technology, School of Electrical and Computer
Engineering, 777 Atlantic Drive, Atlanta, Georgia 30332
bOcean Thin Films Inc., Golden, Colorado 80403
Abstract
Multispectral imaging has shown promise in sub-cutaneous
vein detection and localization in human sub-jects. While many
limitations of single-wavelength methods are addressed in
multispectral vein detection methods, their performance is still
limited by artifacts arising from back-ground skin reflectance
and optimality of postprocessing algorithms. We propose a
background removal technique that enhances the contrast and
performance of multispectral vein detection. We use images ac-
quired at visible wave-lengths as reference for removing skin re-
flectance back-ground from subcutaneous structures in near-in-
frared images. Results are validated by experiments on human
sub-jects. ©The Authors. Published by SPIE under a Creative Commons
Attribution 3.0 Unported License. Distribution or reproduction of this work in
whole or in part requires full attribution of the original publication, including
its DOI. [DOI:10.1117/1.JBO.18.5.050504]
Keywords: multispectral imaging; medical optics instrumentation;
sub-cutaneous imaging; vein detection.
Paper 12806LRR received Dec. 18, 2012; revised manuscript received
Apr. 10, 2013; accepted for publication Apr. 11, 2013; published on-line
May 6, 2013.
Subcutaneous vein detection using near-infrared (NIR) imag-
ing has recently become a subject of study.1–4 Optical absorp-
tion of human skin and muscle plummets in the NIR window
allowing for light to travel to subcutaneous depths before losing
coherence and directionality to diffusion.5,6 Unlike muscle and
skin, blood is a strong absorber of NIR radiation7 which contrasts
the subcutaneous vessels against skin and muscle in NIR imag-
ing. As a result, automatic subcutaneous vein detection has be-
come possible using NIR imaging with extensive application in
catheter insertion at health care facilities.8 It has been shown that
multispectral imaging enhances the performance of automatic
vein detection as it collects multiple layers of information that
are used in image postprocessing software to localize sub-cu-
Address all correspondence to: Fengtao Wang, Georgia Institute of Tech-
nology,School of Electrical and Computer Engineering, 777 Atlantic
Drive, Atlanta,Georgia 30332. Tel: 404 934 0265;
E-mail: wangft@yahoo.com
13 | Ocean Optics E-book
taneous vasculature.4,8 However, optimal performance of mul-
tispectral methods depends on their postprocessing algorithms
that require optimal parameter selection and are susceptible to
artifacts from non-vein background. Additionally, multispectral
methods which use postprocessing software merely detect the
subcutaneous vasculature map without retaining other anatom-
ical information. In this letter, we apply a background removal
and normalization scheme to multispectral vein imaging that
circumvents the limitations of other methods and allows for
high-contrast subcutaneous vein detection while extracting
on vein subcutaneous physiological and anatomical structures.
At visible wavelengths, images of human limbs mainly entail
reflectance from skin, whereas at NIR wavelengths, images are
comprised of subcutaneously penetrated light in addition to
skin reflectance. To extract the subcutaneous components and
remove the skin reflectance, referred to as background herein,
from NIR images, we can use the visible images as reference.
Visible images taken jointly with NIR images using a multispec-
tral imager are used in a normalized subtraction algorithm to
have the background removed from NIR images and bring out
the subcutaneous structures. The proposed method is applied
to multispectral data obtained by imaging naked arms of hu-
man subjects with different skin tone and texture. The output of
the algorithm is a fused image that primarily contains sub-cu-
taneous components. Multispectral imaging in this work is per-
formed using the Spectrocam™ Multispectral Imaging Camera
(Ocean Thin Films, Golden, Colorado). Spectrocam is an imaging
system based on a high speed rotating filter wheel coupled to
an NIR enhanced CCD camera (a Sony ICX285 sensor) through a
Carl Zeiss Distagon 2.8/25 mm ZF-IR lens, as shown in Fig. 1(a). In
Spectrocam, multiple wavelengths are multiplexed in the time
domain, and as a consequence, the spatial resolution remains
un-adulterated. The filter wheel can accept up to eight discrete
filters. The camera software separates the images from each fil-
ter in real time, and presents the eight spectrally resolved imag-
es. In this work, five different filters with the custom-designed
spectral bands are utilized, as shown in Fig. 1(b). Narrow-band
filters at 546 and 570 nm correspond to the de-oxy and oxy
forms of hemoglobin, respectively. The 475 nm broadband filter
encompasses the strong absorption bands of melanin, beta-car-
otene, and hemoglobin. The 615 nm filter covers a region of little
absorbance by skin pigments. The 850 nm filter covers the NIR
band that includes absorption by lipids but excludes absorption
by water.
In the first experiment, Spectrocam was used to perform
multispectral imaging on the anterior left forearm of an Asian
male under indoor ambient-light environment (without special
light source), as shown in Fig. 2. The arm was held still before a
dark background. The exposure time under each individual filter
is set to 10 ms, and altogether the five spectral images are ob-
tained within 100 ms. Thus, image co-registration from different
spectral bands, i.e., motion artifact analysis, is not necessary, es-
pecially for still human subjects. As depicted in Fig. 2, it is clear
that the unfiltered image and the visible image, shown in Fig.
2(b) and 2(d), have lower vein contrast compared to the NIR im-
age [shown in Fig. 2(c)]. Mathematically, the vein-to-skin(back-
ground) contrast ratio, denoted by M, can be defined as below,
 |Uvein - Uskin|
M= , (1)
(Uvein - Uskin)

Where Uvein is the average pixel intensity of the vein area,

and Uskin is the average pixel intensity of intact skin around the
veins. The selection of vein and skin ,areas can be done based (1)
on the pixel intensity where four visually selected adjacent skin
(high-value) pixels are averaged to obtain Uskin, and four visually
selected adjacent vein (low-value) pixels are averaged to obtain
Uvein.
Fig. 1 Spectrocam™Multispectral Imager. (a) Picture of the imaging sys-
tem (courtesy of Ocean Thin Films, http://pixelteq.com/product/spec-
tral-cameras). (b) Transmission spectra of filters populating the filter
wheel in the Spectrocam: narrow-band 570 nm, narrow-band 546 nm,
band-pass 800 to 900 nm, band-pass 560 to 690 nm, and band-pass 410
to 540 nm.

To enhance the contrast of veins against cutaneous back-

where Iref represents the reference visible image, I850 nm the NIR
ground, we use the visible images as reference for skin reflec-
image, factor a the scalar weight for achieving the appropriate
tance and subtract from the NIR image. The reference image
brightness in the fused image, and factor b the normalization
must be normalized to match the background component of
weight to enhance vein contrast of the fused image. The opti-
the NIR image. For the multispectral imaging shown in Fig. 2, we
mization of a and b is important for achieving improved visual
used the proposed normalization/subtraction scheme to arrive
at the fused image which, as shown in Fig. 2(e), has a consid-
erably higher visual vein-to-background contrast. The resulting
fused image (with the 615 nm image as the reference), as shown
in Fig. 2(e), has a contrast ratio M of 0.67, considerably higher
than the contrast of 0.037 in the NIR image. Mathematically, the
image fusion process can be expressed as below:

Ifused = a(I850 nm - bIref), (2)

Fig. 3 Multispectral imaging for subcutaneous vein detection. (a) Un-

Fig. 2 Multispectral imaging for subcutaneous vein detection. (a) Color
image of the anterior left forearm used in multispectral imaging. (b) Un-
filtered white light image with M = 0.01. (c) near-infrared (NIR) image at
850 nm with M = 0.033. (d) Visible image at 615 nm with M = 0.025. (e)
Fused image of 850 and 615 nm with a contrastratio of M = 0.67, and
parameter values of a = 3, and b = 0.64.
filtered white light image with M = 0.04. (b) NIR image at 850 nm with
M = 0.06. (c) Visible images at 475 nm with M = 0.01. (d) Fused images
of 850 and 475 nm with M = 0.61. (e) Visible image at 546 nm with M =
0.002. (f ) Fused image of 850 and 546 nm with M = 0.77. (g) Visible image
at 570 nm with M = 0.009. (h) Fused imageof 850 and 570 nm with M =
0.82. (i) Visible image at 615 nm with M = 0.04. (j) Fused images of 850
and 615 nm with M = 0.47.

14 | Ocean Optics E-book

 Fig. 4 Multispectral im-
aging for subcutane-
ous vein detection on a
hairy anterior forearm.
(a) Color image of the
anterior left forearm
used in multispectral
imaging. (b) Unfiltered
white light image with
M=0.02. (c) NIR image
at 850 nm with M =
0.02. (d) Visible image
at 615 nm with M =
0.01. (h) Fused image of
850 and 615 nm with a
contrast ratio of M = 0.3
in the area with lower
hair density and param-
eter values of a = 2.6,
and b = 0.37.
contrast and brightness in the fused image. In each experiment,
the ballpark optimal values for these parameters are selected
automatically by the algorithm. The scaling factor a is simply
adjusted to map the highest pixel value in the fused image to
the maximum non saturated value in the display scale. For the
normalization factor b, the ballpark value is selected using the
approximate expression below
b = (A850 nm/Avis)[(Mfused - M850 nm)/(Mfused - Mvis)],
Where A850 nm and Avis are average total intensities of the NIR
and reference visible image. For Mfused, a desired ballpark value
(0.5 in our studies) can be used. Nevertheless, to achieve maxi-
mal contrast in the fused images, manual fine-tuning is also per-
formed around the automatically selected optimal values of a
(3)
and b. In practice, the same procedure can be used for optimi-
zation of factors a and b in the clinic where manual adjustment
around the automatic ballpark optimal values can be performed
by the healthcare specialist using immediate visual feedback
from the resulting fused image.
n the second experiment, the proposed technique is applied
to the data from multispectral imaging of the anterior forearm
of a Caucasian male with very low body hair density. As shown
in Fig. 3, a multitude of visible images at 475, 546, 570, and615
nm are acquired and used as the reference images separatelyin
the background removal technique. The images shown in Fig.
3(d), 3(f ), 3(h), and 3(j) are the resulting fused images of the NIR
image, shown in Fig. 3(c), and visible images at 475, 546, 570,
and 615 nm shown in Fig. 3(c), 3(e), 3(g), and 3(i), respectively.
The vein contrast ratios of all four fused images are at least an
order of magnitude improved com-pared to the NIR image. This
demonstrates the low sensitivity of the proposed method to the
reference visible image wavelength. Our results also show that
the performance of the background removal algorithm does not
strictly depend on the spectral band selection of the reference
visible image as long as it is within a reasonable range (~150
nm) of mid-visible. Also, among the advantages of the proposed
technique is that, unlike other multi-spectral methods, the NIR
filter used in the proposed technique does not require sharp
15 | Ocean Optics E-book
narrow-band transmission spectrum. Even with partial overlap
of the NIR filter transmission spectrum with visible background
spectral window, the contrast of the fused image remains high
as the background components are removed from the NIR im-
age and the subcutaneous information not present in the visible
images are retained.
Body hair has a strong reflectance glare and can severely im-
pair the vein contrast. To study the effect of excessive density of
body hair on the proposed method, anterior forearm of a Cauca-
sian male with a high density of gray hair was imaged by Spectro-
cam as shown in Fig. 4. The strong glare of the body hair blocks
the skin underneath, and the subcutaneous veins have a very
low contrast ratio and are almost visually unnoticeable in the
NIR image. The fused image, as shown in Fig. 4(e), still provides
high vein contrast enhancement for the skin area with moder-
ate hair density, while the subcutaneous veins in high-density
hair area remain obstructed. Thus, the pro-posed technique per-
forms well for a moderate density of body hair as shown in Fig.
4. However, for subjects with excessive density of body hair, the
skin must be shaved before using Spectrocam. The cross-polar-
izer technique is also useful to mitigate hair glare, which shall be
addressed in a future study.
In conclusion, we demonstrated that the resulting fused im-
ages of three different human subjects with different skin tone
all entail clear subcutaneous vasculature maps with contrast en-
hancement of an order or higher, compared to the NIR images,
while retaining the surrounding non-vein physiological struc-
tures. The high-contrast and reliable performance of the pro-
posed background removal technique for vein detection and
localization in subjects with low to moderate hair density makes
it extremely useful for relevant health care applications.
Acknowledgements
This work was supported by the National Institutes of
Health(NIH) under the project of “Real-Time Multispectral Ear-
ly-Stage Pressure Ulcer Detector” (Ref. 9).
References
1. H.D. Zeman et al.,“Prototype vein contrast enhancer,” Opt. Eng.
44(8),086401 (2005).
2. V. C. Paquit et al.,“Near-infrared imaging and structured light rang-
ingfor automatic catheter insertion,” Proc. SPIE 6141, 61411T (2006).
3. V. C. Paquit et al.,“Combining near-infrared illuminants to optimizeve-
nous imaging,” Proc. SPIE 6509, 65090H (2007).
4. F. P. Wieringa et al.,“Contactless multiple wavelength photoplethys-
mo-graphic imaging: a first step toward“SpO2camera”technology,”
Ann. Biomed. Eng. 33(8), 1034–1041 (2005).
5. V. V. Tuchin, S. R. Utz, and I. V. Yaroslavsky,“Tissue optics, light dis-tri-
bution and spectroscopy,” Opt. Eng. 33(10), 3178–3188 (1994).
6. R. Anderson and J. Parrish,“The optics of human skin,” J. Investig. Der-
matol. 77(1), 13–19 (1981).
7. S. Wray et al.,“Characterization of the near infrared absorption spec-
traof cytochromeaa3and haemoglobin for the non-invasive monitoring
ofcerebral oxygenation,” Biophys. Acta 933(1), 184–192 (1988).
8. V. C. Paquit et al.,“3D and multispectral imaging for subcutaneous
veinsdetection,” Opt. Express 17(14), 11360–11365 (2009).
9. D. Yi et al.,“Real-time multispectral imager for home-based health
care,” IEEE Trans. Biomed. Eng. 58(3), 736–740 (2011)