Cellphone based Optometry using Hybrid Images

RATIONALE
Measuring refractive aberrations of optical systems is widely used industrially and medically. This
is generally done by using sophisticated systems to yield accurate measurements. There has been
a recent trend in attempting to reduce the cost and to increase the portability of such systems,
specifically for ophthalmic use. (Pamplona et al. 2009; Barsky et al. 2004) Pamplona et al. use an
inverse Shack-Hartmann wavefront sensing approach by placing a microlens array over an LCD
display to measure the warping of the ray-beams from the display by simple user interactions.
The idea of my project is to eliminate the need of the microlens array by creating images that are
inverse filtered with a point spread function such that to a patient with that particular PSF, the
image is rendered invisible. The PSF is then searched for with user inputs, until the patient is
satisfied with the resulting image, much like a digital phoropter.
Therefore, I should be able to calculate the wavefront of the patients eye and assign an eyeglass
prescription from the measured refractive error.
Note: The technique does not involve displaying a deconvolved image. This would not work as showing
a deconvolved image properly requires the use of a lenslet array as in Pamplona et al. My technique
involves displaying an image which is inverse filtered, such that the effect is in ability to see rather than
clarity of picture (as in glasses free displays)

IMPACT
If my system is reasonably accurate, it will significantly reduce the cost of optometry (down to
the cost of an inexpensive smartphone) and will make it portable for use in remote areas. In the
developing world, and especially my country, 20% of blindness is caused by uncorrected
refractive error. These errors go uncorrected due to the large cost of auto refractors and
phoropter, instruments traditionally used to measure these.
The application can serve as an ophthalmic thermometer for patients with prescription
eyeglasses. Approximately 40% of the American population is nearsighted and the application
may be used as a first-sign indicator of a change in power.
HYPOTHESES
This research raises two primary questions:
1. (Image Creation) Is it possible to create an image that is invisible to an eye of a specific
PSF without using a lenslet array?
a. Can we approximate the PSF as a simple Gaussian PSF for spherical aberrations?
2. (PSF Search) Is it feasible to search through the parameters of the PSF to arrive at an
accurate description of the optical aberrations of the patient?

ENGINEERING GOALS
The engineering goal of the project is to be able to create a user friendly smartphone application
that is able to give accurate eyeglass prescription. The aim is to minimize user input while
maximizing accuracy.
METHOD
Creation of Simple Hybrid Images
Images that can be seen in one of two ways have been previously created (Oliva et al. 2006).
These images are known as Hybrid Images and are a sum of low frequencies of one image and
high frequencies of another. The Hybrid Image, J is produced as,
= + (1 )
Where A and B are the two images in question, h is the point spread function and * represents
the convolution operation. Figure 1 shows an example image J.

Figure 1 An example of a hybrid image created using a

Gaussian PSF. Image A is of a '5' and image B is of an '8'

A myopic eye with a prescription of > -3.0 D should find the 8 to be totally invisible, given that
the distance and size is held constant. The patient does not need to use a lenslet array to see the
effect. The eye filtered of the 8 automatically.
Creation of Complex Hybrid Images
The above image was created assuming an elementary Gaussian PSF. However due to the pupil
size and complexity of refractive errors, Gaussian PSFs should only be used with spherical
myopia. More complicated PSFs need to be employed. Some examples are shown in Figure 2.

Figure 2 Examples of complex PSFs generated from

Zernike Polynomials

These PSFs were modeled by constructing a wavefront from Zernike polynomials and hence
calculating the PSF. The first 15 Zernike coefficients were used as parameters to produce these.
Conclusion: A patient with a PSF of h shall only be able to see image A, another patient should
be able to see both image A and B.
SEARCHING FOR THE POINT SPREAD FUNCTION
Linear Search
A variety of searching techniques will be used. For elementary Gaussian PSFs, linear searching
will be employed. The application is turned into a questionnaire which will ask the patient what
they see.

Figure 3 shows the probability of a patient being able to see a

particular image as a function of the standard deviation of the
Gaussian PSF.
Figure 4 is a screenshot of the proposed application.

Figure 4 Screenshot of the application

Figure 3 Examples of questionnaire
answers. 70,75,80,85 mm resp.

Figure 5 shows an example of a sprite that were loaded to perform linear search.

Figure 5 Hybrid Image sprites

Machine Learning Based Search

Searching through the 15 Zernike coefficients is not a feasible task. Instead the PSF is searched
through the three primary coefficients (oblique and vertical astigmatism, and defocus
aberration). The rest of the coefficients are predicted based on a machine learning approach on
existing wavefront data. A multi-class SVM can be used to find the nearest PSF. (the modeler
SVM hereafter)
EXPERIMENTAL EVALUATION
Camera Based
The system is used to compute the focal aberration of any optical system. A USB camera with
variable focal length will be used to evaluate the system. A total of 1200 readings (200 for each
class/focal length) will be taken and the focal length will be varied with 6 classes (70 mm - 100
mm, in multiples of 5 mm)
The display on which the images are shown is 445 ppi, the camera is placed at a distance of 0.73
m, recording with a resolution of 1280 x 720 pixels. (1/3" CMOS sensor, focal distance from 0.3
to , f = 1.7)

Figure 6 Display of experimental setup. Note that the focal length of the camera can be controlled electronically.

The collected data will be divided as 80-20 train-test split on a multiclass SVM (the predictor SVM
hereafter). The accuracy will be measured as a percent accuracy for the SVM predictor, using
standard machine learning statistic.

Wavefront Based
Wavefront data was obtained from 15 patients for a total of 30 eyes. This data allows us to
emulate what a person would exactly see given a particular scene. This model will be used as an
analogue for the eye, in an attempt to prove the validity and accuracy of the approach. Again a
combination of predictor and modeler SVMs will be used during the search.
CONCLUSION
The project aims to measure the refractive aberrations using hybrid images. The method flips the
problem to make it user centric and overcomes the problem of using external hardware by
focusing on invisibility rather than clarity. Even though optometrists use several factors to
determine the final prescription, the system provides a low cost method for a preliminary
prescription. I hope that this method can be used in parts of the developing world where
expensive and bulky instruments, and optometrists are scarce.
RISK & SAFETY

Privacy of Data: All data will be obtained in an anonymized form, it is impossible to

reconstruct personal information from such data.

There are no other safety issue regarding this project.

BIBLIOGRAPHY
Liang, J., Grimm, B., Goelz, S., & Bille, J. F. (1994). Objective measurement of wave aberrations
of the human eye with the use of a HartmannShack wave-front sensor. JOSA A, 11(7), 19491957.
Oliva, A., Torralba, A., & Schyns, P. G. (2006). Hybrid images. ACM Transactions on Graphics
(TOG), 25(3), 527-532.
Pamplona, V. F., Mohan, A., Oliveira, M. M., & Raskar, R. (2010, July). NETRA: interactive display
for estimating refractive errors and focal range. InACM Transactions on Graphics (TOG) (Vol. 29,
No. 4, p. 77). ACM.
Goodman, J. W. (2005). Introduction to Fourier optics. Roberts and Company Publishers.
Hearst, M. A., Dumais, S. T., Osman, E., Platt, J., & Scholkopf, B. (1998). Support vector
machines. Intelligent Systems and their Applications, IEEE,13(4), 18-28.