Ways To Integrate Digital and Physical Media By Using Middleware

Zhen Li ID:1299788 UPI:zli118 zli118@aucklanduni.ac.nz

ABSTRACT

There are more and more media information has been transferred into digital format, but the paper product still cannot be replaced. Those two are total different type, people will use them depends on different situation. Therefore, there is a gap between digital and paper media. How to seamless connect with them is a hard topic, many scientists and researchers work on this topic. There are several different ways to integrate with physical and digital media has been found, some of approaches are matured and can be popularized, some of them just a prototype. However, those approaches are keeping development, and they will be matured and popular in our life. In this paper, several methodologies will be introduced: using middleware such as cell phone and which connect with digital document and physical document; combination of virtual and real document organization, using cabinets and computer system to create an archives management system; and providing graphic user interface allows users interact with digital media and physical resources, such as using digital pen and paper or projector and camera that adopts image processing technology. Although they use different methodologies, the nal goal is that connect physical and digital media together. This paper will discuss those approaches include their advantages and limitations.
Author Keywords

tains audio and movie which people can listen and watch. The paper product only contains words and images. Both media has their own advantages and disadvantages. People use different media depends on their situations. If they want to search information, they will choose to use digital media that provide fast search functionality; if they sit on train or bus, they prefer reading paper product. Somehow people usually exchange digital and physical document. For example, scan paper document, convert them into digital format, or print out digital document for reading. However, people want to integrate both digital and physical media together. Even digital and physical media are closely related to each other, it is not doubt that there is a gap between them. The problem is digital require electric device to input or output. And paper products are totally different, they use pen as tools and display by themselves. Total different objects are hard to link together. In the digital word, key word can link with music, movie or other text, just require simply click. In the physical word, they cannot create link connect with different media. How to combine with advantage of both types of media is a question. Many sciences and researchers try to use different equipments as bridge cross those two. This paper will introduce several literatures which adopt different approaches to achieve the goal of combining with digital and physical media. The reminder of this paper is presented as follows. First of all I will discuss current problem relate to mix physical and digital media, and then depict ndings mentioned in several literatures in nding section. And then discuss their methodologies include advantages and limitations. Finally, give a brief conclusion and identify gaps and shortcomings for this topic at the end.
PROBLEMS

Cell phone, Marker,Camera and Projector, Digital Pen, Electronic Devices

ACM Classication Keywords

HCI, interaction with digital and physical media

INTRODUCTION

Despite more and more media information transfer to digital format such as World Wide Web (www) or digital image and audio data, traditional paper document still use widely. It is unlikely that digital media will replace traditional paper, because paper has many advantages over digital documents. For example, paper can easily carrying, bending, provides high resolution text and images, people can read and write anywhere. However, every coins has two sides, if resource contains lots information, then paper products will be very heavy, and it is hard to search key information from huge paper works. In that point, digital media has advantages over paper products. For instance, people can search key information and make link, edit context easily; digital media con1

Although people can watch and manipulate both digital and physical media, those two does not exist same domination. Digital media require computer or other electronic device convert to human understand format. It is hard to directly connect physical media with digital media. To solve this problem, we need a middleware between those two objects. The middleware can be any tools that people can use. However, the sub-problem is how to make the middleware efcient enough that can be accept by people. Sometimes the idea is very simple, but achieve the idea is very hard. For example, user knows there is a digital copy on server and want to retrieval it, then use cell phone take a picture from

paper, but the returning results may not be correct or reject by server. Therefore, using cell phone is good idea for integrating physical and digital media together, but there are many sub problem have to be considered. Many years, scientist and researchers have tried lots of ways to overcome this problem. They adopt newest technology combine them together to achieve their goal. In the following section, some scientist literatures will be presented, which include interesting nding and methodology.
FINDING

Camera and projector

Thirdly, Chih-Sung Wu and his colleagues [7] present a prototype system called WikiTUI, which combines users reading habits with digital information. It allows reader to access book and interacts with digital media at the same time.
Physical electronic device combine with physical documents

As pervious section mentioned, the general idea is creating a middleware between physical and digital media. Therefore combination those two different objects can be achieved by several ways, which can be grouped by four: base on cell phone, digital pen and paper, projector and camera, and physical electronic device combine with physical archive.
Cell phone

Finally, Thomas Seifried and his colleagues [5] design a Smart Filing System, which provides benets for linking physical document with digital document by electronically enhancing ling cabinets and folders, and provide management and retrieval software for both digital and paper documents. Above ndings are most popular in current research, there are still many research studies at different aspect. Due to time limitation, this paper only discuss above approaches. In next section, the detail of approaches will be described. Meanwhile the advantages and limitation will also be included.
METHODOLOGIES

First of all, there are three papers describing that use mobile phone as middleware that connects with physical document and digital document or multiple media. Liu and Doermann describe a way to retrieve digital document from physical source by using cell phone in their paper [4]. The main process is that use cell phone takes a picture from physical document, and then retrieve digital version of document from database after processing picture. Finally the digital document will store in cell phone memory. Secondly, Koichi Kamijo and his colleagues [1] also use cell phone as middleware that integrate physical and digital document, but they pay more attention on invisible hyperlink marker rather than image process technology. The invisible hyperlink marker seems like the hyperlink used in web pages. Users can use cell phone or web camera with backlight, take image then analyze image. Finding out the real information behind those markers, and then device can display or download the content. Finally, Qiong Liu and his colleagues [3] depict similar methodology as above, the embedded media markers seems combine the ideas of Koichi et al and Liu and Doermann. The embedded media marker is near transparent marker on the document, it does not affect the document appearance. And users use cell phone take a picture, then associated media will display on the phone.
Digital pen and paper

From above section, the general methodology for those papers is that use some device as middleware. Using cell phone seems quite popular in this research area. The advantages of using cell phone are: rstly, the cell phone is very popular in modern age, almost everyone has own cell phone. Secondly, cell phone is quite light and easily to carry, and it has become more and more powerful and looks like a small personal computer. The basic methodology for using cell phone as middleware is that: rst of all, using cell phone to take a snapshot from paper or book. And then submit information to server after processing the image. Finally, server retrieve related data, and sent them back to cell phone. From above nding section, there are three papers focus on combining physical and digital media by cell phone. However, even they study at same topic, they focus on different aspects. Liu and Doermann focus on using photo analyzing technology. And Koichi et al and Qiong et al focus on how to add special marker on paper and recognized it. The main technology used in Liu and Doermann s paper relies on image processing technology, which analyzes image and process it, and then identify the words on images. (Using indexing, token pairs) Indexing is reduce the surface form of glyphs using low level features such width, height, peaks and valleys for text recognition [4]. For example, due to photo is taken by different angle, the image will be distorted, and the text may appear different shape in photo. The letters such as cl, el may looks like d, li looks like h. And then build a confusion matrix which replace letter to similar appearance, then using this language model to recognize the word in picture. Each alphabet as a token then use token pairs to index, for example searches can be pared to searches for, searches points, searches-are and so on. And then use K-nearest to choose best one to retrieval. In server side, document is ranked according to the hits of token pairs, and then veries the page by using token triplet, if page exists in database then identies it, otherwise reject the query. 2

Secondly, there are two papers describing that use digital pen and paper interact with digital document and physical printout. Chunyuan Liao and his colleagues [2] present a pen-gesture based command system called PapierCraft for paper, it allows user to do several digital operation such as use digital pen to select specic paper document content for several digital operations such as copy paste to separate paper note with a paste mark. Jurgen [6] develops in systematic and inductive empirical process and denes a set of simple interactions for pen and paper user interfaces (PPUIs), which as similar as PapierCraft. It adopts several ideas of PapierCraft. The PPUIs can easily support complex activities, also contribute to new interaction strategies for pen and paper interaction, and create links between physical paper and digital document.

Authors evaluate this page in several different aspects: retrieval using token pairs, token triplet verication, usability, success rate and speed. Finally, they get over 96 precent success rates on document database with 100k pages, and this methodology may apply to non Latin -languages such as Chinese and Japanese in the future. However, the quality of image that taken by cell phone camera is the limitation. Koichi et al and Qing et al realize try to improve the accuracy of photo recognition by adding special marker on paper. When user takes a picture from paper, the per-installed cell software will analyze the marker rather than whole picture. Since those markers have been specially designed, they have two advantages. Firstly, they can be reorganized easily and fast by per-installed cell software. Secondly, they will not disturb the original documents. Koichi Kamijo and his colleagues [1] invent a kind of invisible hyperlink marker, which marker likes the hyperlink used in web pages, but it is invisible for human eyes, unless users use LED (Light Emitting Diode) to illuminate the marker in order to receive the information undercover by invisible marker. The advantage of this invisible marker is that it will not disturb original documents. As Figure 1 show below:

error. And the information extraction speed is quit slowly. Moreover, they require printout has marker on it already, if the paper does not have any marker on it, then this approach cannot do any thing about that. Qiong et al [3]describe similar methodology as above, the embedded media marker is kind of watermarker, it is near transparent marker on the document, it does not affect the document appearance. And users use cell phone take a picture, then associated media will display on the phone. The paper describes a technique to generate embedded media markers on documents, which includes identications features with minimal interruption to the original document. As the Figure 2 show below:

Figure 2. embedded water marker

This technology has many advantages compare with other approach. First of all, the client system only require recognizing the embedded water markers, it does not require complex image process technology. And it contains several different media icons, those media icons have been special designed and can be captured by camera easily. Also EMM can be linked to multiple media such as audio, video, text, webpage and image, which bring lost of benets to end user. This approach is recognize image faster than invisible markerd, but it also requires pre-exist marker on paper. In addition to using cell phone, some researchers concentrate to improve peoples reading experience, which also require integrate physical and digital resource. Chih et al present a prototype system called WikiTUI, it allows reader to access book and interacts with digital media at the same time. The whole system constructed by camera, projector, wiki system, input device, and physical book. Camera and projector are input and output for this system. Camera captures the trace of users gure movement. And the projector project visual pattern on desktop, which allow user to watch or interact. Wiki system contains different format of media such as video, Web links and images. The work environment as the Figure 3 shows. For example, when readers read the book Load of Ring, the projected can play movie beside the page, just require users simply touch a ngertip. After evaluation, there are several limitations: the interface of GUI should change likes page number, different readers have different reading modes; users have to match book page manually; trace problem; and privacy problem. However, the WikiTUI system provide unique experience for reader to 3

Figure 1. invisable marker

Users can use cell phone or web camera with backlight, take image then analyze image. Finding out the real information behind those markers, and then cell phone can display or download the content. There are two technical problem should be solved: overlapping areas and extraction with low specication devices. Fortunately, Authors gure out those problems. They use Skipbit method to solve overlapping problem, and using one invisible ink color to solve the problem of extraction information by low resolution camera. The Skipbit method also allows markers embed more bit than other maker methods. After authors doing some evaluation, they found several limitations: the compatibility between invisible and visible ink is quite low due to high rate of bit

Figure 3. WikiTUI

Figure 4. PapierCraft

reading book, they can get more information from Wiki system on the Internet than information come from book. The movement tracing technology helps users to interact with digital resource in real time. Besides, pen and paper are widely used as input for people editing paper document. Many years ago, some researchers create digital pen system. This device can use just like normally pen and paper. People can get use to it fast. Nowadays, it has been extended into digital pen gesture based command system. This kind of system can do more function such as create link between physical document and digital media, copy or move piece of content. Chunyuan et al and Jurgen work on this kind of system. In their prototype system, digital pen can do even more digital operations. Chunyuan et al [2] present a pen gesture based command system called PapierCraft, user can use digital pen to select specic paper document content for several digital operations such as copy paste to separate paper note with a paste mark. To do this, user rst printout the digital document, the system will automatically register hard copy with original digital version. If user draw a pair of cropping marks to select text or gure in printout by drawing a pigtail. The PapierCraft will capture it and executes command in responding digital document. After that, user can paste the content into other paper note with paste mark. Beside, PapierCraft provide link function, which allows user draw a hyperlink mark on document link to digital version or others. Moreover, there are ve different commands provided by PapierCraft: exception, exception with keyword, exception with command name, hyperlink and stitching. As the Figure 4 shows. The advantage of PapierCraft is that user can learn the command faster, and apply them with high success rate. Jurgen [6] design a new interaction framework by extend exist system called pen and paper user interfaces (PPUIs), 4

which not only support complex activities, but also contribute new interaction strategies for pen and paper interaction. The advantage of this system is that system is not restricted to small surface such as camera and projector based system, user can work on documents anywhere in the same room, because data stored on pen for later processing. There are six core interactions dened as operation for user performance: inking, clicking, moving, altering shape, combing and associating. Those core function as similar as Chunyuan et al s PapierCraft. Besides, it can tag documents by focusing on paper stickers. It looks like traditional paper bookmarks that require using digital paper bookmarks. When users want to create a bookmark, they user stick digital paper bookmarks on printout, and then use associating operation create link between digital and physical documents. Finally write keyword on the bookmark. Once the bookmark has been created, it can be modied such as deleted or removed later on. The advantage is that not only get feedback from physical paper, but also from digital document. Finally, there is other ways to integrate visual and digital media. Thomas Seifried and his colleagues[5] design a smart ling system, which link physical document with digital document, and provide management software for both digital and paper documents. The whole system focuses on organization of the physical and digital document rather than link them together. It constructed two part: one is electronically augmented physical cabinets, folders; another is computer management system for controlling, searching and so on. Moreover, the physical document is often organized the same way as electronic version. So the folder structure will be same. The cabinets contain folders and folders contain les, these are as same as the virtual folder in computer system, it provides three main functionalities: searching, browsing and annotation. Each folder has electrical device such as LED, Button, ARTag Markers and so on, those devices connect with cabinets and computer system. For ex-

ample, the LED can be indicator either when users search le in computer that indicate which folder contains tag le, or users try to nd digital version of le, triggered the LED can display results on the monitor. The example shows as the Figure5:

adopting digital pen gesture based command system interact with digital and physical documents; work environment with camera and projector and electronic smart ling system combine with physical document. Those four ideas open up our imaginations, and indicate the direction for other researchers and scientists to keep study. They may not be perfect at the moment, but we believe they will nd out best solution for people in the future.
FUTURE WORK

Even there are lots of approaches to achieve the goal of integrate digital and physical media. They still are not perfect. Using cell phone as middleware is a smart way that can be accepted by people. However, the image capturing is one of limitations of cell phone. Different angle, resolution and other conditions will affect the image hard to be process, and cause rest of process suspending. Other approaches such as using projector and camera, or digital pen and paper, there will be limited at special work environment. Therefore, researchers and scientists still have to focus on combining the best of techniques to create new methodologies or improving the exits approaches to bridge the gap between digital and physical media in the future.
REFERENCES

1. K. Kamijo, M. Minami, and H. Morikawa. An invisible hyperlink marker. pages 1 10, sep. 2009. http://dx.doi.org/10.1109/ICSPCS.2009. 5306395/.
Figure 5. Smart Filing System

Besides, the Smart Filing System provides synchronize function, written notes in front of folder will display note on the system as well. In traditional archive management system, digital and physical resource normally disconnected, the Smart Filing System creates seamless connection between physical and digital resources, and combines with their advantages together. Users can interact with physical documents as similar as digital system does.
LIMITATION

2. C. Liao, F. Guimbreti` ere, and K. Hinckley. Papiercraft: a command system for interactive paper. In UIST 05: Proceedings of the 18th annual ACM symposium on User interface software and technology, pages 241244, New York, NY, USA, 2005. ACM. http://doi.acm.org.ezproxy.auckland. ac.nz/10.1145/1095034.1095074/. 3. Q. Liu, C. Liao, L. Wilcox, A. Dunnigan, and B. Liew. Embedded media markers: marks on paper that signify associated media. In IUI 10: Proceeding of the 14th international conference on Intelligent user interfaces, pages 149158, New York, NY, USA, 2010. ACM. http://doi.acm.org.ezproxy.auckland. ac.nz/10.1145/1719970.1719992. 4. X. Liu and D. Doermann. Mobile retriever: access to digital documents from their physical source. Int. J. Doc. Anal. Recognit., 11(1):1927, 2008. http://dx.doi.org.ezproxy.auckland.ac. nz/10.1007/s10032-008-0066-4/. 5. T. Seifried, M. Jervis, M. Haller, M. Masoodian, and N. Villar. Integration of virtual and real document organization. In TEI 08: Proceedings of the 2nd international conference on Tangible and embedded interaction, pages 8188, New York, NY, USA, 2008. ACM. http://doi.acm.org.ezproxy.auckland. ac.nz/10.1145/1347390.1347410/. 5

The limitation for digital pen and paper approaches and using camera and projector as the equipment is quite expensive and necessary, and limit at special work environment. Building a Smart ling system require lots of complex works, and it only suitable for small business company. If the archive is huge such as library, it will be difcult for user to interact with physical and digital media. Therefore, using cell phone will be better than those approaches. However, using cell phone have to consider how to take a valuable picture, because angle, light and other factor will affect the quality of image, and make them hard to analyze later on.
CONCLUSION

Crossing the gap between digital and physical can be achieved many ways. This paper only discusses four of them. Using cell phone based on marker or image process technology;

6. J. Steimle. Designing pen-and-paper user interfaces for interaction with documents. In TEI 09: Proceedings of the 3rd International Conference on Tangible and Embedded Interaction, pages 197204, New York, NY, USA, 2009. ACM. http://doi.acm.org.ezproxy.auckland. ac.nz/10.1145/1517664.1517707/. 7. C.-S. A. Wu, S. J. Robinson, and A. Mazalek. Turning a page on the digital annotation of physical books. In TEI 08: Proceedings of the 2nd international conference on Tangible and embedded interaction, pages 109116, New York, NY, USA, 2008. ACM. http://doi.acm.org.ezproxy.auckland. ac.nz/10.1145/1347390.1347414/.