Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

Messages in the Medium: An Experimental Investigation of Web Advertising Effectiveness and Attitudes toward Web Content
Jeffrey Parsons, Katherine Gallagher, and K. Dale Foster Faculty of Business Administration Memorial University of Newfoundland St. Johns, NF, A1B 3X5, Canada

Abstract Although there is much practical interest in Web advertising as a component of marketing on the Internet, there is little theory or evidence about how it works in the context of Web content. This paper reports a laboratory experiment that examines the impact of advertising format (less or more intrusive), medium (print or Web) and Web content structure (linear or non-linear) on a broad range of measures of advertising effectiveness and evaluation of Web content. Several hypotheses based on human information processing considerations are developed and tested. The results show that advertising effectiveness is constant across advertising formats, media, and Web content structures. Evaluation of content was found to be unaffected by medium or Web content structure, but positive related to the presence of advertising, and negatively associated with the level of intrusiveness of advertising format.

1. Introduction
In just a few years, advertising on the Internet has developed from an unacceptable activity to a widely used and highly visible presence on the World Wide Web. In particular, advertising has become closely associated with many types of Web content, ranging from search engines and Web directories (such as Yahoo! and AltaVista) to online news and entertainment services (such as CNN and People Online). Since users expect much Web content to be free and are generally unwilling to pay for it [8], many content providers rely on advertising as a source of revenue to reduce or eliminate the direct cost to users of their content. The quantity and quality of advertising on the Internet has changed with the evolution of Web technology. Early advertising consisted of (usually unwelcome) text-only messages posted to Usenet

newsgroups. As the Web and Web browsers gained in popularity, graphical banner ads became widely used in Web pages. Banners quickly evolved to include animation and sound. Recently, banners appearing as pop-ups in new browser windows have increased the prominence of on-line advertising. The Web has penetrated the general population in industrialized countries much more rapidly than did other communication technologies when they were introduced. For instance, radio achieved 50 million users 38 years after its introduction, television after 13 years, and the Internet after only four [4]. This rapid diffusion of a new communications medium provides an unprecedented opportunity to study user responses to advertising and content in that medium. This paper describes an experiment that examines user responses to advertising and information content on the Web under varying forms of content and advertising structure. Section 2 offers a framework that motivates a number of propositions concerning user responses to information and advertising on the Web and in traditional media. Section 3 describes the design of an experiment to operationalize and test the propositions. Section 4 presents the results of the experiment. Section 5 discusses the theoretical importance of the findings. Section 6 concludes by discussing the practical implications of the study and offering suggestions for future research.

2. Motivation for the study

Our understanding of whether and how Web advertising works is in its infancy, but is attracting a lot of attention from both researchers and practitioners. Hoffman and Novak [10] identify the need to adapt and reconstruct advertising, as the hypertextual structure of the Web changes consumers from relatively passive targets of marketing strategies to more active participants in the marketing process. Ghose and Dou

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

[6] suggest that firms should take advantage of the potential for interactivity in Web advertising. Grossman [7] proposes that developing commercially effective content for Web sites may require new approaches. Meanwhile, there is evidence that many advertisers are not taking full advantage of the capabilities of the Web as an advertising medium. A global study of 1,800 commercial Web sites from 40 industries [9] found that promotion of goods and services is largely done in conventional ways, such as product news, catalogs and portfolios, previews and samples, special offers and discounts, and contests and sweepstakes. Some advertisers seem to think that any Web site is better than no Web site at all [1]. However, there is also concern among many advertisers about the Webs effectiveness as an advertising medium, and its ability to provide a competitive advantage [2]. In this research, we examine the behavior of Web users with respect to advertising in Web sites versus traditional media, focusing on the impact of different advertising formats and ways of structuring content. For the purpose of characterizing this behavior, we broadly classify Web content as information (including persuasive messages such as advertising) and entertainment. We likewise classify use of Web content as information- or entertainment-seeking. This paper focuses on the information dimension of Web content and the information-seeking behavior of Web users.

2.1 Advertising effectiveness

User reactions to Web advertising in the context of information-seeking activities are of great interest to both advertisers and providers of Web content who use advertising as a source of revenue. Perhaps the most basic issue of interest is whether advertising works at all in an online environment. The effectiveness of advertising in traditional media is well established. There is also evidence that banner ads are effective even if users do not follow the banners to the sponsors Web site [11]. Hence: Proposition 1: Advertising is effective (both in traditional media and on the Web). The unique qualities of the Web as a communication medium raise interesting questions about the relative effectiveness of advertising in comparison to traditional media, as well as differential effectiveness under varying conditions of Web content linearity. Gallagher, Foster, and Parsons [5] found no difference in the effectiveness of identical advertising appearing in equivalent print and hypertext material, even though evaluations of the hypertext material were significantly more negative than those of identical print material. That finding suggests that users employ

different information processing strategies for dealing with content and advertising, possibly because they must process advertising very efficiently in order to cope with the large amount of it they are exposed to on a daily basis. The efficiency may involve quickly determining whether the advertised product is personally relevant to the consumer at the time the material is encountered. We expect that this imperative is strong enough to preclude differences in advertising effectiveness due to medium. Hence: Proposition 2: There will be no difference in advertising effectiveness between traditional media and the Web. Proposition 2 is silent about possible differences in Web advertising effectiveness depending on the extent to which the material containing the advertising makes use of the potential non-linear structure of hypertext. However, if consumers indeed employ different processing strategies for advertising and content, the degree of interactivity of the content in which the advertising is embedded should have no bearing on the effectiveness of that advertising. Therefore: Proposition 3: There will be no difference in advertising effectiveness between linear and non-linear hypertext content. In addition to understanding whether advertising on the Web works and whether it is affected by the structure of Web content, another issue facing Web advertisers is whether one form of Web advertising is superior to another. In particular, some Web advertising formats are more disruptive to information seeking than others. For example, pop-up ads appear in a new browser window and require the user to take some action (e.g., close the pop-up window) to return to the original information-gathering activity. In contrast, banner ads can be ignored by scrolling beyond the banner. Nevertheless, the differential processing strategy discussed above suggests that, regardless of the level of advertising disruptiveness, advertising will be recognized as advertising and processed accordingly. Hence: Proposition 4: There will be no difference in Web advertising effectiveness due to the level of disruptiveness of the advertising.

2.2 Evaluation of web content

In addition to understanding advertising effectiveness on the Web, content providers must be concerned about issues related to customer evaluations of the content itself, and whether those evaluations are affected by the presence of advertisements. One important objective of Web content providers is to develop content appropriate for the medium. Providing content on the Web can be distinguished

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

from the providing content in other media in several ways. First, the Web combines elements of traditional media such as print, radio, and television. A Web site can include text, graphics, audio, and/or video. Therefore, the Web can be seen as a richer information medium than earlier media [10]. On this basis, one might expect that information presented on the Web would be preferred over equivalent information presented in a traditional medium. However, the simple notion that media richness determines user evaluation ignores the potential roles of user expectations associated with a given medium, and of effective use of the medium. A second major way in which content on the Web differs from that of traditional media is in the way in which the material is presented to and processed by users. In traditional media, information presentation is inherently linear, with the order of traversal largely determined by the content creator. Information consumers have only limited ability to control exposure to information by skipping (e.g., portions of an article or of a taped audio or video program) or changing channels (switching to another article or switching stations). On the Web, however, the order of exposure to information need not be linear. Appropriate use of hyperlinks by content creators provides Web users with much greater control over the information they are exposed to and the order in which they are exposed to it [12]. Users can become active participants in the construction of information content suited to their information needs [10]. Experienced Web users will be aware of the different capabilities of the Web versus traditional media as described above. We therefore expect they will hold a set of expectations or preconceptions regarding the appropriate structure of information in traditional and online media which, in turn, affect their subjective evaluation of the quality of content presented in different media. In particular, users will expect Web content to support non-linear navigation and to offer a richer multimedia experience. Therefore: Proposition 5: Content presented in a traditional medium will be evaluated more positively than the same content presented linearly on a Web page. Likewise: Proposition 6: Web content will be evaluated more positively when it supports non-linear navigation than when it simply duplicates the linear structure of a traditional medium. In addition to providing content appropriately structured in relation to the medium, another issue facing Web content providers is how to support users in information gathering. Unlike Web use for entertainment, use for information gathering is clearly

goal-oriented. In a goal-oriented information search, we expect that satisfaction with the experience (or information gathered) will be influenced by the extent to which the interactive environment supports the task. To the extent that the interaction with the material disrupts or interrupts the task at hand, users evaluations of the information they are exposed to will be more negative. As mentioned earlier, information-seeking on the Web can be interrupted by online advertising. Prior research on advertising effectiveness suggests that the mere presence of advertising in online content can negatively affect user evaluations of the content [5]. Therefore: Proposition 7: Web content will be evaluated more positively without advertising than with advertising. In addition, advertising content on the Web can cause varying levels of disruption (e.g., banner ads versus pop-ups). Therefore, we expect: Proposition 8: Web content will be evaluated more negatively the more disruptive the advertising included with the content.

3. Experimental design and hypotheses

In this section, we describe an experimental study designed to operationalize and test the propositions developed in Section 2. The study compared advertising effectiveness and consumer evaluation of content in print, linear Web design, and non-linear Web design, using varying forms of advertising appropriate to each medium. Independent Variables and Experimental Materials: A primary independent variable of interest was the medium in which content is presented. A natural medium for Web comparisons is print, since the text and graphics of print correspond to the most popular format of Web material and, therefore, print material can be easily transformed to the Web. However, Web content can be structured in a multitude of ways that constrain or support user navigation through the content. Therefore, in this study, two levels of Web content linear and frames were developed. The linear form structured the content in the same way as the print condition, with each Web page having the content of a corresponding printed page. The only navigation permitted between pages allowed users to select a continue link at the bottom of a page to proceed linearly to the next page of material. The frame version divided the content according to subsections, with each subsection listed in a frame on the left side of the page. Users could select topics to view in any order, with the corresponding material appearing in a separate frame, thus allowing a greater degree of non-linear navigation through the material.

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

A second important variable of interest involved the presence and type of advertising used. In the print material, two levels of this variable were usedno advertising or advertising. In the Web material, three levels of this variable were usedno advertising, banner advertising and pop-up advertising. These represented increasing levels of disruptiveness to participants in reading the associated material. The print materials consisted of a promotional brochure in the style of a magazine article, providing information of interest to tourists visiting St. Johns, Newfoundland, a medium-sized Canadian city. The brochure was three (no advertising condition) or four (advertising condition) (8.5x11 inch) pages long, and included color photographs. In addition, the advertising version contained five advertisements for products and services likely to be of interest to tourists visiting the city. The advertisements were distributed throughout the text of the brochure, as opposed to appearing in one location. One of the advertisements was for a fictional coffee shop opening soon in the city. This advertisement was the focus of the measures of advertising effectiveness. The Web materials contained the same text and photographs in several Web pages. The linear version allowed only sequential navigation through the article as a series of Web pages, while the frames version allowed navigation through the article in any order according to subsections of the article which appeared in a frame on the left side of the computer screen. In addition, the banner versions (both linear and frames) contained banner advertisements distributed at various points in the article (in both conditions, each advertisement appeared after the same point in the text of the article). Each banner was a hyperlink leading to a separate Web page that was an advertisement containing additional information about a company operating in the city. The advertisement also contained a back link that returned the reader to the point in the article where the banner appeared. Finally, the pop-up versions (both linear and frames) spawned pop-up advertisements in separate tiled browser windows (550x70 pixels). The pop-ups were spawned according to the current page being visited, with the ordering and location corresponding to the ordering and location of banners in the banner versions. Each pop-up was a hyperlink that, if selected, spawned a new full-screen browser window containing the appropriate advertisement. The advertisements were identical to those in the banner conditions, except there was no back link. Instead, highlighting the notion of task interruption, participants needed to close the browser window or select the correct browser from the Windows task bar in order to return to the article.

Participants: In this research, we compare users' evaluation of content and advertising effectiveness in both print and Web formats. As the Web audience is growing in both numbers and demographic diversity, there is currently considerable variation within this audience in terms of comfort and familiarity with the Web. This variation has been identified as a potential confound in studies of this nature [3]. In the medium to long term, there will be fewer and fewer new users, and more and more experienced users. From a research point of view, therefore, we prefer to use more experienced users, as they are likely to be more representative of the typical Web user of the next several years. University students are better representatives of more experienced users than can be recruited through traditional procedures such as mall intercepts. We therefore selected participants from a population of experienced Web users. Students taking business courses at our university are all experienced users, as use of the Web is a requirement in several introductory courses. Procedure: One hundred and seventy-five third-year undergraduate business students volunteered to participate in the study. They were randomly assigned to one of eight groups: Print-No Advertising (PN), Print-Advertising (PA), Linear-No Advertising (LN), Linear-Banner Advertising (LB), Linear-Pop-up Advertising (LP), Frame-No Advertising (FN), FrameBanner Advertising (FB), Frame-Pop-up Advertising (FP). Participants were run in sessions of 20 or fewer. They were told the purpose of the study was to examine whether the communication of information was equally effective in print and Web formats. Participants were asked to read an article about St. Johns as if they were planning a vacation to the city. The instructions made no mention of the accompanying advertising material so that exposure to the advertising would be as natural as possible. Participants were given as much time as they wished to read the article. As each participant completed the article, the administrator either collected the brochure or turned off the computer monitor so that participants were unable to check back to the material as they answered questions about it. Each participant then completed a questionnaire. Dependent Variables: We measured participants evaluation of the article in the brochure or Web site, along with the effectiveness of the accompanying advertising (which participants may or may not have seen and/or noticed). The questionnaire began with several questions about the article. The first ten questions focused on participants attitude to the article as a whole. Participants used a five-point scale (strongly agree to strongly disagree) to indicate how they felt about a

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

range of statements about the article (interesting, annoying, liked, boring, good, entertaining, involving, would enjoy reading again, would be useful to potential visitors to St. Johns). The next four questions required respondents to indicate their attitude to the city of St. Johns as a tourist destination (appealing, enjoyable, satisfying, would travel to St. Johns if planning a vacation). Additional evaluative data were collected via a set of 27 adjectives displayed in alphabetical order. Participants were asked to tick off any adjectives that they felt described the article. The questions about the article were followed by a series of questions about the advertisements that participants may or may not have seen. General evaluation of the advertising was solicited first. Participants were asked if they remembered seeing any advertising. Those who did were then asked to indicate on a five-point scale their level of agreement with four general statements about the advertising in the article (attention-getting, distracting, suitable for article, more interesting than the article). Next, unaided recall was measured among those participants who remembered seeing advertisements in the article. Participants were asked to list as many of the types of advertised products as they could recall, and for each product type, to list the brand name. The questions then focused in greater detail on one of the five advertisements, which announced a coffee shop opening soon. Only the logo was presented in the questionnaire. The logo did not suggest the type of product or business represented. First, aided recall was measured. Participants were asked whether they recognized the logo (if not, they went on to the next section), what kind of business was being advertised, and the name of the business being advertised. Then,

attitude toward the ad was measured by asking participants to indicate on a five-point scale their level of agreement with nine statements about the ad. These statements were modifications of nine of the ten questions about the article, substituting the word ad for article (the question not paralleled was whether this would be useful to potential visitors to St. Johns). Next, the 27 descriptive adjectives used earlier in relation to the article were presented again. Participants ticked off any adjectives that they felt described the ad. Demographic and Web usage data were also collected. Finally, the effectiveness of the coffee ad was measured using a combination of attitudinal, behavior intention, and behavioral measures. For the attitudinal measure, participants were asked to indicate, on a fivepoint scale, their level of agreement with three statements about each of the advertised coffee brand and a fictitious coffee brand (impression about the brand, quality of the brand, taste of the brand). For the behavioral intention measure, participants were asked to indicate for each brand, their level of agreement with a statement about whether they would purchase the brand. The behavioral measure of advertising effectiveness consisted of asking participants to choose a sample of ground gourmet coffee to take home, either the advertised coffee shop brand or the fictitious brand (neither currently available or known in the area). The strength of participants preference for the chosen brand was measured on a seven-point scale with one brand at each end. Operational Hypotheses: The propositions developed in Section 2 can now be restated in terms of the experimental groups and dependent variables identified above (See Table 1).

Table 1. Operational Hypotheses Theoretical Proposition Operational Hypothesis* Advertising is effective H1: A(Advertising) > A(No Advertising) No difference in advertising effectiveness between H2: A(Print) = A(Web) traditional media and the Web No difference in advertising effectiveness between H3: A(Linear) = A(Frames) linear and non-linear hypertext content No difference in advertising effectiveness due to H4: A(Banners) = A(Pop-ups) varying levels of advertising disruptiveness Evaluation of content will be more positive in a H5: E(Print) > E(Linear Web) traditional medium than in linear Web format Evaluation of content will be more positive in a non- H6: E(Frames) > E(Linear) linear Web format than in a Linear Web format Evaluation of Web content will be more positive H7: E(Web-No Advg) > E(Web-Advg) without advertising than with advertising Evaluation of content will be more negative the more H8: E(Pop-ups) < E(Banners) disruptive the advertising included in the content
*A(X) denotes an overall measure of effectiveness of advertising under condition X, while E(X) denotes an overall measure of the evaluations of the article in condition X.

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

4. Results
4.1 Advertising effectiveness
Hypotheses 1 through 4 deal with advertising effectiveness. Note that participants in the no advertising groups (PN, LN, FN) did not see any advertisements. Additionally, the instructions to participants made no reference to advertisements, so participants in the advertising groups may have ignored the advertisements, or may not have followed the links from the banners or pop-ups to the Web pages with additional product information. Hypothesis 1 states that advertising is effective. Since participants in the no advertising groups were not exposed to the advertisements, their scores on measures of recall of the advertisements was zero. Therefore, we focus the analysis for this hypothesis on the effectiveness of the ads in influencing attitudes, behavioral intentions, and behavior with respect to the target brand (Caribou Coffee). As Table 2 shows, compared to participants who saw no advertisements, participants exposed to advertisements had more positive attitudes toward the brand (t131.802 = 2.428, p < .01) and had higher behavioral intention toward the brand (t170 = -2.537, p < .01). Moreover, a significantly greater proportion of participants who were exposed to advertising chose the target brand of coffee (50% vs. 33%) (
2 = 3.839, p = .05), and the strength of preference for that choice was greater for participants exposed to advertising (t160 = -1.631, p = .05). Collectively, these results show strong support for Hypothesis 1. Hypothesis 2 predicts that there will be no difference in advertising effectiveness between Print and Linear Web formats. Since we are comparing here only groups who were exposed to advertising, we use unaided and aided recall in addition to attitude, behavior intention, and behavior as measures of advertising effectiveness. There were no significant differences1 between participants who were exposed to material in a Linear Web format and those who were exposed to Print material on attitudes, behavioral intentions, and behavior with respect to the target brand. There were also no significant differences in the mean number of product categories or of brands recalled, nor were there significant differences in total number and number of positive and negative adjectives used to describe the target ad. However, compared to participants who were exposed to Linear Web material, participants exposed
Due to space restrictions, some tables containing details of the analysis have been omitted. Complete results can be obtained by contacting the lead author.
1

to Print material evaluated the advertising more positively (mean 2.00 vs. 0.94) (t25.570 = 1.914, p < .05). Differences in advertising effectiveness might also be detected by comparing the proportions of each group who remembered seeing any ads and, of those, the proportion who correctly recalled (unaided) that coffee was an advertised product category, and the name of the target brand (Caribou). Differences in advertising effectiveness might also be detected by comparing the proportions of each group who, when prompted with the target brand logo, remembered seeing the logo, recalled that it was for coffee, and recalled the brand name. Finally, differences in effectiveness might be detected by comparing the proportions of each group who selected each of the 27 adjectives available for describing the target advertisement. Across all of these comparisons, only two significant differences were found. Participants exposed to Linear Web materials were more likely than those exposed to the Print materials (43.8% versus 0%) to describe the target advertisement as easy to forget (
2 = 5.987, p = .01), while participants exposed to Print materials were more likely than those exposed to the Linear Web materials (60.0% versus 18.8%) to describe the target ad as informative (
2 = 4.626, p < .05). These results provide no basis to reject Hypothesis 2. Hypothesis 3 predicts no difference in advertising effectiveness between participants exposed to materials in Linear Web and Frames Web formats. For this analysis, we combined participants for each of the advertising formats (banner or pop-up) within Linear and Frames conditions. Since our hypothesis is that there will be no differences, combining the two groups will show significance for smaller differences between the groups. Thus, if we fail to find differences, this is a more stringent test than if we were to separate groups by ad format. We used the same measures of advertising effectiveness used for Hypothesis 2. There were no significant differences in attitudes, behavioral intentions, or behavior between participants exposed to Linear and Frames materials. In addition, there was no significant difference between the Linear and Frames groups in the mean number of product categories recalled. However, participants exposed to Frames materials did recall more brands than those who saw Linear materials more (mean 1.81 vs. 0.92) (t65.934 = -2.829, p < .01). There was no difference in overall evaluation of advertising, overall attitude toward the target ad, total number of adjectives used to describe the article, number of positive adjectives, or number of negative adjectives used to describe the target ad. More participants in the Frames group (29.5%) indicated that they recognized the logo of the

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

Table 2. t-test results for attitude, intention, and behavior measures of advertising effectiveness Mean Std. Dev. Levene F T Sig. No Ad. Ad. No Ad. Ad. Attitude toward brand* -0.046 0.40 1.20 1.12 5.183 -2.428 .008 Behavioral intention ** -0.300 0.009 0.76 0.80 0.417 -2.537 .006 Preference strength*** 0.51 0.73 0.80 0.85 1.877 -1.631 .05
*Scores on this composite item were in the range from 6 to +6. ** Scores on this item were in the range from 2 to +2. ***Score on this item was in the range 0 to 3.

target brand than in the Linear group (20.6%) (
2 = 4.121, p = .047). Notably, however, there was no difference between the groups in the proportions that correctly identified the business or the brand. These results offer no evidence to reject Hypothesis 3. The two significant findings are not more than would be expected by chance with so many variables being tested. There is no difference in advertising effectiveness between Linear and Frames Web conditions across a broad range of measures. Hypothesis 4 predicts that there will be no difference in advertising effectiveness between Banner advertisements and Pop-up advertisements. For the purpose of this analysis, we combine participants in each of the hypertext formats (linear or frames) within Banner and Pop-up groups. Again, since our hypothesis is that there will be no differences, combining the two groups will show significance for smaller differences. We used the same measures of advertising effectiveness as we did for Hypotheses 2 and 3. There were no significant differences in attitudes, behavioral intentions, or behavior between the Banner and Pop-up groups. There were also no significant differences between the Banner and Pop-up groups in the mean number of product categories recalled or in the mean number of brands listed. There was a significant difference in overall evaluation of advertising, with the Banner groups having a higher overall evaluation (mean 1.11 vs. 0.03) (t73 = 2.125 , p < .05). There was no significant difference in overall attitude toward the target ad, or in total number and number of positive and negative adjectives used to describe the target ad. More participants in the Banner group chose the adjectives clever (38.9% vs. 6.7%,
2=4.626, p=.032) and original (44.4% vs 13.3%,
2=3.750, p=.053) than in the Pop-up group. These results show no substantive evidence to reject Hypothesis 4. The three significant findings are not more than would be expected by chance with so many variables being tested. Moreover, the minor significance differences are consistent with anecdotal evidence regarding the general annoyance that Web users seem to feel toward Pop-up advertising. People do not seem to like Pop-up advertising, but on a

compelling range of measures, it is as effective as Banner advertising.

4.2 Evaluation of Web Content

Hypotheses 5 through 8 deal with evaluation of the non-advertising content. The analysis focuses on participants attitude toward the article and their attitude toward the city described in the article, as well as the adjectives they selected from the list of 27 to describe the article. Hypothesis 5 predicts that evaluation of the article's content will be more positive when it appears in Print than when it appears in a Linear Web format. However, there were no significant differences in overall attitudes toward the article, in attitudes toward the city, or number of adjectives selected. When we compare the proportions of each group that selected each of the 27 adjectives available for describing the article, we find some significant differences, as shown in Table 3. A greater proportion of participants in the Linear Web groups chose the adjectives amusing and confusing, while a greater proportion of participants in the Print group chose the adjectives believable, easy to forget, and familiar. Table 3. Cross tabulation of adjectives chosen to describe the article (sig. diffs. only) Percent Sig.
2 Print Linear Amusing 0 6.3 3.356 .034 Believable 78.4 61.9 3.618 .028 Confusing 0 6.3 3.356 .034 Easy to forget 31.4 15.9 3.846 .025 Familiar 43.1 27.0 3.267 .036 Collectively, these results show very little evidence of any differences in evaluations of print and linear Web material. Some differences exist in proportions of participants in each group choosing certain adjectives, but these show no clear pattern. Hence, Hypothesis 5 is rejected. Hypothesis 6 predicts that evaluation of Frame Web content will be more positive than evaluation of Linear Web content. Participants in the Linear groups chose

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

significantly fewer negative adjectives to describe the article than did those in the Frames groups (mean 0.44 vs. 0.75) (t73 = -1.718 , p < .05). However, there were no significant differences in overall attitudes toward the article or in attitudes toward the city of St. Johns. As Table 4 shows, a greater proportion of participants in the Linear Web groups chose the adjectives convincing, fast moving, and well done, while a greater proportion of participants in the Frames group chose the adjectives easy to forget, and pointless. Table 4. Cross tabulation of adjectives chosen to describe the article (sig. diffs. only) Percent Sig.
2 Linear Frames Convincing 30.2 11.5 6.528 .005 Easy to forget 15.9 32.8 4.834 .014 Fast moving 9.5 1.6 3.617 .023 Pointless 0 4.9 3.175 .037 Well done 42.9 23.0 5.549 .009 Collectively, these results show some evidence of differences in evaluations of Linear and Frames Web material. Moreover, those differences are in the direction opposite to that predicted in Hypothesis 6. That is, there was an overall somewhat more positive evaluation of the article by people in the Linear groups. Hence, Hypothesis 6 is rejected. Hypothesis 7 states that evaluation of Web content will be more positive without advertising than with advertising. Participants in the No Advertising Web groups chose significantly more negative adjectives to describe the article than did those in the Advertising Web groups (mean 1.15 vs. 0.56) (t73 = 2.166 , p < .05). There were no significant differences in overall attitudes toward the article or in attitudes toward the city of St. Johns. Table 5 shows that a greater proportion of participants in the No Advertising Web groups chose the adjectives dull, easy to forget, Imaginative, and informative, while a greater proportion of participants in the Advertising Web groups chose the adjective true to life. Collectively, these results show some evidence of differences in evaluations of Web material with and without advertising, but primarily in the opposite direction predicted by Hypothesis 7. That is, there is overall a slightly more positive evaluation of the article by people in the Advertising groups. The exception is that participants found the article more imaginative without advertising. Hence, Hypothesis 7 is rejected.

Table 5. Cross tabulation of adjectives chosen to describe the article (sig. diffs. only) Percent Sig.
2 No Ad Ad Dull 35.0 12.2 4.424 .017 Easy to forget 50.0 24.4 3.617 .023 Imaginative 20.0 2.4 5.509 .009 Informative 70.0 87.8 2.883 .045 True to life 15.0 39.0 3.618 .028 Hypothesis 8 predicts that evaluation of Web content will be more positive the less disruptive the advertising. In other words, participants in the Banner groups should evaluate the article more favorably than participants in the Pop-up groups. Participants in the Banner groups had a significantly more positive overall evaluation of the article (mean 8.20 vs. 5.05) (t73 = 2.832 , p < .01). There were no significant differences in overall attitudes toward the city of St. Johns, or in number of adjectives used. Table 6 shows that a greater proportion of participants in the Banner groups chose the adjectives convincing, lively, and well done, but a smaller proportion chose the adjectives dull and familiar. Table 6. Cross tabulation of adjectives chosen to describe the article (sig. diffs. only) Percent Sig.
2 Banner Pop-up Convincing 36.6 9.8 9.289 .002 Dull 7.3 19.5 2.625 .05 Familiar 9.8 41.5 10.82 <.001 Lively 14.6 2.4 3.905 .024 Well-done 41.5 24.4 2.706 .05 Collectively, these results show some evidence of differences in evaluations of Web material between banner and pop-up advertising, in the direction predicted. That is, there is overall a more positive evaluation of the article by people in the Banner groups. Hence, Hypothesis 8 is supported. Table 7 summarizes the results.

5. Discussion
The results for Hypothesis 1 reaffirm the idea that advertising works. Hypotheses 2 through 4, on the other hand, show no differences in advertising effectiveness across a range of variations in advertising format (less or more instrusive) and in the structure of the vehicle in which the advertising is carried (linear or

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

Theoretical Proposition Advertising is effective No difference in advertising effectiveness between traditional media and the Web No difference in advertising effectiveness between linear and non-linear hypertext content No difference in advertising effectiveness due to varying levels of advertising disruptiveness Evaluation of content will be more positive in a traditional medium than in linear Web format Evaluation of content will be more positive in a non-linear Web format than in a Linear Web format Evaluation of Web content will be more positive without advertising than with advertising Evaluation of content will be more negative the more disruptive the advertising included in the content

Table 7: Summary of Results Operational Hypothesis A(Advg) > A(No Advg) A(Print) = A(Web)

Result A(Advg) > A(No Advg) A(Print) = A(Web)

A(Linear) = A(Frames)

A(Linear) = A(Frames)

A(Banners) = A(Pop-ups)

A(Banners) = A(Pop-ups)

E(Print) > E(Linear Web)

E(Print) = E(Linear Web)

E(Frames) > E(Linear)

E(Frames) = E(Linear)

E(Web-No Advg.) > E(Web-Advg.) E(Pop-ups) < E(Banners)

E(Web-No Advg.) < E(Web-Advg.) E(Pop-ups) < E(Banners)

non-linear). Generally, failure to find significant differences across experimental groups on a range of dependent variables would be disappointing. However, there was an underlying theoretical notion motivating the hypotheses for lack of difference in advertising effectiveness; namely, that people employ different strategies to process advertising and content. Since Hypothesis 1 found that, in general, the advertising was effective, the absence of differences across advertising format and structure of the content in which it was carried is consistent with the use of different processing strategies for advertising and content. We believe further research is needed to strengthen this finding and develop a better understanding of how and why advertising is processed differently from content. The results with respect to evaluation of content are less clear. The lack of significant differences in the evaluation of Print and Linear Web materials (Hypothesis 5) fails to replicate an earlier finding of more negative evaluation of Linear Web material (Gallagher, Foster, and Parsons 1999). The current experiment supports the rather counterintuitive idea that Web users do not appear to expect a greater degree of interactivity in Web content. At a minimum, it indicates that failure to meet user expectations about Web material does not impair user evaluation of the material. In sum, additional theory and research are needed to further determine the circumstances under which differences in expectations exist or matter. In particular, it will be valuable to actively elicit Web user

views on the level of interactivity they expect from Web material. Similarly, the lack of significant differences in the evaluation of Linear and Frames Web material (Hypothesis 6) lends additional credence to the notion that Web users may not expect a greater degree of interactivity in Web content, or that if such expectations exist, they do not affect evaluation of content. The results on the effect of advertising on the evaluation of content are the most surprising in this study. Contrary to the expectation that advertising will negatively affect user evaluations of material (Hypothesis 7), based on the findings of [5], we found weak evidence that the presence of advertising may actual contribute to higher evaluations of Web content. One possible reason for this finding is that participants who received the Web materials containing no advertising felt that the material was less realistic. We expect that, in the year since the prior study was conducted, user familiarity and experience with the Web and the increasing level of advertising it carries, has grown. In that case, the absence of advertising in promotional material for a tourist destination may be viewed as a deficiency. Further research with different types of Web content may be useful in determining whether the subject matter of the content interacts with advertising in determining user evaluations of the material.

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000

Finally, the support for Hypothesis 8 clearly indicates that higher levels of Web advertising intrusiveness negatively affects evaluation of the Web content in which the advertising appears (despite the fact that advertising effectiveness does not appear to be impaired by the level of intrusiveness). This is consistent with anecdotal evidence about the annoyance caused by pop-up advertising, as well as the theoretical notion that intrusiveness or disruption impairs user satisfaction with goal-oriented information gathering. Further research is needed to explore in greater depth the interaction between intrusiveness and content evaluation, including the effect of the perceived relationship between the intrusive material and the content in which it appears.

perspective that higher levels of intrusiveness do not improve advertising effectiveness.

7.

References

6.

Conclusions

We believe this research contributes to building a deeper theoretical understanding of the relationship between content and advertising in an important new communications medium. In addition, the results speak directly to practitioners Web advertisers and content providers who sell Web advertising to support their content. The findings with respect to Web advertising effectiveness suggest that advertisers should consider advertising on the Web. Notwithstanding the low click-through rate, Web advertising is effective. Moreover, advertisers may not need to be concerned about evaluating the interactive quality of Web sites in which they advertise focusing on the exposure to the target market available through the site may be more important. Finally, there does not seem to be any advantage in developing more intrusive forms of Web advertising. From a Web content providers point of view, the advertising effectiveness findings in this study may be useful in helping convince potential advertisers to engage in Web advertising. The study may also help alleviate potential content providers concerns that selling advertising may lead to less use of their content by Web users advertising may in fact improve evaluation of content (particularly where there is a strong relationship between the content and the advertised products). Finally, content providers may want to discourage ever more intrusive forms of Web advertising, not only for selfish reasons, but from the

[1] Berthon, Pierre, Leyland F. Pitt, and Richard T. Watson (1996), The World Wide Web as an Advertising Medium: Toward an Understanding of Conversion Efficiency, Journal of Advertising Research 36(1): 43-54. [2] Bush, Alan J., Victoria Bush and Sharon Harris (1998), Advertiser Perceptions of the Internet as a Marketing Communications Tool, Journal of Advertising Research, 38(3): 17-27. [3] Bezjian-Avery, Alexa, Bobby Calder, and Dawn Iacobucci (1998), New Media Interactive Advertising vs. Traditional Advertising, Journal of Advertising Research 38(4), 23-32. [4] Fox, Robert (1999), News Tracks, Communications of the ACM, 42(5), 9-10. [5] Gallagher, Katherine, K. Dale Foster, and Jeffrey Parsons (1999), An Experimental Comparison of the Effectiveness of Print versus Hypertext Advertising, in Anne Lavack (Ed.), Proceedings of the Annual Conference of the Administrative Sciences Association of Canada, Marketing Division, Saint John, NB, Canada. [6] Ghose, Sanjay and Wenyu Dou (1998), Interactive Functions and Their Impacts on the Appeal of Internet Presence Sites, Journal of Advertising Research 38(2): 2943. [7] Grossman, Lawrence K. (1994), Reflections on Life along the Electronic Superhighway, Media Studies Journal 8(1): 27-39. [8] Hansell, Saul and Amy Hoarman (1999), "Subtly Trolling for Online Shoppers," Financial Post (National Post) 1(105), F27. [9] Ho, James (1997), Evaluating the World Wide Web: A Global Study of Commercial Sites, Journal of ComputerMediated Communications 3(1). http://jcmc.huji.ac.il/vol3/issue1/ho.html [Accessed 22 October 1998]. [10] Hoffman, Donna L. and Thomas P. Novak (1996), Marketing in Hypermedia Computer-Mediated Environments: Conceptual Foundations, Journal of Marketing 60(3): 50-68. [11] HotWired (1996), The HotWired Advertising Effectiveness Study. http://hotwired.com/brandstudy/ [Accessed 7 July 1998] [12] Jackson, Michele H., Assessing the Structure of Communications on the World Wide Web, Journal of Computer-Mediated Communication, 3(1). http://jcmc.huji.ac.il/vol3/issue1/jackson.html [Accessed 22 October 1998]

0-7695-0493-0/00 $10.00 (c) 2000 IEEE

10