IAB & AOP Online Advertising Effectiveness Research - Best Practice Guidelines 1 - Background 1.

1 Demonstrating effective return on investment from online advertising expenditure is increasingly important for agencies and advertisers because of more sophisticated approaches to planning and executing online advertising campaigns and the ongoing need to evaluate return on investment across all advertising activities. 1.2 In order to ensure that research conducted in this area is as effective as possible it is essential that everyone involved from research agencies, media agencies, media owners and ad operations teams understands their different roles in delivering this research. It is also essential that all parties view the delivery of research results as a collaborative process and, whenever possible, share and exchange information and results. The purpose of this document is to act as a best practice guide for running online advertising effectiveness research and also to act as an educational piece for those who might not have managed these research projects previously. 1.4 This increasing utilisation of online advertising effectiveness research has meant several issues, largely consistent across the industry, have emerged. Some of these include: Possible negative impact on the user experience because of the increasing number of surveys. Declining response rates because of the high number of surveys conducted and the increasing savviness of internet users in completing such surveys (especially when incentivized). Concern about the quality of responses due to users being well experienced in responding to advertising research, and thereby biasing results. The lack of availability of ad inventory for these surveys.

1.5 The Online Advertising Effectiveness Research - Best Practice Guidelines have been drawn up by IAB and AOP in response to demand from members to ensure that research requested by media agencies and clients is mutually beneficial and is conducted as effectively as possible. The following aims to outline some of the key aspects for consideration when planning and setting up online advertising effectiveness.

2 - When should online ad effectiveness surveys be used? 2.1 Online ad effectiveness surveys are best used when the campaign objectives cannot be measured by standard web metrics (for example that show click thru rates, newsletter sign ups) alone; for example when the aims of the campaign are to increase brand awareness or to change or improve brand perceptions. 2.2 Online ad effectiveness surveys should only be used in conjunction with campaigns of a significant size. However the size of the campaign (usually measured in terms of the number of impressions booked) with which it is appropriate to use this research technique will vary from site to site therefore advertisers and research suppliers should liaise with individual site owners to establish the size of the campaign that the site owner deems appropriate for ad effectiveness research. 2.3 Media owners will also provide guidance to suppliers as to the minimum number of impressions (both in terms of campaign size and in terms of impressions needed to generate responses) needed to look at survey results on an individual site basis. 2.4 Co-branding a survey (using the media owners brand) will usually increase response rates. Respondents are more likely to accept survey invitations that include the site owners brand because of their existing relationship with the host website. However the use of any media owners brand (both on any survey invitation and the survey itself) must be agreed by the media owner in advance and the media owner should be given sign off on the design. 3 - Online ad effectiveness surveys - timing 3.1 Media owners should be made aware that the advertiser / agency is interested in running an ad effectiveness study in conjunction with an advertising campaign as early as possible, preferably at the planning stage. If the advertiser / agency is liaising with the sales department at the media owner then the sales department should inform the research department immediately to facilitate implementation. The research supplier should also contact the media owners research department at the first possible opportunity. The media owners research department will also be able to advise the advertiser of the availability of running research and advise on any internal projects that the media owner is planning. 3.2 Timings will also vary as to which methodology is being used a pre / post methodology will require the survey to run prior to the advertising activity (so more notice will be needed) while an exposed / non-exposed methodology does not require any sample gathering prior to the advertising campaign. 3.3 Media owners require a minimum of four weeks notice in order to run an online advertising effectiveness study, although it may be possible to run an ad effectiveness study at shorter notice. This lead time is necessary so that the media owner can assess the impact of the study on other research projects running on the site. If surveys are requested with
2

less than four weeks notice then all parties should be aware that this may affect the availability of surveys, response rates and the quality of results. 3.4 Timings and deadlines (to enable set up and testing) will be longer for new research suppliers in this area. 4 - Online ad effectiveness surveys - sharing of results 4.1 Media owners that wish to see results of ad effectiveness surveys should agree access to these as early as possible. Ideally media owners should be given access to results for their own sites and be able to benchmark these against campaign averages. Ownership of the data rests with the advertiser (that has commissioned and paid for the research); therefore the media owner should liaise directly with the advertiser (or their agency) regarding access to the results. The research supplier should also facilitate the sharing of data and acknowledge the media owners important role in the research process. 4.2 In the case of co-branded surveys media owners should always be given access to results (because of the key role of the media owners brand in increasing response rates to the survey). 4.3 When discussing sharing results media owners should contact the research department (rather than the media planner) at the advertising agency. 4.4 The research supplier should aim to maximise communication with the media owner at all times and keep them informed with survey progress at all times. Where possible the research supplier should inform the media owner when the target sample is reached and/or when the survey is taken down although it is desirable to achieve as big a sample size as possible (and therefore might be preferable to keep the survey live for as long as possible), the advertiser and research supplier should be sensitive to the media owners needs to run other online research projects and to minimise user disruption. 4.5 Research suppliers should feed completion rates back to media owners as this will help media owners build historical records regarding completion rates and contribute towards the success of future effectiveness surveys. Media owners value regular communication from suppliers in this area and are more likely to wish to work with suppliers able to provide this. 5 - Online ad effectiveness surveys approval before the survey goes live 5.1 Ideally the research department in the media owner should be given the chance to approve the survey before it goes live. From the media owners point of view it is important to validate that the survey does not contain any questions that they are not happy for their website / brand to be associated with. The research supplier should ensure that the survey complies with Market Research Society guidelines and policies of the host media owner.

5.2 The maximum length of any online ad effectiveness survey should be five to seven minutes this is approximately 20 to 25 questions. This will help maximise completion rates and ensure the validity of data collected. The research supplier should also consider the user experience when completing the questionnaire and vary question type where possible this will also help ensure the validity of data collected. It is also best practice to be upfront about the amount of time that it will take to complete the survey so that the user knows what to expect in terms of survey length. 6 - Online ad effectiveness surveys frequency of user exposure 6.1 It is strongly recommended that the disruption to users caused by exposure to research invitations is kept to a minimum. The more intrusive the invitation then the fewer the number of user exposures should be. For example if a survey invitation is served via a banner advertisement (that does not require user action to decline) then it is acceptable for the user to be exposed on multiple occasions, however if the invitation is intrusive (for example via an overlay) then the number of exposures should be limited. The research supplier should agree the number of exposures with the media owner before the survey goes live. 6.2 Its recommended that each user is only exposed to each survey once in its lifetime. If a user has actively declined to complete a questionnaire at invitation stage then that user should not be invited to complete the same questionnaire again. 7 - Online ad effectiveness surveys - inventory 7.1 Media owners are under no obligation to provide free inventory to allow suppliers to carry out ad effectiveness surveys. 7.2 Research suppliers should liaise with individual media owners regarding the availability of ad effectiveness surveys. Media owners will seek to limit surveys to minimise the impact on user experience. 7.3 Based on historical completion rates the Media owner will be able to provide the advertiser and the research supplier with guidance regarding the number of impressions needed to achieve the sample target. 8 - Online ad effectiveness surveys sample size and robustness of results 8.1 There should be a minimum sample of 75 respondents within any group that is broken out within a survey for example it is acceptable to break out gender in an overall sample of 200 where there are 110 women and 90 men but it is not acceptable to break out age groups using the same sample where the sample within each age group is less than 75 e.g. aged 16 24 = sample size of 60, aged 25 34 = sample size of 55 etc.

8.2 If results and findings are based on smaller sample sizes then this should be indicated within the results report and a small sample size warning applied. 8.3 If using an exposed / unexposed methodology suppliers should also seek to achieve a balance between the exposed and unexposed sample. 9 - Online ad effectiveness surveys - interpretation of results 9.1 Advertisers and research suppliers should understand that there is a potential difference in results generated using a methodology that gathers data from users shortly after they are exposed to advertising (using a site based questionnaire) and a methodology that gathers data after a period of time has elapsed since exposure to the advertising (using a panel based methodology). 10 - Summary of responsibilities of those involved in ad effectiveness surveys 10.1 Research suppliers To make contact with the media owner as simple as possible by providing a single point of contact To liaise with all parties to ensure smooth project management of ad effectiveness surveys To monitor and optimise the quality of the questionnaire and to respect the respondents online experience To adhere to IAB / AOP guidelines and to liaise closely with media owners regarding inventory and sample targets

10.2 Sales person at media owner Inform research department within media owner as soon as conversation about running and ad effectiveness survey begins to ensure that research department is able to advise on the availability for the survey as early as possible and then plan in the survey effectively. Be open and realistic with all parties about ad effectiveness survey guidelines, availability of surveys, necessary inventory, possible completion rates and timings after consultation with the research department. Agree the level of impressions needed to complete the survey with the research department and sell these impressions to the agency / supplier at the appropriate value. Request access to the survey results at the earliest opportunity.
5

10.3 Research department at media owner To provide prompt and clear feedback to sales teams regarding the availability of online effectiveness surveys and to give guidance on appropriate inventory levels to meet survey requirements. To liaise with the ad operations team and to plan in and manage surveys so that user experience is not adversely affected To approve the quality and content of surveys To approve the look and feel of the overlay invitation and provide advice on other invitation methods, including text links and banners

10.4 Ad operations team at media owner To ensure the swift and accurate implementation of online ad effectiveness surveys by liaising with suppliers as appropriate To ensure that the correct number of impressions are delivered so that the research can be conducted effectively

10.5 Research department at agency To liaise closely with all parties regarding the aims of the survey, content of the questionnaire, sharing of results and overall project management

10.6 Media planner at agency Must inform sales person at media owner as soon as is possible regarding need for ad effectiveness survey and facilitate any necessary conversations between research departments at agency and media owner

11 - The following media owners, agencies and research suppliers have contributed to the development of these guidelines and have committed to adhering to them. ANY Media AOP BBC Worldwide BSkyB Dynamic Logic Future Publishing IAB ITV Media Contacts Metrix Lab
6

Microsoft Advertising News International Survey Interactive Yahoo! 12 - For further information please contact the AOP or the IAB. Tim Cain Head of Research and Insight Association of Online Publishers T: 020 7400 7510 E: Tim.Cain@ukaop.org.uk www.ukaop.org.uk Tim Elkington Head of Research Internet Advertising Bureau T: 020 7050 6951 E: Tim@iabuk.net www.iabuk.net