Beruflich Dokumente
Kultur Dokumente
This task has short instructions at the top of the task, and 14 questions. Each question has an
instructions section with examples, they are provided below.
Version: 2017-04-24
Task: You will be presented with the following in sequential order: a query, an ad
creative and a landing page. You will have to:
1. Research the query and try to understand the user intent. You will then have to
assess how clear the user intent is to you.
2. Next, you will be presented with an ad and you will have to assess a few different
features of the ad and the relationship of the ad to the query.
3. Finally, you will be shown the landing page for the ad, and you assess a few
different features of the landing page and the relationship of the landing page to
the query as well as the ad.
4. Each question you will be asked is contained within a box. At the bottom of the
box, there are examples and instructions. Please make sure to read through
these instructions and examples in detail and refer to them as frequently as
necessary while you are completing these tasks.
Query: If the query is absolutely incomprehensible to you, choose the "Unrateable
query" flag. You will not have to answer any of these questions in the task after this and
can go on to task submission. If you can comprehend the query to some degree, go
ahead with the rest of the task.
Question 1: How clear is the user intent to you?
In this section we want to know whether the user's intent is clear to you or not. After
researching the query, if you understood none of the words, then select the "unrateable
query" flag. Assuming you do understand at least some of the words in the query and
broadly what the user might be trying to achieve, try to think of the possible things the
user might be looking for or trying to do.
If you can only think of a single thing that the user might be looking to do, then the user
intent should be "Very Clear". On the other hand, if you can think of a few distinct
things (2 - 5 or so) that might be valid user intents, then the rating should be
"Somewhat clear". If there can be so many possible user intents that it is impossible to
enumerate them, then the rating should be "Very unclear".
Rating Description Examples
Very unclear There can be no Query: [life]
reasonable user Although the query is comprehensible,
intents or many it is absolutely not clear what the user
reasonable distinct was looking for. There are just too
user intents. many possible user intents to
enumerate.
Query: [cash]
The user can be looking for ways to
make money. Or the user could be
looking to borrow money. Or the user
could be looking to store cash at a
bank. The possibilities are endless and
since there can be many
interpretations, it is not clear what the
user is looking for.
Query: [www]
It is really not clear what the user is
looking for and there are no
reasonable user intents that can be
attributed to this query.
Somewhat clear There can be a few (2- Query: [chocolate chip cookies]
5 or so) reasonable It is a bit unclear what the user is
distinct user intents. looking for. The user might be looking
for recipes on how to make chocolate
chip cookies. Or the user might be
looking for online stores to purchase
chocolate chip cookies. It's possible
that the user is looking for information
about different kinds of chocolate chip
cookies. Since there can be a few
different valid user intents, the user's
intent is not very clear.
Query: [how to fix roof]
The user intends to fix some problem
with a roof. It is unclear what the
problem is and hence what targeted
information to show the user to satisfy
their intent. The user might want
information about roofing services, or
about roofing supplies or even tutorials
on how to fix roofs.
Query: [best buy santa monica]
There could a few different user intents
behind this query. The user could be
looking for directions to the Best Buy in
Santa Monica. Or the user might want
to determine if there is a Best Buy in
Santa Monica or they could be looking
for the website for Best Buy with the
store location set to Santa Monica.
Very clear There is only a single Query: [amazon.com]
reasonable user The user intends to visit amazon.com.
intent.
Query: [best budgeting app iphone]
The user is interested in finding good
financial budgeting apps for the
iphone.
Question 2. What is the scope of the concepts covered by the ad in relation to the
concepts covered by the query?
The query and the ad are both about sleeping products,
but the query is about bases while the ad is about
mattresses. Since both concepts are about bedding
products, they overlap partially but they are also each
about different, unique products so the overlap is only
partial.
Query: [recipes]
The query and the ad overlap because they are both
broadly about cooking but the user is searching for
recipes, while the ad is about cooking ware.
Query: [plymouth nc]
The query is for Plymouth, a city in North Carolina, and
the ad is for a very specific news article about North
Carolina's governor. There is some degree of overlap
because both the query and the ad are about North
Carolina, but the query and the ad are about
fundamentally different concepts, so this is a case of a
"Partial Overlap".
Ad The range of Query: [jewellery macys ]
Narrowe concepts
r covered by the
ad is narrower
than those
covered by the
query.
The user is looking for jewellery at Macy's. The ad is for
earrings and hence is narrower in scope.
The query is specifically for books at amazon, but the ad
is for the broader amazon website.
The query is looking specifically for frozen posters but the
ad is for Disney posters in general.
Same The range of Query: [amazon.com books]
concepts
represented by
the ad and
query are the The ad is looking for books on amazon.com and the ad is
same. for books on amazon.com. The ad also describes other
The ad does not concepts like "Amazon Prime", "Deals" etc., but the
cover any important concepts for both the ad and the query are the
concepts not same.
present in the
query and Query: [Women's sweaters JCPenny]
vice-versa
Question 3. Does the ad misinterpret the concept the user specified in the query and
target a completely different concept with a similar spelling or pronunciation?
Sometimes the same or similar phrases can refer to two entirely different concepts.
Determine the concept that the user is likely referring to and then assess whether the ad
targets a completely different concept with the same or similar spelling or pronunciation.
Rati Description Examples
ng
Yes The ad misinterprets Query: [paris]
the concept the user
specified in the query
and targets a
completely The ad is for the purchase of plaster of paris which is
differentconcept with a very likely an incorrect interpretation of the query. The
similar spelling or user probably want to see information about Paris,
pronunciation. France.
Query: [transformers 2]
No The ad does not target a Query: [hair laser treatment growth]
completely different
concept with a similar
spelling or pronunciation
to what the user The user is looking for laser treatment for hair growth.
specified in the query. The ad is showing treatments for hair removal. These
are related concepts and so even though this is a bad
ad, the ad still targets the same concept the user is
looking for. If instead, the ad targeted "Hair" the
movie, then the response to this question would have
been "Yes".
Query: [Pizza Santa Monica]
Question 4. Does this ad miss a critical requirement for satisfying the user intent?
Think about all the concepts and requirements the user has specified in the query. Now
for each requirement, ask yourself how likely the user is to consider an alternative. If it is
very likely that the user will consider an alternative, then the requirement is non-critical.
If missing this requirement or providing an alternative would be unacceptable to most
users, then it is critical. If an ad drops both a critical and a non-critical requirement
select "Misses critical requirement" since that leads to worse ad experience for the
user.
Rating Descriptio Examples
n
Misses The ad Query: [hair laser treatment growth]
critical misses one
requiremen or more
t critical
requiremen The user is looking for laser treatment for hair growth. But
ts that the the ad is showing treatments for hair removal. The ad
user completely missed the key requirement of hair growth
specified in instead of hair removal from the query and it is very unlikely
the query. that the user can be swayed to accept this ad as satisfying
their intent.
Query: [womens sweaters]
The user is looking for womens' sweaters but the ad is for
childrens' sweaters. This ad completely missed the
requirement of "women" and hence misses a key
requirement since the user is unlikely to be satisfied by
children's sweaters.
Query: [wow stutter]
The user is looking for information about a specific problem
while rendering graphics in the game World of Warcraft,
called stuttering. The ad, instead, provides information
about the stuttering, the speech impediment. The ad has
thus completely missed the key requirement that information
about stuttering should be in the context of the video game
World of Warcraft.
Query: [Pizza Santa Monica]
The query is for Pizza in Santa Monica while the ad is for
Pizza Delivery in San Francisco. The ad completely misses
the critical requirement that the pizza delivery should be in
Santa Monica since San Francisco is too far away from
Santa Monica for a pizza delivery.
The user is looking for women's shoes at Macy's. The ad is
for shoes at Zappos. Although this is not a perfect match
and the store is different from what is specified in the query,
the exact store is not such a critical requirement to
significantly affect user satisfaction. The user will likely find
equally good shoes at Zappos.
Doesn't The ad Query: [nike men's shoes]
miss any does not
requiremen miss any
ts requiremen
ts that the
user
specified in
the query.
Question 5. Is this ad awkwardly phrased or difficult to make sense of?
Is the ad text awkwardly phrased or difficult to make sense of? One possible way
this can happen is when the ad is "machine generated" and terms from the query are
stuffed into the ad creative resulting in a nonsensical or ungrammatical ad. Machine
generated ads aren't always bad, just the ones which are nonsensical or
ungrammatical. Please read the full ad creative text.
Rating Description Examples
Yes The ad text Query: [del mar food fair]
is
awkwardly
phrased or
difficult to
make sense
of.
It is obvious that the word "food" has been taken from the
query and inserted into the ad creative in an awkward
manner resulting in an ad which says "Map Food". This
doesn't quite make sense and this ad is likely to have been
machine generated.
No The ad text Query: [amazon.com books]
is not
awkwardly
phrased or
difficult to
make sense
of. The ad creative is not nonsensical or ungrammatical.
Question 6. What is the relationship between the products, services or information
advertised by the ad to what is specified in the query?
In this section, we want to know whether the Ad Creative offers what is specified in the
query, whether it offers a viable substitute or if it doesn't offer a viable substitute or
related product at all.
Rating Description Examples
No The Query: [Google pixel]
Relationsh information,
ip products or
services
advertised by The query is for a Google Pixel phone, but the ad
the ad and interprets it to be the movie "Pixels" released in 2016.
those
specified in
the query are
not related in
any way.
Related The Query: [HP printers]
products/servi
ces or
information
advertised in
ad does not The ad is for ink for printers. Ink does not perform the
perform the same function as printers but it is necessary for printers to
same function work.
as what is
Query: [HP printers]
specified in
the query, but
it performs a
related
function. They
might be
tangentially
relevant to the
user.
Query: [Beach Cities Car Wash Venice]
Car wash detergent is related to car washes because it is
used in car washes.
The query is for job listings on the site Monster.com, but
the ad is for the competitor site Indeed.com.
Question 7. How good is the ad relative to the top 3 search results?
In this section, we want to know how good the ad is relative to the search results. Look at the
top 3-5 search results to form your judgment. Please ignore Product Listing Ads (the sliding
panel of ads at the top of the results page) when making your decision. B elow, the panel
of ads displaying different Nikon digital cameras are Product Listing Ads.
Some examples for each of the rating categories:
Both Both the ad Query: [what is the phone number for Company X]
Ad and the
and search Company X does not have an online presence. The ad and the
Searc results are search r esults are for another unrelated company.
h Bad unlikely to
satisfy the
user intent.
The ad is for a travel company that organizes tours in Spain.
The search results point to webpages containing information
about Spanish cuisine.
The search results are from some other websites while the ad is
from newegg.com. So, the ad is likely to be more relevant to the
user than the search results.
Question 8. Does the LP have any issues that make it unusable / infeasible? Select all
that apply.
In this section, we want to determine if there are any serious issues that make it
infeasible to satisfy the user's intent. Select all the options that apply.
Rating Description Examples
Location The landing page is Query: [pizza Santa Monica]
mismatch for a product/service Landing page: The LP is for a pizzeria
in a location too far in Santa Barbara which is 85 miles
away from the user away from Santa Monica. It is
to be feasible. unreasonable to expect the user to
travel that far to get pizza.
Language Parts of the landing Query: [hotels paris]
mismatch page are in a foreign Landing p age: While large parts of the
language. LP are in E
nglish, the hotel descriptions
within the LP are in French. This affects
the user's ability to understand content
on the page relevant to their intent.
Out of stock The product / service Query: [NIKE FREE RN FLYKNIT 2017]
the user is looking Landing page: The LP has other Nike
for is out of stock / shoes available but not the model
no longer available. specified in the query. The image /
listing for this specific model is marked
as Out of Stock.
Query: [tickets superbowl 2016]
Landing page: The LP has tickets to
other football games but not the 2016
SuperBowl since it is already over.
Question 9. Does the LP have any issues that affect user’s interaction with the page?
Select all that apply.
Think about the user's interaction with the landing page when they visit it. Do they have
any negative experiences on any of the following dimensions? Select all that apply.
Rating Description
The LP is cluttered. Text and images are not well
Cluttered formatted and the LP is not aesthetically appealing. It
is not clear whether the information the user is looking
for is on the page and the user has to make an effort
to locate it.
Interstitial / form When the user visits the LP, they first see an
interstitial (content that prevents the user from
accessing the LP) or a form that is difficult to dismiss
or requires extra steps. The screenshot below shows
an example of an interstitial.
Distracting ads The LP has ads which attract the user's attention away
from the main content. For example, flashing ads or
too many ads or ads which are shocking and
surprising.
Prods the user to stay The LP prods the user to stay / makes it difficult for
them to leave the page through repeated messages or
alerts.
Bad grammar The LP consists of content which has bad grammar or
awkward phrasing
Question 10. Does the landing page have any issues that might undermine the trust
or safety of the user? Select all that apply.
Think about whether the ad is trying to deceive or mislead the user. A landing page
should be considered deceptive even if it technically has disclaimers/clarifying text, but
the text is hard to find, and likely to confuse the user.
Rating Description Examples
LP-AC mismatch The LP does not Query: [NIKE FREE RN FLYKNIT 2017]
provide what the ad Ad Creative: The ad creative promises
creative promises. Nike shoes.
Landing page: The LP doesn't actually
have any Nike shoes.
Pretends to be The LP falsely Query: [tickets LA Opera]
official claims to be the Landing page: The LP falsely claims that
official page when they are the official site to buy tickets to
it's not, or has a the Los Angeles Opera, but it is not. The
disclaimer on the official site is actually that of the LA
page but the opera.
disclaimer is hard
Query: [iphone repair]
to find.
Landing page: The LP falsely claims that
they are the official site for repairing
iphones, but the official site is actually
within the apple.com domain.
Query: [United Airlines]
Landing page: The LP looks very much
like the official united airlines page, but
has small print at the bottom of the page
indicating that they are not affiliated with
United Airlines. This could
mislead/confuse users into believing that
it is the official page
Provides false The LP provides Query: [vitamin C]
information false or misleading Landing page: The LP sells vitamin C
information based supplements and says that these
on common sense. supplements can help the consumer
lose weight. There is no medical
evidence supporting this claim.
Unexpected The LP leads to an Query: [songs by band x]
download unexpected Landing page: When the LP loads, it
download without starts downloading an .exe file which is
explicit permission supposed to play music available on the
from the user. site. The user never explicitly gave
permission to download this .exe file.
Question 11. What is the scope of the concepts covered by the landing page in
relation to the concepts covered by the query?
Partial overlap There is some degree Query: [Nike Men's shoes]
of overlap between age: LP contains shoes by
Landing p
the scopes of the LP different c ompanies but not Nike.
and the query, but
their scopes cover
fundamentally
different sets of topics
and one is not
contained within the Query: [Frozen poster]
other. Landing page: The LP contains posters
about Disney movies but not about
Frozen.
There is some degree of overlap
between the LP and query since both of
them cover Disney movies, but the LP
doesn't contain a poster specifically for
the Frozen movie. Note that this rating
for the LP is different from the ad
creative rating (for the same example
in the ad creative section).
Query: [women's sweaters]
Landing page: The LP contains
children's sweaters.
Question 12. Does this landing page miss a critical requirement for satisfying the
user intent?
Think about all the concepts and requirements the user has specified in the query. Now
for each requirement, ask yourself how likely the user is to consider an alternative. If it is
very likely that the user will consider an alternative, then the requirement is non-critical.
If missing this requirement or providing an alternative would be unacceptable to most
users, then it is critical. If the landing page drops both a critical and a non-critical
requirement select "Misses critical requirement" since that leads to a worse ad
experience for the user. You answered this question for the ad creative above, but
the answer need not be the same for the landing page.
Rating Description Examples
Misses critical The LP misses one Query: [catalonia bicycle tours]
requirement or more critical Landing page: The LP is about biking in
requirements that British Columbia, Canada.
the user specified The user is looking for biking tours in
in the query. Catalonia, Spain. While the LP correctly
identifies the biking requirement, it
misses the key requirement that the
biking tour has to be located in the
vicinity of Catalonia, Spain.
Query: [zipcode mesquite tx]
Landing page: The LP is sells mesquite
smoked wood for grills.
The user is looking for the zipcode for
the location Mesquite, TX. The LP
instead is for mesquite smoked wood. It
therefore missed the key requirement
that the user was looking for a zipcode
and not a product with the same name.
Misses The LP misses one Query: [macy's womens shoes]
non-critical or more non-critical Landing page: The LP is for Zappos
requirement requirements that and contains women's shoes.
the user specified This LP misses the “macy’s”
in the query. requirement specified by the user. But,
this is likely non-critical since the user
can find womens shoes at zappos as
well.
Query: [leather sofas]
Landing page: The LP has fabric sofas.
This misses the non-critical requirement
that the sofas should be leather.
However, it is possible that the user
might willing to accept fabric sofas.
Doesn't miss any The LP doesn't Query: [amazon.com]
requirements miss any Landing page: amazon.com homepage.
requirements that
the user specified
in the query.
Query: [nike men's shoes]
Landing page: The LP sells Nike Mens
shoes.
Question 13. What is the relationship between the products, services or information
in the landing page to what is specified in the query?
In this section, we want to know whether the landing page offers what is specified in the
query, whether it offers a viable substitute or if it doesn't offer a viable substitute or
related product at all. This is very similar to the same question above for the ad creative,
but consider the landing page instead. You answered this question for the ad
creative above, but the answer need not be the same for the landing page.
Rating Description Examples
No Relationship The information, Query: [Sony playstation]
products or services Landing page: The LP is about the
available on the LP history of Sony the company and
and those specified in hence does not offer a viable
the query are not they substitute or a related product/service
related in any way. for what is specified in the query.
Related The product/service Query: [Sony playstation]
or information in the Landing page: The LP is about
LP does not perform playstation game reviews. It does not
the same function as actually offer any consoles or games
what is specified in for sale.
the query, but it
Query: [zumba classes]
performs a related
Landing page: The LP is for fitness
function.
classes. A user who is interested in
zumba classes could be interested in
other fitness classes as well.
The product/service Query: [Sony playstation]
Viable substitute or information in the Landing page: The LP has Nintendo
LP performs the same Wii consoles and video games. It
function as what is does not contain any Sony
specified in the query playstations.
and hence it can be
used as a viable Query: [HP printer]
substitute by the user. Landing p age: The LP has Epson
printers. Epson printers are a viable
substitute for HP printers.
Same The product/service Query: [plumbing service venice]
or information in the Landing page: The LP is for a
LP is exactly what is plumber servicing the Venice, Los
specified in the query. Angeles area.
Query: [Sony playstation]
Landing page: The LP contains
different video game consoles such
as the Sony Playstation, the XBox
360 and the Nintendo Wii.
Question 14. Does the landing page help the user make progress towards their goal?
Imagine that the user has entered a query into Google, and is presented with a page of
search results (SERP, Search Results Page) and some ads, including the ad being
rated in this task. The user then clicks on this ad and visits the landing page. Think
about whether the user is substantially closer to achieving their goal than they were
before clicking on the ad. (i.e. on the google search results page).
Please click on the query link provided above and look at the Google search results
before answering this question. When you are considering the Google search results
page, please ignore any other ads from this same advertiser (e.g. Product Listing Ads,
these are the “Shop for X” pictures that sometimes show on top of the page).
Rating Description Examples
Negati Visiting the LP Query: [buy fondant molds]
ve takes the user Before c licking LP:
Progre farther away from
ss their goal,
compared to the
Google search
results page. The
user could get to
their destination
from the Google
search results
page with fewer
clicks / less effort
than from this
page.
The user will
have to go back
to the search
results page, or
restart their task
in order to make
meaningful
progress.