Deeplinks Title Quality Crowdsourcing Guideline

Overview Task Description

Deeplinks are links that lead to subpages in a domain which help
the user locate sub-intents within the specified results.
Deeplink titles are the link’s actual words of text that represent
content on a Deeplink page.
Relevant Deeplink titles are considered comprehensive if they are
in the same language as the page content, do not contain any
grammatical issues, and accurately represent the page itself.
Follow the steps in the Hitapp page as below.
1. Understand the Deeplink title and review the Deeplink page.
2. Rate the Deeplink title quality as it relates to the Deeplink page
you reviewed.
3. Please answer the questions presented:

ATTENTION:
*If the Deeplink page is a dead page (e.g. couldn’t load, or page is blank), mark Q1, Q2 and Q3 ‘No’ and submit the HIT. Do NOT attempt
to assess the title for garbled text or relevancy to the dead Deeplink page.
*If the Deeplink title and the Deeplink page are both in the same foreign language, mark Q1, Q2 and Q3 ‘No’ and submit the HIT. Do NOT
attempt to assess the foreign language title for garbled text or relevancy to the Deeplink page.
Deeplink Title Defects Judging Sample
Aligning with the questions, we give some Deeplinks title defects judging samples as below and indicate the correct answer:

Question 1: Is the title in different language than the page content? The defect type is Language Mismatch, which means language of Deeplink
title does not match language of Deeplink page.
o Exceptions: the title is a named entity (i.e. brand name, food, restaurant name, etc.)

Deeplink Title Deeplink Page Rating Reason

बिहार प्रशासबिक सु धार बिशि http://www.bpsm.bih.nic.in/
News in Gujarati http://www.divyabhaskar.co.in/i
ndex1.html
Céline https://www.celine.com/en
el gaucho http://elgaucho.com/dine/bellev
ue/
YES title in Indian but page in English, mismatch
YES title in English but page in Indian
NO "Céline" is a French brand name. No matter what languages the
landing page is, it’s OK to provide the brand name in French.
NO "el gaucho" is a Spanish restaurant name. No matter what
languages the landing page is, it’s OK to provide the restaurant
name in Spanish.

Question 2: Is the title misleading/Ambiguous? The defect type is Misleading/Ambiguous, which means Deeplink title does not represent the
Deeplink page content clearly, or the Deeplinks title misleads users about the page content.
o NOTICE: the title might be incomplete, if you could predict the missed part and judge the title misleading/ambiguous, answer
YES/NO, if you can’t predict the whole title, choose the ‘Title incomplete and can’t predict/judge.’

Query Web result title/URL Deeplink Title/URL Rating Reason

Ebay lego lego | eBay Other YES Ambiguous: title is not informative enough to clearly
represent the content of the landing page
Amazon Amazon.com - Official Site Official Site YES Misleading: most users would expect official site of
amazon.com from this Deeplink title but the page is
product searching list of ‘Official Site’.
Facebook Facebook - Log In Facebook Login YES Misleading: most users would expect the general login
for Facebook from this Deeplink title but the page is a
‘Facebook Login’ plugin provided to developers.
 oklahoma OMES: Human Capital Current NO Although the title is incomplete, the title is most
government Management (HCM) - Employment Oppo… possible to be ’current employment opportunities’,
jobs State Jobs which represents the Deeplink page well.
openings
auburn Auburn Tigers softball - Official Athletics Site Title Title is incomplete and missing the most important part,
softball Official Site of Y… incomplete can’t judge the relevance between the title content and
and can’t the Deeplink page content.
predict/judge
Xfinity Comcast Cable - Official XFINITY Help: How YES Ambiguous: the critical information is cropped.
Site can I res …
Xfinity Comcast Cable - Official Reset Your Modem ?- NO Despite weird punctuation, title is not misleading of
Site XFIN … landing page
Cross Cross Country Education Login NO Specify “No” to all questions if the Deeplink page is a
country dead page
education
Gary Danko Gary Danko - Official Site HTML YES Misleading: expectation is a page about HTML, not a
page about a restaurant.
wcspermits Oversize Load Permits | Customer Login YES Misleading: Deeplink page is not a login page
WCS PERMITS

Question 3: Does the title contains typo/wrong capitalization/weird punctuation? The defect type is Garbled/Bad Text, which means Deeplink
title incomplete, /contain typos/incorrect capitalization/incorrect punctuation. (Incorrect capitalization: The title should follow APA title case
rules, an explanation of which can be found here. We permit single-instance case errors in a title (that is, one mistake), but not more than one.

o Exceptions: the title is a named entity or common abbreviation. (i.e. brand name, food, restaurant name, etc.)

Deeplink Title/URL Rating Reason

Games A-Z NO The "-" symbol in "A-Z" makes sense in written English
Ohters YES Title contains obvious typo (should be Others)
Obits NO ‘Obits’ is a common abbreviation of ‘Obituaries’
play Games now YES incorrect capitalization of marked letters
NikeStore. Shop the YES Incorrect punctuation ‘.’ as well as incomplete
E! Shows NO Read the web result landing page and you'll find "E!" is a named entity in this website
Contact Us | Contact Us YES repeating title with ‘|’, wrong punctuation
Contact Us | Email NO Page info delineated with ‘|’ is ok
uverse.com YES URL-like titles are considered bad.
Powerschoolnotice YES One space between two natural language words is necessary. “Power School Notice” is good.
YouTube NO Capitalization and spacing are good because it’s a named entity. Similar cases are “CapitalOne”, “Capital
One”, “PizzaHut”, “Pizza-Hut”.