SURVEY OF OPEN SOURCE

FULL TEXT SEARCH

SOLUTIONS

Curtis Spencer, curtis@cruxlux.com

SEARCH WITHOUT FULL TEXT
• SQL “Like %product%”
– Easy to setup, but…
– SQL statements get too complex (giant OR and
(?,?,?,?,?,?,?,?,?,?)
– Indexes on many columns become unwieldy and
slow down inserts
– Limited to prefixs of bigger text columns
– Separation of Power: Join Index vs. Full Text
• Outsource to Google
– Hosted Solution
– Can only reach data that you actually render to
html
FULL TEXT GOAL
 Return matches by relevance rather than
pure equality value match
 Precision vs. Recall

Precision – Are the results accurate?

Recall – Did we get all the results we expected?
 Natural Language Search
Queries such as “What is the fastest animal?”
FULL TEXT IMPLEMENTATION
 Inverted Index Data Structure
Index of words to document’s location on disk
 Tokenization, Stopwords
Internationalization Challenges
 Basic Query Languages
Boolean match, relevance, proximity, etc.
“World Series +Poker –Baseball”
Based on Apple’s Search Kit Impl
LANGUAGE STEMMING
 Reduce inflected words to their root
Increase recall
Decrease inverted index size
 Internationalization Challenges
Language detection of the dataset to determine
which stemming algorithm to use
Complexity proportional to the level of
morphology
 Porter Stemming Algorithm
Examples: names -> name, departed -> depart,
Mariners -> marin, Marin -> marin
 Snowball Project has a lot of different
stemming implementations.
MYSQL FULL TEXT
• Pluses
– Integrated into MySQL
– Easy to use without learning a new library
• Minuses
– Indexes bigger than memory tend to be slow
– Scalability options are limited
– Can slow down insertions, deletions
– CJK is lacking
SPHINX
• Pluses
– Very Fast
– Supports many data sources
– Retrieval can be integrated into MySQL
– Distributed Searching is a scaling option

• Minuses
– Configuration can be tricky
– Live index updates accomplished by delta
indexing
– Internationalization (besides Russian) is left as an
exercise for the reader
LUCENE/SOLR
• Pluses
– Java, so easy to integrate into client software as
well as web
– Stable
– Distributed Searching
– Powerful Query Language
– Extensible API
– Good Internationalization Support
• Minuses
– Java
– Configuration is a pain
WHEN TO USE WHAT
Questions?
THANK
YOU!