Foursquare & MongoDB

Harry Heymann May 21, 2010

Foursquare Overview
A location based social network. Allows users to "check in" at bars, restaurants, shopping destinations, etc to share their location with friends. Rewards users with virtual prizes (points, badges) that can sometimes lead to real life rewards (5 checkins at a restaurant might get you a free appetizer) ~1.3M registered users. ~615k checkins/day. Nearly 50M checkins total. Very rapid growth.

Basic technical details

Written in scala (a somewhat new language what you'd get if Java & ML had a baby. It runs on the JVM) Uses a web framework called Lift Originally used a single PostgreSQL instance as the data store. Scaling up on on a SQL database can be frustrating (replication & sharding don't work as easily as one would like) so we're moving to MongoDB. Built in geospatial capabilities (obviously very important to foursquare) are a very nice bonus.

Transition to Mongo
Currently writing checkins, tips, venues (and various things related to venues) to MongoDB. All writes still go to PostgreSQL as well. Slowly migrating various reads. Exclusively use mongo for our "Who's here" server. A short lived record of where our users are at any given time (contains 3 hours worth of data). Migrating geo related queries first. Other items later. Checkins a high priority (due to the fact that they represent the bulk of our data).

Geospatial Indexes
MongoDB conveniently supports geospatial indexes out of the box. Currently limited to Earth like dimensions (+/- 180 degrees in each of 2 axes). It cheats to make the math easier/faster by assuming a flat earth where 1 degree of latitude or longitude is always the same distance. This is fine as long as you are dealing with relatively small distances (as foursquare does) Implemented using geographic hash codes atop standard MongoDB b-trees

Creating Geospatial Indexes

Indicate the "2d" index type: db.venues.ensureIndex ({latlng: "2d"}) Specify additional fields if you plan on using compound geospatial queries (more on these in a moment): db.venues. ensureIndex({latlng: "2d", closed: 1, keywordList: 1})

Take care: 1k limit on key size

If you have a compound geospatial index and you don't take care it can be easy to go over the MongoDB limit of 1K. For example if the index is on {latlng: "2d", keywordList: 1} then the following venue would be a problem: {latlng: [40, -72], keywordList: ["some", "venue", "with", "a", "whole", "lot", "of", "different", "words", "in", "the", "name", "of", "the", "venue", "it", "just", "keeps", "going", "on", "and", "on", "forever", "without", "seeming, "to", "ever", "stop"]} In these cases the individual item will be dropped from the index making it impossible to query.

Basic Geospatial Queries

Find the closes 20 venues to a given location: db.venues. find({latlng: {$near: [40.72, -73.99]}).limit(20) Find up to the closes 20 venues to a given location that are within 1 degree of the location: db.venues.find ({latlng: {$near: [40.72, -73.99, 1]}).limit(20)

Foursquare uses this to find nearby venues, tips, specials, and various other geolocated data.

Complex Geospatial Queries

If you have a compound geospatial index defined you can query on additional fields and still use the index: db.venus. find({latlng: {$near: [40.72, -73.99]}, closed: false} (because we generally don't want closed venues) Basic search: db.venus.find({latlng: {$near: [40.72, -73.99]}, closed: false, keywordList: $all: ["nyc", "seminar"]})

Bounded geospatial queries

Foursquare doesn't do much of this, but it's possible to find all of the items in a collection that are within a given circle or square: db.venues.find({latlng: {"$within": {"$box": [[40, -72], [41, -73]]}}}) db.venues.find({latlng: {"$within": {"$circle": [[40, -72], 0.5]}}} Can be combined with complex geospatial queries that were demonstrated on last slide. In general though $near will be more useful/performant than $within.

MongoDB & Scala/Lift

Lift has a generic ORM layer called record for which there is a MongoDB implementation. Originally called scamongo, but as of Lift 2.0 M5 it's integrated into the core Lift codebase as lift-mongodb It's a very thin wrapper around the Java driver provided by 10gen It's new, so has some wards/oddities, but should improve fairly rapidly (foursquare is working on this)

A foursquare venue using lift-mongo

class Venue extends MongoRecord[Venue] with MongoId[Venue] with GeolocationMongo[Venue] { object venuename extends StringField(this, 255) object address extends OptionalStringField(this, 50) object closed extends BooleanField(this) // etc }

Basic ORM operations work as you might expect

val venue = Venue.createRecord .venuename("NYC Seminar & Conference Center") .address("71 W 23rd St") .city("New York").state("NY").zip("10010") venue.save val query = QueryBuilder.start(Venue.venuename.name) .is("Gramercy Tavern") .get() venue = Venue.findAll(query)

PS: We're hiring

All sorts of roles: engineering, operations, business. See http://foursquare.com/jobs or come talk to me.