Beruflich Dokumente
Kultur Dokumente
Ben Firshman
Elegant Queues
Performance:
* Incredibly important, especially now web applications doing increasingly complex things.
* Amazon found that 100ms of latency caused 1% loss in sales
* Extra half a second on Google search results caused 20% drop in traffic
* Especially when altering data, it still needs to be responsive:
* Posting a tweet to followers
* Uploading images
* Importing API data
* Running stuff asynchronously
Tasks
❖ Independent components
❖ Loosely coupled
❖ Easily scalable
* Essentially:
* Stuff you can send off to run in the background.
* Stuff the user doesn’t immediately care about.
Application
? ?
? ?
Worker Worker
Message Queue
Message Queue
Message Queue
Application Application
Message Queue
* Nice thing with message queues: we can have multiple applications push things to the
queue.
* Means the frontend servers can be scaled horizontally without any of the task
infrastructure caring
Application Application
Message Queue
* Nice thing with message queues: we can have multiple applications push things to the
queue.
* Means the frontend servers can be scaled horizontally without any of the task
infrastructure caring
Application Application
Message Queue
Worker Worker
Application Application
Message Queue
Worker Worker
Application Application
Message Queue
Worker Worker
Application Application
Message Queue
Worker Worker
Application Application
Message Queue
Worker Worker
Application Application
Message Queue
Worker Worker
* This is where celery comes in
* Ask Solem Hoel has written an excellent piece of software that provides everything you
need for a task queue
Celery
❖ Worker daemon
Celery
❖ Worker daemon
❖ Worker daemon
class ImportTweetsTask(Task):
def run(self, username, **kwargs):
return twitter.Api().GetUserTimeline(username)
tasks.register(ImportTweetsTask)
* Suppose you’re building a service which lets you update your status, and you want to
import your latest tweets
Code!
>>> ImportTweetsTask.delay(username=’londonpython’)
>>> result.ready()
False
>>> result.ready()
True
❖ Statistics
* Stats: how many tasks have been executed by type, time taken etc
http://www.celeryq.org/
py.test
py.test
❖ No boilerplate tests
❖ Extensibility
* No boilerplate: Automatic detection of tests and use of built in Python features like assert
Code!
def test_equality():
x = 5
assert x == 5
def test_not_equal():
y = 9
assert y != 2
def test_articles_exist(articles):
assert articles.objects.count() > 0
def test_something(client):
assert 'Success!' in client.get('/path/').content
def test_article_view(rf):
request = rf.get('/article/4/')
response = article_view(request, 4)
assert 'Test title' in response.content
Funcargs cont.
@py.test.urls('myapp.test_urls')
def test_something(client):
assert 'Success!' in client.get('/some_path/')
http://pytest.org/
http://github.com/bfirsh/pytest_django/
Feeds
syndication-view
❖ Backwards compatible
syndication-view
❖ Backwards compatible
class ArticleFeed(Feed):
title = "example.com news"
link = "/sitenews/"
description = "Latest news from example.com."
def items(self):
return Article.objects.all()[:15]
Yuck.
from articles.feeds import ArticleFeed
feeds = {
'articles': ArticleFeed,
}
urlpatterns = patterns('',
# ...
(r'^feeds/(?P<url>.*)/$',
'django.contrib.syndication.views.feed',
{'feed_dict': feeds}),
# ...
)
Code!
class ArticleFeed(Feed):
title = "example.com news"
link = "/sitenews/"
description = "Latest news from example.com."
def items(self):
return Article.objects.all()[:15]
Much better.
urlpatterns = patterns('',
# ...
(r'^articles/feed/$', ArticleFeed()),
# ...
)
http://github.com/bfirsh/syndication-view/
❖ Efficient retrieval.
* Efficient retrieval as opposed to a simple parent field and retrieving children with related
name
Code!
class Category(models.Model):
name = models.CharField(max_length=50)
parent = models.ForeignKey('self', null=True,
blank=True, related_name='children')
mptt.register(Category)
Code!
class Category(mptt.Model):
name = models.CharField(max_length=50)
parent = models.ForeignKey('self', null=True,
blank=True, related_name='children')
Code!
class Category(mptt.Model):
name = models.CharField(max_length=50)
parent = models.ForeignKey('self', null=True,
blank=True, related_name='children')
http://github.com/bfirsh/django-mptt/
http://benfirshman.com/
@bfirsh
Credits
http://www.flickr.com/photos/ilovetechnology/3048585444/ http://www.flickr.com/photos/chrissie64/1425138918/
http://www.flickr.com/photos/tomhakase/312785554/ http://www.flickr.com/photos/patlejch/2951799028/
http://www.flickr.com/photos/pensive-reflections/3829587788/ http://www.flickr.com/photos/mckenzieo/1601233059/
http://www.flickr.com/photos/purpleslog/183842413/ http://www.flickr.com/photos/webel/2429902979/