Sie sind auf Seite 1von 56

Pick ‘n’ Mix

Ben Firshman
Elegant Queues
Performance:
* Incredibly important, especially now web applications doing increasingly complex things.
* Amazon found that 100ms of latency caused 1% loss in sales
* Extra half a second on Google search results caused 20% drop in traffic
* Especially when altering data, it still needs to be responsive:
* Posting a tweet to followers
* Uploading images
* Importing API data
* Running stuff asynchronously
Tasks

* Running TASKS asynchronously


Tasks

❖ Independent components
❖ Loosely coupled
❖ Easily scalable

* Independent components that can run concurrently


* Loosely coupled: only depends on initial input
* Thanks to which, they’re easily scalable.

* Essentially:
* Stuff you can send off to run in the background.
* Stuff the user doesn’t immediately care about.
Application

* Say we have our application


* and a bunch of tasks that need running
* What runs these tasks?
Application

* Say we have our application


* and a bunch of tasks that need running
* What runs these tasks?
Application

? ?
? ?

* Say we have our application


* and a bunch of tasks that need running
* What runs these tasks?
Application

Worker Worker

* Workers process the tasks.


* Unlike “ghetto” queues done with cron jobs and databases, these are processes that have
tasks pushed to them
* These could either be on the same server as the application, or, thanks to independence,
spread out onto many
* Some way for application to publish tasks that need to be completed and distributing to
workers
Application

Message Queue

* Message Queues are useful for this. (explain)


* The messages can just be pickled tasks
Application

Message Queue

* Message Queues are useful for this. (explain)


* The messages can just be pickled tasks
Application

Message Queue
Application Application

Message Queue

* Nice thing with message queues: we can have multiple applications push things to the
queue.
* Means the frontend servers can be scaled horizontally without any of the task
infrastructure caring
Application Application

Message Queue

* Nice thing with message queues: we can have multiple applications push things to the
queue.
* Means the frontend servers can be scaled horizontally without any of the task
infrastructure caring
Application Application

Message Queue

Worker Worker
Application Application

Message Queue

Worker Worker
Application Application

Message Queue

Worker Worker
Application Application

Message Queue

Worker Worker
Application Application

Message Queue

Worker Worker
Application Application

Message Queue

Worker Worker
* This is where celery comes in
* Ask Solem Hoel has written an excellent piece of software that provides everything you
need for a task queue
Celery

❖ Worker daemon
Celery

❖ Worker daemon

❖ API for defining and executing tasks


Celery

❖ Worker daemon

❖ API for defining and executing tasks

❖ Handles communication with message queue (AMPQ or


STOMP)
Code!

from celery.task import Task


from celery.registry import tasks
import twitter

class ImportTweetsTask(Task):
def run(self, username, **kwargs):
return twitter.Api().GetUserTimeline(username)

tasks.register(ImportTweetsTask)

* Suppose you’re building a service which lets you update your status, and you want to
import your latest tweets
Code!

>>> from myapp.tasks import ImportTweetsTask

>>> ImportTweetsTask.delay(username=’londonpython’)

* This is the way a simple task is run.


* If you’re just writing something that writes the result to the database, this is fine
Code!
>>> result = ImportTweetsTask.delay(
username=’londonpython’)

>>> result.ready()
False

>>> result.ready()
True

>>> [t.text for t in result.result]


[u'I will also be looking into paying for pizza for
DJUGL attendees next Thursday at Global Radio offices
in Leicester Square.', ...]
map

>>> import operator

>>> from celery.task import dmap

>>> dmap(operator.add, [[1, 2], [4, 2], [3, 8]])


[3, 6, 11]

* Callable is pickled, so it must be available on the worker’s path


More goodies

❖ Periodic tasks as a distributed replacement for cron

❖ View for Ajax to check status

❖ Statistics

❖ Task sets for progress bars

❖ Retry tasks on failure

* Stats: how many tasks have been executed by type, time taken etc
http://www.celeryq.org/
py.test
py.test

❖ No boilerplate tests

❖ Useful test failure information

❖ Extensibility

* No boilerplate: Automatic detection of tests and use of built in Python features like assert
Code!

def test_equality():
x = 5
assert x == 5

def test_not_equal():
y = 9
assert y != 2

def test_articles_exist(articles):
assert articles.objects.count() > 0

* “funcargs” provide fixtures


pytest_django

❖ Creates test database and environment

❖ Ensures every test runs within a transaction

❖ Provides a few useful funcargs and utilities


Funcargs

def test_something(client):
assert 'Success!' in client.get('/path/').content

def test_article_view(rf):
request = rf.get('/article/4/')
response = article_view(request, 4)
assert 'Test title' in response.content
Funcargs cont.

def test_middleware(settings, client):


settings.MIDDLEWARE_CLASSES = (
'app.middleware.SomeMiddleware',
)
assert 'Middleware works!' in client.get('/')
Decorators

@py.test.params([dict(a=1, b=2), dict(a=3, b=3)])


def test_equals(a, b):
assert a == b

@py.test.urls('myapp.test_urls')
def test_something(client):
assert 'Success!' in client.get('/some_path/')
http://pytest.org/

http://github.com/bfirsh/pytest_django/
Feeds
syndication-view

❖ An attempt to clean up django.contrib.syndication


syndication-view

❖ An attempt to clean up django.contrib.syndication

❖ Feeds as class based views


syndication-view

❖ An attempt to clean up django.contrib.syndication

❖ Feeds as class based views

❖ Backwards compatible
syndication-view

❖ An attempt to clean up django.contrib.syndication

❖ Feeds as class based views

❖ Backwards compatible

❖ Eventually, refactoring generation of feeds including full


Atom support
Code!

from django.contrib.syndication.feeds import Feed


from articles.models import Article

class ArticleFeed(Feed):
title = "example.com news"
link = "/sitenews/"
description = "Latest news from example.com."

def items(self):
return Article.objects.all()[:15]
Yuck.
from articles.feeds import ArticleFeed

feeds = {
'articles': ArticleFeed,
}

urlpatterns = patterns('',
# ...
(r'^feeds/(?P<url>.*)/$',
'django.contrib.syndication.views.feed',
{'feed_dict': feeds}),
# ...
)
Code!

from syndication.views import Feed


from articles.models import Article

class ArticleFeed(Feed):
title = "example.com news"
link = "/sitenews/"
description = "Latest news from example.com."

def items(self):
return Article.objects.all()[:15]
Much better.

from articles.feeds import ArticleFeed

urlpatterns = patterns('',
# ...
(r'^articles/feed/$', ArticleFeed()),
# ...
)
http://github.com/bfirsh/syndication-view/

If you have any pet hates of contrib.syndication, let me know!


django-mptt
django-mptt

❖ Making trees out of Django models.

❖ Efficient retrieval.

❖ Methods added to models for traversing trees


(get_children(), get_root(), get_decendants() etc)

* Efficient retrieval as opposed to a simple parent field and retrieving children with related
name
Code!

from django.db import models


import mptt

class Category(models.Model):
    name = models.CharField(max_length=50)
    parent = models.ForeignKey('self', null=True,
blank=True, related_name='children')

mptt.register(Category)
Code!

from django.db import models


import mptt

class Category(mptt.Model):
    name = models.CharField(max_length=50)
    parent = models.ForeignKey('self', null=True,
blank=True, related_name='children')
Code!

from django.db import models


import mptt

class Category(mptt.Model):
    name = models.CharField(max_length=50)
    parent = models.ForeignKey('self', null=True,
blank=True, related_name='children')
http://github.com/bfirsh/django-mptt/
http://benfirshman.com/
@bfirsh
Credits
http://www.flickr.com/photos/ilovetechnology/3048585444/ http://www.flickr.com/photos/chrissie64/1425138918/
http://www.flickr.com/photos/tomhakase/312785554/ http://www.flickr.com/photos/patlejch/2951799028/
http://www.flickr.com/photos/pensive-reflections/3829587788/ http://www.flickr.com/photos/mckenzieo/1601233059/
http://www.flickr.com/photos/purpleslog/183842413/ http://www.flickr.com/photos/webel/2429902979/

Das könnte Ihnen auch gefallen