Asynchronous Email Gotchas in App Engine

Summary: It's easy to send email asynchronously using App Engine and Django, but beware path differences between development and production environments.

Sending email is a standard part of running a website.  For instance, we might email the admins when a new user registers, or email a user when they've requested a password reset.  These emails are triggered while handling an HTTP request.  Therefore, if something goes wrong, it could interfere with the user's session by returning a 500 error or by slowing down the response.



Thus, we wish to send emails asynchronously, by detaching a process from the one that's handling the HTTP request.  If the email fails, we can catch exceptions and retry without slowing down the response.

In App Engine, this is very easy.  Simply add a task to the task queue.  A shortcut is to use the deferred library.  Create an object representing the email message as part of the request, then pass that object into a deferred method that actually sends the message (and catches errors).

  from google.appengine.ext import deferred

  # ...

  email = AnEmailObject(subject = subject, body = body,

                        sender = sender, to = addresslist)

  deferred.defer(deliver_email_task, email)

Easy enough, right?  Turns out the App Engine development environment fails to replicate the production environment exactly, tricking you into a false sense of security.  In development, unfortunately, a task can depend on any code that's been made available on your stack.  For example, if you're running Django (perhaps via App Engine Patch), your tasks can depend on the patched version of Django's EmailMessage class.  However, once you run these tasks on production, you'll see errors: you're referencing libraries that aren't addressable. This is because you're bypassing code (e.g., in Django, main.py) that initiates path manipulation - calls like sys.path.append('foo').

There are two fixes:

1. Fix the paths, described in the article on the deferred library. This may be substantial effort, depending on your web framework. In App Engine Patch, for example, this change would require modifications to modules inside the framework itself, which would make future App Engine Patch upgrades more difficult.

2. Eliminate dependencies in background tasks. To do this, create a python module (e.g., "tasks.py") that will house the code that you wish to run asynchronously in your project.  For instance, you might create a method called "send_email(message)".  To invoke this method,call deferred.defer(tasks.send_email, my_message_obj).  However, please note that your my_message_obj should probably just be a container for basic data types - though it's possible to pass around references to objects from your web container at this point, if you attempt to invoke those methods, you'll again see exceptions. This tasks module can depend on the App Engine libraries, which conveniently include a service for sending email.

Hope this helps! Let us know if you've come up with better solutions.

Links:

• http://code.google.com/appengine/docs/python/taskqueue/
• http://code.google.com/appengine/articles/deferred.html
• http://code.google.com/appengine/docs/python/mail/
  

Photo by qwrrty.