2020-07 ( Django 3.1rc1 )

Upgrading this blog to Django 3.1

See also

The first Django 3.1 release candidate came out earlier this week, and I’ve upgraded my blog to this (a one-line change I made through the GitHub web interface thanks to Heroku continuous deployment).

I also upgraded my Heroku PostgreSQL database to v11 so I could take advantage of the new search_type=”websearch” option for PostgreSQL full-text search added in Django 3.1—James Turk contributed that improvement and wrote about it on his blog.

This means my blog search engine now supports advanced operators, like “phrase searches” and -subtractions—try that out here: “signed cookies” -django

Django 3.1rc1

See also

Django news 33

See also

This is the final chance to try 3.1 features before the release on August 3rd.

See also

https://realpython.com/django-user-management/

See also

Huey as a minimal task queue for Django by J.V. Zammit ( https://fediverse.org/jvzammit )

See also

Warning

Can not schedule tasks under 60 seconds.

See https://github.com/coleifer/huey/blob/master/huey/consumer.py If periodic tasks are enabled, the scheduler will wake up every 60 seconds to enqueue any periodic tasks that should be run.

Introduction

Are you considering adding a task queue to your Django project ? Then this article should be useful to you.

When I was younger task queue with Django project meant celery task queue.

Now that I’m older there are simpler alternatives.

The simplest I found was Huey . But the ideas presented here apply to evaluating all task queues for your Django project.

Is celery heavyweight ?

A colleague achieved significant gains in task execution time by moving off celery. To dramatiq.

This was on a Python 3 project I didn’t work directly on. By significant I mean ~50% throughput. In this case, “tasks” were about handling a simple message that wrote one row to the database, at most. With no real processing of that message.

At about the same time the above happened, I was listening to the DjangoChat podcast.

The below is a transcript from the “Caching” episode from November 2019. Transcript here. The podcast folks were discussing caches. And brought up the usual suspects; Memcached and Redis.

On mentioning Redis, Carlton Gibson calls out how easy it is to add a queue when you have Redis in place. And how much of an “overkill” celery is. Emphasis in the below quote is mine: 

have Redis? Yeah. You want to use a queue.

So let’s take a good queue package. So, you know, everyone always
talks about celery, but celery is overkill for, you know, the majority
of use cases.

So what’s a good package? Well, there’s one called django-q, which
I love and have fun with. That’s nice and simple.
And that’s got a Redis back end. So you pip install, right or, you
know, apt install Redis. And then you pip install django-q into
your project, you know, a little bit of settings, magic, and you’re
up and running [..]

I considered these packages to see if I could adopt a more lightweight alternative to celery. Before I continue, by “lightweight” I mean, “lightweight in terms of”:

  • package size and dependencies

  • code that I would need to rework to transition from celery to this task queue

  • works with Redis, a pre-existing component in my stack

The packages I compared considered:

  • dramatiq

  • django-q

  • huey (Recommended by Adam Johnson, Django blogger I follow).

I decided to move on with Huey. It was not a clear-cut decision.

My mindset was not about installing the best. It was about installing a minimal package. That removes my dependency on celery/django-celery. And allows me to continue taking that project’s codebase to Python 3.

Why not dramatiq? I did not go with dramatiq because of two reasons:

  • It required installation of another package. The “Advanced Python Scheduler”, APS, to allow scheduling of tasks.

  • django-dramatiq, while maintained by the author of dramatiq itself, is “yet another package”.

After my experience with django-celery any extra package scared me. I did not want to end up being unable to use a main library due to a smaller accompanying library not being maintained.

In comparison, for Huey I would only need to pip install huey. The Django integration part is part of the package. Docs here.

In addition, dramatiq is intended for rabbitmq. I see no need for advanced messaging such as that enabled by rabbitmq right now.

Why not django-q ? It is actively maintained. And targeted for Django. And offers a lot of features. But by the looks of it, it offered many features I was not going to be using. At the cost of being less lightweight than Huey.

The above does not mean dramatiq or django-q are not great packages. Far from it. I have them in mind in case the use case changes.