Fork me on GitHub

Project Notes

Huey Task Priority

Using the huey task priority feature.

Notes

Huey added support for Task priority since >=2.1.0.

Demo Setup

Setup pre-requisites..

$ python --version
Python 3.7.3
$ pip install -r requirements.txt
...

Huey requires redis - the demo is setup to expect redis to be available on localhost:6379.

Demo Program

Client

The demo/client.py simply runs a batch of tasks

$ python client.py
enqueuing tasks...
... lo_priority_task 0
... hi_priority_task 0
... lo_priority_task 1
... hi_priority_task 1
... lo_priority_task 2
... hi_priority_task 2
... lo_priority_task 3
... hi_priority_task 3
... lo_priority_task 4
... hi_priority_task 4
... lo_priority_task 5
... hi_priority_task 5
... lo_priority_task 6
... hi_priority_task 6
... lo_priority_task 7
... hi_priority_task 7
... lo_priority_task 8
... hi_priority_task 8
... lo_priority_task 9
... hi_priority_task 9
collating task results...
all tasks complete. results:
lo_priority_task 0 completed at 1629903692.143894
hi_priority_task 0 completed at 1629903693.145322
hi_priority_task 1 completed at 1629903694.150147
hi_priority_task 2 completed at 1629903695.154731
hi_priority_task 3 completed at 1629903696.160894
hi_priority_task 4 completed at 1629903697.1665761
hi_priority_task 5 completed at 1629903698.167873
hi_priority_task 6 completed at 1629903699.1702242
hi_priority_task 7 completed at 1629903700.171425
hi_priority_task 8 completed at 1629903701.1741529
hi_priority_task 9 completed at 1629903702.1754398
lo_priority_task 1 completed at 1629903703.1772852
lo_priority_task 2 completed at 1629903704.1804092
lo_priority_task 3 completed at 1629903705.184512
lo_priority_task 4 completed at 1629903706.189753
lo_priority_task 5 completed at 1629903707.191603
lo_priority_task 6 completed at 1629903708.192822
lo_priority_task 7 completed at 1629903709.194748
lo_priority_task 8 completed at 1629903710.195903
lo_priority_task 9 completed at 1629903711.199563

Tasks

The demo/tasks.py module implements two asynchronous task:

  • lo_priority_task - runs with lower priority
  • hi_priority_task - runs with higher priority

To use task priority, PriorityRedisHuey is used instead of RedisHuey.

Then tasks can be defined with a priority (high priorty number runs before low priority number) e.g.

@huey.task(priority=10)
def lo_priority_task(serial):
    ...

Running a single worker by default, with period tasks enabled:

$ cd demo/
$ huey_consumer.py tasks.huey
[2021-08-25 23:01:27,602] INFO:huey.consumer:MainThread:Huey consumer started with 1 thread, PID 78042 at 2021-08-25 15:01:27.602097
[2021-08-25 23:01:27,602] INFO:huey.consumer:MainThread:Scheduler runs every 1 second(s).
[2021-08-25 23:01:27,602] INFO:huey.consumer:MainThread:Periodic tasks are enabled.
[2021-08-25 23:01:27,602] INFO:huey.consumer:MainThread:The following commands are available:
+ tasks.lo_priority_task
+ tasks.hi_priority_task
[2021-08-25 23:01:31,143] INFO:huey:Worker-1:Executing tasks.lo_priority_task: 3bf71e75-a2d8-4a9a-92ac-0b3d17d9819d p=10
lo_priority_task 0 completed at 1629903692.143894
[2021-08-25 23:01:32,143] INFO:huey:Worker-1:tasks.lo_priority_task: 3bf71e75-a2d8-4a9a-92ac-0b3d17d9819d p=10 executed in 1.000s
[2021-08-25 23:01:32,145] INFO:huey:Worker-1:Executing tasks.hi_priority_task: 48e31aef-01b6-4719-ab07-45c20e0fdb44 p=20
hi_priority_task 0 completed at 1629903693.145322
...

Credits and References

About LCK#187 pythonhueyasyncqueues
Project Source on GitHub Return to the Project Catalog

This page is a web-friendly rendering of my project notes shared in the LittleCodingKata GitHub repository.

LittleCodingKata is my collection of programming exercises, research and code toys broadly spanning things that relate to programming and software development (languages, frameworks and tools).

These range from the trivial to the complex and serious. Many are inspired by existing work and I'll note credits and references where applicable. The focus is quite scattered, as I variously work on things new and important in the moment, or go back to revisit things from the past.

This is primarily a personal collection for my own edification and learning, but anyone who stumbles by is welcome to borrow, steal or reference the work here. And if you spot errors or issues I'd really appreciate some feedback - create an issue, send me an email or even send a pull-request.