Exploring Redis III. Reliable queue.

Introduction

In this article, I'll be using Redis as a "reliable queue". As the articles before, I will try to keep it simple, is better this way.

Prerequisites

Same as previous article.

A reliable queue in Redis 🤔, hugh?

Well yes, it can be done and actually is pretty good. Let's show you a queue first, I won't enter in too much technical detail, because a queue is pretty intuitive. Take a look at this pretty sketch:

The idea is to have a FIFO(First in, first out) behavior with the queue. But having just this is not enough, because network problems could happen, or one of the parts that consumes from the queue just crashed, what happen with the element that has being processed?

With Redis we could manage this using LPUSH and BRPOPLPUSH, you could use also RPOPLPUSH as the non-blocking alternative. Let's see how to use these two commands. Having a redis-server, and two instances of redis-cli running,

On the first instance of the redis-cli, we push elements into the queue like this:

LPUSH tasks file1 file2 file3

tasks is the queue, and file1, file2 and file3 are the elements that we are pushing into the queue. Until now, we just have a simple queue, nothing reliable with this. Let's see how to pop elements from the queue.

BRPOPLPUSH tasks tasks_backup 10

Now, comes the reliable part. In this command we see two new things, one is the tasks_backup. The particularity of this command is that it pops an element from the queue tasks, but also push that same element to a backup queue, so in case the consumer crash, there's still a copy of the element in the backup queue. This, of course, adds more space consumption, after you have a successful pop you could just destroy this list since you don't need it anymore.

The 10, represents the timeout in seconds that you'll be waiting for an element in the queue, given that BRPOPLPUSH is a blocking operation you may not want to wait forever.

Example use case

Suppose that you have an application, that needs to download videos, given a request from a user. You could handle the download in the same application, but this is not so smart, because the time you start getting more traffic this could be a significant load for the application alone. Instead of managing the download by the application, you, as a pretty smart engineer, decide to have some workers to do all these heavy tasks.

Now, we need a way to communicate between the application and the workers. You could just establish a direct connection between the application and every single worker, 🤨, no no we want to add more workers as we need without bothering the main application with some nasty configuration. Instead, we are going to use a middleman, a broker, a good guy, we are going to use Redis, because this is an article about Redis of course.

General behavior, of our application, user ask for some video, application process request and push this as a task into Redis. Then the amazing workers, at the other end, pop this tasks, one or more remember we don't have just one user, that would be sad 😔.