How to Setup Queue Jobs in NextJs with BullMQ
Learn How to integrate BullMQ with Next.js to create a background job queue for processing tasks in the background. We will setup the BullMQ queue, create a job, and process the job in the background.
In the world of web development, Next.js is a popular choice for building web applications. It is a React framework that provides a great developer experience with features like server-side rendering, static site generation, and more. Next.js has become a viable full-stack framework that can be used to build complex and scalable web applications. When building complex web applications, we may need to perform asynchronous tasks in background in a sequence of order and the best way to do this is by using queues. In this article, we will explore how to use BullMQ with Next.js and Redis to create a background job queue for processing tasks in the background.
BullMQ is a Node.js library that provides a simple and reliable way to implement Redis-based queues in Node.js applications. It also provides advanced features such as job retries, job progress tracking, and concurrency control.
We will setup the BullMQ queue, create a job, and process the job in the background. Let's get started?
Installing Bullmq
First, install BullMQ and Redis within your Next.js project.
npm install bullmq ioredis --savenext step is to create a queue worker, which will pull the tasks from queue and process them.
We will create all our worker files inside src/workers directory (if you are not using src directory, you can put it in the project root folder.).
Create workers/test.worker.ts file.
// src/workers/test.worker.ts
import { Worker, Queue } from 'bullmq';
import Redis from 'ioredis';
const connection = new Redis(process.env.REDIS_URL?);
export const testQueue = new Queue('testQueue', {
connection,
defaultJobOptions: {
attempts: 3,
backoff: {
type: 'exponential',
delay: 3000,
},
},
});
const testWorker = new Worker(
'testQueue', // this is the queue name, the first string parameter we provided for Queue()
async (job) => {
const data = job?.data;
console.log(data);
console.log('job executed successfully');
},
{
connection,
concurrency: 5,
removeOnComplete: { count: 1000 },
removeOnFail: { count: 5000 },
}
);
export default testWorker;Now, we will add a script to our package.json file which will be use to process the queue. You also need to install dotenv and tsx package if not already installed.
npm install dotenv tsx --save// package.json
{
...
"scripts": {
...
"worker:test": "dotenv -e .env.local -- npx tsx --watch src/workers/test.worker.ts"
},
...
}make sure you add the correct path in script based on your directory structure.
Add jobs to the queue
Now we have created our test worker, we can add jobs to the queue. you can add the jobs to the queue from your server-side code (e.g., route handlers, API routes and server actions).
we will create an api route to add the job.
// src/api/add-test-job.ts if you are using pages directory
// src/app/add-test-job/route.ts if you are using app directory
import { NextResponse } from 'next/server';
import {testQueue} from '@/workers/test.worker';
export async function GET(){
const data = {
message: 'This is a sample job'
}
await testQueue.add('testJob', data,{ removeOnComplete: true });
return NextResponse.json({'status': 'job added to the queue'});
}When you visit your route URL, it will add the job in queue.
Running the worker
Now its time to process our test queue. To run the worker, do to your terminal and navigate to your project directory and run the below command :
npm run worker:test
It will process the pending queue and if hit your route again and add another job, it will process automatically.
Cool, Thats It? You have successfully integrated BullMQ with NextJs and Redis. You can now start building your own custom queue system with BullMQ. I hope you enjoyed this tutorial. Please share it with your friends and colleagues and don't forget to follow me on Twitter and Github.
