When you think of long-running processes and time-consuming tasks the first thing you will think of is queues, which is the best option out there, letting your users wait for an email to send while staring at a loading spinner is not a great UI experience, and in today’s fast-paced life who has the time to do this.

Laravel's core includes queues and a couple of great drivers, such as Sync, Database, Redis, AWS SQS, Beanstalk, and MongoDB. These drivers can be configured to complement the functionality and make the queues a highly scalable, reliable, and maintainable component of your system. But having your queues process your background processes is not just a matter of activating the feature, you will need a good understanding of how they run and process, best practices for optimizing the jobs, and have them run reliably.

When you don’t follow the rules for writing your jobs they can quickly turn into a nightmare of using a lot of disk space, running very long, and timing out in certain aspects, and you end up losing jobs and having to babysit your jobs which is not ideal since we want to reliably, send tasks to the background and they should process.

Queue Types

Sync

This is the default shipped with all new Laravel installs and the easiest way to start with your queues. No configuration is required, but this will still process the jobs as if you do not have any queues configured, and your user will wait for the action to complete before the request continues. This is great for development and local testing, but just not a good option for production environments.

Database

Using the database driver is good for local development and small applications. I know many people say don’t use a database driver in production but when you want to start a small SAAS to keep costs to a minimum while you don’t have a lot of traffic it is perfectly fine to use the database driver. If you process 5000 jobs a month it will not make a huge performance impact on your database. This is up to you and your judgment, you know your application and monitor it. I think of it the same as your CPU and RAM in your application you will know when you need to increase those.

To create the job tables you can run the following commands.

php artisan make:queue-table

php artisan migrate

Once this has been done change your queue drive in your .env to database.

QUEUE_DRIVER=database

Link to the documentation on the Laravel site for the database driver.

Redis

When using Redis as a queue driver it is advised to use Laravel Horizon to monitor and view your queued jobs. Horizon is an excellent option to run and gives you insight into how your jobs are performing and what queues are handling what kind of job loads.

When you have set up Redis on your server or created a separate Redis server & installed the Horizon package, change your queue drive in your .env to redis.

QUEUE_DRIVER=redis

Horizon Configurations

Horizon has many great config options to set and manage your queues, but sometimes this can get overwhelming, especially if you are only starting out with queues.

Balancing

You can choose between simple, auto & false. The simple strategy splits incoming jobs evenly between worker processes. The auto strategy, which is the configuration file's default, adjusts the number of worker processes per queue based on the current workload of the queue. When the balance option is set to false, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration.

Tries

Number of attempts to try a given job before it is marked as failed and moved to the failed_jobs table in your database.

Queues

You can specify queus to your config and also add multiple queues to a single configuration. When adding multiple queues, the order of the queue does make a significant difference.

Take the below config for example:

'supervisor-1' => [
      'connection' => 'redis',
      'queue' => ['sms_priority', 'sms'],
      'balance' => 'auto',
      'autoScalingStrategy' => 'time',
      'maxProcesses' => 1,
      'maxTime' => 0,
      'maxJobs' => 0,
      'memory' => 128,
      'tries' => 1,
      'timeout' => 60,
      'nice' => 0,
],

If there are 20 jobs in the sms queue it will start processing immediately, and 3 jobs gets added to the sms_priority queue horizon/redis will finish processing the current job it is busy with and start processing the 3 sms_priority jobs and once they are done it will continue with the rest of the jobs in the sms queue. This is a very nice little feature to group similar queues together instead of adding multiple queues as this can start getting very cumbersome to maintain.

Some things to look out for

Queue wait time

When running queues I usually run into the issue where we constantly get emails that a specific queue has reached a long wait time trigger. When you know some of the jobs might be running for long times you can set the wait times for each queue you configure. Setting these to reasonable values is a great option as you still want to get notified when the queues run for much longer than they should.

'waits' => [
    'redis:default' => 60,
    'redis:sms' => 14400, // 4 hours
    'redis:sms_priority' => 7200, // 2 hours
],

AWS SQS

The SQS driver is also a great option if you do not want to manage your own Redis servers and infrastructure, and AWS gives you a decent amount of requests (1 Million) with their free tier. This is a great option but comes with a bit of a learning curve, to set up the users and their permissions, as I don’t want to create a single user with all permissions or the root account, but that is a separate tutorial on it’s own.

When you have installed the AWS SDK change your queue drive in your .env to sqs.

QUEUE_DRIVER=sqs

Once this is done all your jobs will run and function as normal.

Other Drivers

With the other drivers, there are a couple of extra packages that you need to install to get them to work. https://laravel.com/docs/11.x/queues#other-driver-prerequisites

Rules

There are some boundaries, rules if you want to call them that when writing queues to keep in mind.

• Keep your Jobs lean, If your queued job accepts an Eloquent model in its constructor, only the identifier for the model will be serialized onto the queue. When the job is actually handled, the queue system will automatically re-retrieve the full model instance and its loaded relationships from the database. This approach to model serialization allows for much smaller job payloads to be sent to your queue driver.

• We have used repository patterns and service patterns which is great for keeping code standards and separation of concerns but passing in a huge service class to the constructor makes the job very bulky and uses a lot of disk space. We have moved to doing eloquent queries in the jobs or splitting out service classes to single responsibility and only passing the bare minimum to the job to make it more efficient.

What makes this such a powerful Laravel feature is that you can change providers and no other changes are required on your job classes or code.