Running custom Artisan commands with Supervisor

If you ever used Laravel queues on the server, you probably came across a section in the <a title="Running Supervisor" href="https://laravel.com/docs/10.x/queues#supervisor-configuration" target="_blank" rel="noopener noreferrer">documentation</a> about needing to run a Supervisor on your server in order to keep your queue processes going in case they are stopped for whatever reason. What you would typically need to do is <a title="Configuring Supervisor" href="https://laravel.com/docs/10.x/queues#configuring-supervisor" target="_blank" rel="noopener noreferrer">configure</a> the Supervisor to run the queue:work Artisan command.

I won't get too much into detail now on how different Artisan commands regarding queues are functioning behind the scenes, but if we take a closer look at the queue:work command, for example, essentially what is going on is that the jobs are being run inside a while loop, and if certain conditions are met, the worker will be stopped and the command will exit. After that, the Supervisor will run it again. 

Here is an example of what a trimmed-down and simplified version of that loop might look like (taken from the daemon function inside the Illuminate/Queue/Worker.php file):

while (true) {
    // Get the next job from the queue.
    $job = $this->getNextJob(
        $this->manager->connection($connectionName), $queue
    );

    // Run the job if needed.
    if ($job) {
        $jobsProcessed++;

        $this->runJob($job, $connectionName, $options);
    }

    // Get the worker status.
    $status = $this->stopIfNecessary(
        $options, $lastRestart, $startTime, $jobsProcessed, $job
    );

    // Stop the worker if needed.
    if (! is_null($status)) {
        return $this->stop($status, $options);
    }
}

Now, if we were to take that example as a starting point, we could create our own command that could be run indefinitely (if needed) with the help of the Supervisor.

Let's say we created an app for the Handyman Tools Store. The user selects an item, fills in the necessary data, and submits an order for a specific tool. As soon as the order is created, it is automatically set to a pending status. To be able to monitor that kind of order and include some logic of our own, we could create a custom Artisan command that would check for all pending orders from time to time, and if any are found, we could dispatch a Laravel job that would further process that order (maybe check if the tool is in stock, can it be shipped to the user's location, and send an email to the user about that order status).

Here is an example of such a command:

namespace App\Console\Commands;

use Illuminate\Console\Command;

class MonitorOrders extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'orders:monitor';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Monitor pending tool orders and dispatch them for processing.';

    /**
     * Execute the console command.
     */
    public function handle()
    {
        while (true) {
            // Get all the orders with the pending status
            $orders = Order::where('status', 'pending')->get();

            // If there are no orders with a pending status wait for a second then check again
            if ($orders->doesntExist()) {
                sleep(1);

                continue;
            }

            foreach ($orders as $order) {
                // Dispatch the order for further processing to a job
                ProcessOrder::dispatch($order)
                    ->onQueue('orders')
                    ->onConnection('database');
            }

            sleep(1); // Potentially take a break between runs
        }
    }
}

What if we wanted to specify the number of jobs that should be processed,  similar to the <a title="Specify number of jobs" href="https://laravel.com/docs/10.x/queues#processing-a-specified-number-of-jobs" target="_blank" rel="noopener noreferrer">--max-jobs </a>flag? If the maximum number of jobs is reached the worker could exit and release any memory that may have accumulated. To achieve that, we could introduce a new option to our command, like this:

/**
 * The name and signature of the console command.
 *
 * @var string
 */
protected $signature = 'orders:monitor
                        {--max-jobs=100 : The number of jobs to process before stopping}';

Then, in our handle method, we can set the initial number of the processed jobs and get the maximum number from the option we defined previously:

/**
 * Execute the console command.
 */
public function handle()
{
    $jobsProcessed = 0;
    $maxJobs = $this->option('max-jobs');

    while (true) {
            // Get all the orders with the pending status
            $orders = Order::where('status', 'pending')->get();

As soon as the job is sent to be processed, we will increment the $jobsProcessed variable and check if the worker should exit. If that is the case, we would just return a default success exit code (0). After the worker exits, the Supervisor will restart it again.

foreach ($orders as $order) {
    // Dispatch the order for further processing to a job
    ProcessOrder::dispatch($order)
        ->onQueue('orders')
        ->onConnection('database');

    // Increase the number of processed jobs
    $jobsProcessed++;

    // Stop the command if the number of jobs reaches the maximum number set
    if($jobsProcessed >= $maxJobs) {
        return 0; // Success
    }
}

When you update the code inside of the command and deploy it to the server, the change won't be reflected until the Supervisor has had a chance to actually restart the worker. To resolve that issue, you could restart the Supervisor itself on deploy (and reread the configuration file if you updated it):

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl restart

Or better yet, we could create another custom command that will cause our initial orders:monitor command to exit, again allowing the Supervisor to restart it. We could do that by using the built-in Cache functionality, similar to how the <a title="Queue restart" href="https://laravel.com/docs/10.x/queues#queue-workers-and-deployment" target="_blank" rel="noopener noreferrer">queue:restart</a> command does it.

Here is an example of how our orders:restart command could look like:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\Cache;

class RestartOrders extends Command
{
    /**
     * The console command name.
     *
     * @var string
     */
    protected $name = 'orders:restart';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Restart orders monitor command after the current job.';

    /**
     * Execute the console command.
     *
     * @return void
     */
    public function handle()
    {
        Cache::forever('orders:restart', true);
    }
}

Now, we need to amend our orders:monitor command to check if the orders:restart has been triggered. Make sure to include the Cache facade first:

use Illuminate\Support\Facades\Cache;

And then update the loop like so:

foreach ($orders as $order) {
    // Dispatch the order for further processing to a job
    ProcessOrder::dispatch($order)
        ->onQueue('orders')
        ->onConnection('database');

    // Increase the number of processed jobs
    $jobsProcessed++;

    // Stop the command if the number of jobs reaches the maximum number set
    if($jobsProcessed >= $maxJobs) {
        return 0; // Success
    }
    
    // Get the restart command value from the cache
    $restartCommand = Cache::get('orders:restart', false);

    // Stop the command if needed and remove the entry from the cache
    if($restartCommand) {
        Cache::forget('orders:restart');
        return 0; // Success
    }
}

That's it. Now you can add the  orders:restart command to your deploy procedure, and you won't have to worry about any changes in your code not being reflected.

As per usual, you can check out the <a title="Gist example" href="https://gist.github.com/goran-popovic/2aac3715ce25588941e23889b44c88ba" target="_blank" rel="noopener noreferrer nofollow">Gist</a> for source code.

What's left for us to do is add a supervisor configuration file, named, for example, monitor-orders.conf, in the /etc/supervisor/conf.d directory (location of the directory in most cases) that starts (and restarts, if needed) our command, which will in turn continually monitor for any pending tool orders.

[program:monitor-orders]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan orders:monitor --max-jobs=300
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=forge
numprocs=8
redirect_stderr=true
stdout_logfile=/home/forge/app.com/worker.log
stopwaitsecs=3600

The setup above uses an example from the documentation, but you may not need to spawn 8 processes (numprocs), depending on the number of orders you receive. The path to the Artisan command on your server might be completely different; it could be something like php /var/www/my-website.com/artisan so make sure to double check that as well as the user value and the path to the worker.log file.

We made it. Following the basic example of the queue:work command and the instructions from the Laravel documentation, we were able to create our own custom command that will monitor for pending orders in our app for Handyman Tools. With the help of the Supervisor, the command will be run automatically in the background and restarted if anything unexpected happens. 

Of course, the examples above are just a start; you could build a lot more complex logic around them; you could introduce more conditions by which the pending orders should be processed or not; you could add other useful options and arguments; and so on. 

All in all, I hope this article helped you get a better understanding of Artisan commands that are run with the help of the Supervisor.