Back

Run multiple queue jobs the same time, with Beanstalk and Supervisor??


amityweb posted 4 years ago

I have an app that will loop through a lot of domains and run CURL on them.

With Laravel's Queues using sync, they run one after the other. So say it takes 1 second per call, it will take approx. 100 seconds for 100 domains.

What I would like is all of them to call the same time. I read online I should use Beanstalk as the queue driver, then run Supervisor to run artisan queue:work many times.

I have set all this up, but it still seems to call them one at a time.

I wondered if it is indeed possible to run them all the same time, and if so if I am missing something obvious.

Supervisor conf is as follows, I read I need set numprocs and process_name as I have to run 100 workers the same time.

[program:myprog]
command=php artisan queue:work
directory=/home/myprog
stdout_logfile=/home/myprog/app/storage/logs/supervisord.log
redirect_stderr=true
numprocs=100
process_name=%(program_name)s_%(process_num)s
autostart=true
autorestart=true

supervisord -n shows all the processes starting with the following lines, although each of those take like 1 second or so to start again, and I can see it stops. So wondered if its because these processes are slow to start and then stop anyway, I dont know.

2014-02-09 21:07:19,487 INFO spawned: 'myprog' with pid 14778
2014-02-09 21:07:20,490 INFO success: myprog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-02-09 21:07:22,669 INFO exited: myprog (exit status 0; expected)
2014-02-09 21:07:22,670 INFO received SIGCLD indicating a child quit

Thanks

amityweb replied 4 years ago

I THINK it may be I have supervisor v2 as its Centos 6 and multiple processes was introduced in supervisor 3. So going to see if I can upgrade and see what happens then.

amityweb replied 4 years ago

Installed Supervisor 3 using this process as its not in a Centos repo http://rayed.com/wordpress/?p=1496, but still seems slow. Doesn't seem its all working same time, although I can see all the work processes in top, so they''re running.

Its probably my measly AWS EC2 Mini instance, might try a better server.

stokic replied 4 years ago

Did it work for you after upgrading to a decent server? :)

fanshine replied 3 years ago

Hi, How about your attempt with supervisor? As I'm trying it now and it doesn't work for multi queues worker. A little confused

awnali replied 1 year ago

Any solution for this problem?


Sign in to participate in this thread!



We'd like to thank these amazing companies for supporting us