I have an app that will loop through a lot of domains and run CURL on them.
With Laravel's Queues using sync, they run one after the other. So say it takes 1 second per call, it will take approx. 100 seconds for 100 domains.
What I would like is all of them to call the same time. I read online I should use Beanstalk as the queue driver, then run Supervisor to run artisan queue:work many times.
I have set all this up, but it still seems to call them one at a time.
I wondered if it is indeed possible to run them all the same time, and if so if I am missing something obvious.
Supervisor conf is as follows, I read I need set numprocs and process_name as I have to run 100 workers the same time.
[program:myprog] command=php artisan queue:work directory=/home/myprog stdout_logfile=/home/myprog/app/storage/logs/supervisord.log redirect_stderr=true numprocs=100 process_name=%(program_name)s_%(process_num)s autostart=true autorestart=true
supervisord -n shows all the processes starting with the following lines, although each of those take like 1 second or so to start again, and I can see it stops. So wondered if its because these processes are slow to start and then stop anyway, I dont know.
2014-02-09 21:07:19,487 INFO spawned: 'myprog' with pid 14778 2014-02-09 21:07:20,490 INFO success: myprog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2014-02-09 21:07:22,669 INFO exited: myprog (exit status 0; expected) 2014-02-09 21:07:22,670 INFO received SIGCLD indicating a child quit
I THINK it may be I have supervisor v2 as its Centos 6 and multiple processes was introduced in supervisor 3. So going to see if I can upgrade and see what happens then.
Installed Supervisor 3 using this process as its not in a Centos repo http://rayed.com/wordpress/?p=1496, but still seems slow. Doesn't seem its all working same time, although I can see all the work processes in top, so they''re running.
Its probably my measly AWS EC2 Mini instance, might try a better server.
Hi, How about your attempt with supervisor? As I'm trying it now and it doesn't work for multi queues worker. A little confused
Sign in to participate in this thread!