I have a set of basic classes that talk to my app and send emails upon queue events that come from an iron.io push queue. I have verified that my queue events are firing correctly and being received OK locally but on our staging environment, the queue seems to be jammed in some way.
Example app flow
1: User fills out a form, some data is created, a queue event is pushed to Iron.io: Queue::push('ConfirmationWorker@fire', $some_data, 'my_queue');
2: Worker looks roughly like this:
class ConfirmationWorker {
public function fire($job, $json) {
...
Mail::send(...)
$job->delete();
}
}
3: Some checks are done and then an email is fired
4: Job is deleted
Diagnosis
No issues on local and production - worked on this environment prior to changes today
Requests are being correctly fired to staging server by Iron.io - plus route is definitely accessible
Mail config is definitely correct
Iron.io is hitting the correct IP, verified with them
No failures seem to occur
Have tried restarting php (using php5-fpm), re-deployed app (totally new checkout of repo), restarted webserver (nginx), php artisan queue:restart
-- no luck
Potential causes
IP of staging environment changed today, but hostname remains the same + verified that environment is set to 'staging'
Earlier today, I had made a simple change to the job workers, but contained an exception. Some jobs were not deleted. But I cannot work out how to correctly free it up.
Running out of things to try! Note that I've repeatedly been triggering that app flow in the staging environment and a single mail hasn't fired since I deployed the syntax problem described above.
Thanks!
Turns out that this was due a problem on Iron.io's side. The IP of our staging environment got updated, but something on their end was cacheing the old IP. I could see tcp i/o timeout failures after attaching an errors queue to the push queue. 24h later, the problem is resolved as Iron is hitting the correct IP now.
Sign in to participate in this thread!
The Laravel portal for problem solving, knowledge sharing and community building.
The community