The way we handle a similar flow (importing anywhere from 1 day to a few years of data), we have one master command that simply queues the smaller processes so they can be processed one by one. The actual flow is:
- Either run a cron every night that runs php artisan import, or manually run the command
- The ImportCommand will loop through all accounts and either the previous day or a date range if specified
- For each day that needs to be imported, we queue a separate job on a queue
- Our queue listener will pull from that queue and process days individually
That way if one job fails, you don't lose the jobs behind it. You can also add more concurrency by adding more workers.
Last updated
1 year ago.
0
Sign in to participate in this thread!
Moderators
-
driesvints
Joined 16 Dec 2013
-
joedixon
Joined 23 Jun 2014
-
tvbeek
Joined 8 Jan 2018
-
tomhatzer
Joined 28 Oct 2021
Laravel
Laravel News
Laracasts
Laravel Podcast