The way we handle a similar flow (importing anywhere from 1 day to a few years of data), we have one master command that simply queues the smaller processes so they can be processed one by one. The actual flow is:
That way if one job fails, you don't lose the jobs behind it. You can also add more concurrency by adding more workers.
Sign in to participate in this thread!
We'd like to thank these amazing companies for supporting us