I have built a business application using Laravel that utilizes several very large datasets. There is a significant amount of processing and calculation that must be performed to prepare some aggregate data for display. Obviously, we are using Cache to prevent the user from having to wait for the pre-processing to occur, when possible. In some cases, it can take more than 30 seconds to crunch all the data and prepare the page for display. The datasets are refreshed by services that run every 4 or 24 hours depending on the dataset and so it is necessary to expire the cache to make sure that the data on the application does not get too stale. This means that about every 4 hours, some unlucky user is going to have to have an extremely long page load time.

I am looking for ideas that folks might have about how to re-structure the application in such a way so that, when a service runs to update a dataset in our database, a process/job/queue would run for the application and would update any affected cached objects. The idea is that we could set the cache to never expire and then force cache updates related to the data being refreshed rather than the user triggering its expiration and re-generation.

Any ideas, code, or psuedo-code are appreciated!