Best Practice...Call API within loop.

posted 5 years ago

QUESTION TO THE FORUM, what is best practice to call GET API many-many times within a loop?

I have ~1,000 employees. I need to loop through each of them...no problem there.

Within my loop, for each employee, I need to reach out to a 3rd party server via an API, GET a few of the data elements sent back to me...and store those elements to mySQL

I have it running using PHP and calling cURL each loop interaction...BUT IT IS VERY_VERY_VERY SLOW.

foreach (employee) {

 STEP #1 GET from API URL: http://www.apidomain.com/users?employeeID=.employeeID

 STEP #2 based upon response from the GET API, store the data retuned from the API to mySQL

}

Thanks for advice!!!, Tom

Last updated 3 years ago.

benoit1980

replied 5 years ago

I think your problem is how everything is structured.

It is was me, I would have created the fetch employees method on the API side of things, then create only 1 single API request in my front end app.

If you cannot have this implemented on the API side and have to send 1 single request to their API each time you want to pull and employee, I am sorry to say that it will become a very non efficient way to access the data.

You really need to pull them all in 1 request and perhaps set a scheduling every 5 minutes that pulls the new data to your server: https://laravel.com/docs/6.x/scheduling

This is of course only if the data is not edited from your side(only as a view), otherwise mismatch will happen if someone changes the data on the API side + your side. In this case you will need to use the laravel transactions to lock the table when editing.

tlombard911

replied 5 years ago

Thankyou @bennyboy

I agree with your assessment , and unfortunately I have no control what APIs the 3rd party make available.
I can only do a GET employee data based upon one employee ID at a time, so I am forced to send 1 single request to their API each time you want to pull and employee.

Curious what you think is the best option to make those single requests within an undetermined # of employees in a loop....just do a cURL call for every employee?

Thoughts?

benoit1980

replied 5 years ago

Yes that's right, with this type of setup you are going to be really stuck. There is no way around I am afraid :-( You will have to do 1 call by employee because the API only let you do that. A lot of requests for nothing, sad.

If you are only viewing the data locally(not editing it), the other option you have is to do all your requests every 10 minutes in one go(cron job) and have a "copy" of all the employees in your local database. Try to find out how many calls you can do per hour, if not a problem, reduce this to 5 minutes. But....what happens if someone at the other end amends a detail and you have a copy of 5 minutes old? What you could do is send another async request behind the scene when you open the employee record(in your local copy) and have this request to double check and see if your local records match the remote records. At least by doing this you won't have to wait to access the record as you will access a 5 minutes old record each time but have some kind of updates always running in the background that notifies you that the record you are currently reading is incorrect. Still inefficient in term of requests but that's the only solution I could think of to open your user profile in a much faster way.

Moderators