Hello Everyone,
I have a requirement of retrieving 5 million records from MySQL database and comparing each value with given value to remove duplicate values.
As I am using "chunk" method to fetch 0.05 million and putting into array. so 0.1 million chunk is running. It is taking nearly an hour or more than a hour to fetch all records from db.
I need to fetch all records from db and check is it duplicated or not, if not then insert new entry.
This is how I am doing. Is there any efficient method to fetch all records in minimal time?
TableModel::query()->select('user','user_status','user_rating')->orderBy('id')->chunk(50000, function($users)
use (&$user_listsArray,&$user_status_listsArray,&$user_rating_listsArray)
{
foreach ($users as $user)
{
$md5_listsArray[] = $user->user;
$ioc_listsArray[] = $user->user_status;
$rating_listsArray[] = $user->user_rating;
}
});
Thanks in advance.
Rather than looping over every item in the DB, can you run a select query to check for duplicates? I don't know exactly what you're trying to compare, but something like this might do the trick:
@Ben Sampson , I have to insert new user to the table which contains 5 million records. I have to check whether this new user is already present in table or not. If not, then insert user else update his rating.
Ok, I would approach it by looping over each new user to be added (can't tell if you're adding just 1 or more than 1 user at a time). You can run a select query to see if they exist in the table already, if they do, update them. If they don't then insert them.
Sign in to participate in this thread!
Laravel
Laravel News
Laracasts
Laravel Podcast