Support the ongoing development of Laravel.io →
Laravel Database Eloquent

Hello Everyone,

I have a requirement of retrieving 5 million records from MySQL database and comparing each value with given value to remove duplicate values.

As I am using "chunk" method to fetch 0.05 million and putting into array. so 0.1 million chunk is running. It is taking nearly an hour or more than a hour to fetch all records from db.

I need to fetch all records from db and check is it duplicated or not, if not then insert new entry.

This is how I am doing. Is there any efficient method to fetch all records in minimal time?

TableModel::query()->select('user','user_status','user_rating')->orderBy('id')->chunk(50000, function($users)
    use (&$user_listsArray,&$user_status_listsArray,&$user_rating_listsArray)
    {
        
        foreach ($users as $user)
        {
          
            $md5_listsArray[] = $user->user;
            $ioc_listsArray[] = $user->user_status;
            $rating_listsArray[] = $user->user_rating;

        }
    });

Thanks in advance.

Last updated 3 years ago.
0

Rather than looping over every item in the DB, can you run a select query to check for duplicates? I don't know exactly what you're trying to compare, but something like this might do the trick:

https://stackoverflow.com/a/688551

0

@Ben Sampson , I have to insert new user to the table which contains 5 million records. I have to check whether this new user is already present in table or not. If not, then insert user else update his rating.

0

Ok, I would approach it by looping over each new user to be added (can't tell if you're adding just 1 or more than 1 user at a time). You can run a select query to see if they exist in the table already, if they do, update them. If they don't then insert them.

0

@Ben Sampson,Thanks. will approach as same as you.

0

Sign in to participate in this thread!

Eventy

Your banner here too?

DPS dps Joined 30 May 2018

Moderators

We'd like to thank these amazing companies for supporting us

Your logo here?

Laravel.io

The Laravel portal for problem solving, knowledge sharing and community building.

© 2025 Laravel.io - All rights reserved.