Back

Laravel 5.8 memory leak


Demo Repo

https://github.com/fukuball/Leak58

Laravel memory leak example

It is normal to loop and process data in script, I found some weird memory leak in Laravel, and wonder how this happened. I have some workaround to prevent the memory leak, but it can't solve the root cause, so I provide some cases to demo the memory leak, hope someone can solve this issue.

Install

$ composer install
$ php artisan migrate

Data seeding

$ php artisan leak_test_data

Usage

Case 1

This first case demo a simple loop cause the memory leak:

$ php artisan leak_test leak

Some details:

$albums = Album::take(10000)->get();
foreach ($albums as $album) {
    $songs = $album->songs; // cause memory leak
}

And we can see the memory goes up and never come back:

...
550 start
#executions = 550 - mem: 14799640
550 end
551 start
#executions = 551 - mem: 14801728
551 end
552 start
#executions = 552 - mem: 14803800
552 end
553 start
#executions = 553 - mem: 14805872
553 end
554 start
#executions = 554 - mem: 14807944
554 end
555 start
#executions = 555 - mem: 14810024
555 end
556 start
#executions = 556 - mem: 14812112
556 end
557 start
#executions = 557 - mem: 14814192
557 end
...

I know there is N+1 query in it, but simple loop with simple qurey should not cause memory leak, it happend in Laravel.

Case 2

This second case demo a simple loop with N+1 query, but no memory leak:

$ php artisan leak_test no_leak

Some details:

$albums = Album::take(10000)->get();
foreach ($albums as $album) {
    $songs = $album->songs()->get(); // why this don't cause the memory leak?
}

We can see the memory usage is stable:

...
439 start
#executions = 439 - mem: 13659472
439 end
440 start
#executions = 440 - mem: 13659456
440 end
441 start
#executions = 441 - mem: 13659464
441 end
442 start
#executions = 442 - mem: 13659456
442 end
443 start
#executions = 443 - mem: 13659456
443 end
444 start
#executions = 444 - mem: 13659464
444 end
445 start
#executions = 445 - mem: 13659456
445 end
446 start
#executions = 446 - mem: 13659464
446 end
...

This is reasonable, although there is N+1 query, but should not cause memory leak.

Case 3

Third case demo a simple loop and use "with" to solve N+1 query.

$ php artisan leak_test leak_solve_by_with

Some details:

$albums = Album::with(['songs'])->take(10000)->get();
foreach ($albums as $album) {
    $songs = $album->songs; // cause memory leak
}

We can see the memory usage is always same:

...
1239 start
#executions = 1239 - mem: 16015944
1239 end
1240 start
#executions = 1240 - mem: 16015944
1240 end
1241 start
#executions = 1241 - mem: 16015944
1241 end
1242 start
#executions = 1242 - mem: 16015944
1242 end
1243 start
#executions = 1243 - mem: 16015944
1243 end
1244 start
#executions = 1244 - mem: 16015944
1244 end
1245 start
#executions = 1245 - mem: 16015944
1245 end
1246 start
#executions = 1246 - mem: 16015944
1246 end
1247 start
#executions = 1247 - mem: 16015944
1247 end
...

This is trivial, this solve the N+1 query, get all the data first, and Laravel use the data and no need to query again and again, so if we can get all the data into the memory, the script will excute perfectly.

Case 4

This case demo a common case when we write OOP, we use use some method in model, and model will get the necessary data to proceed the work.

$ php artisan leak_test leak_weird

Some details:

$albums = Album::take(10000)->get();
foreach ($albums as $album) {
    $songs = $album->processSomethingToReturn();
}

// in Album.php
public function processSomethingToReturn()
{
    $songs = $this->songs; // this cause memory leak
    // do something here...
    return $songs;
}

And we can see the memory goes up and never come back:

...
453 start
#executions = 453 - mem: 14598224
453 end
454 start
#executions = 454 - mem: 14600296
454 end
455 start
#executions = 455 - mem: 14602376
455 end
456 start
#executions = 456 - mem: 14604456
456 end
457 start
#executions = 457 - mem: 14606528
457 end
458 start
#executions = 458 - mem: 14608616
458 end
459 start
#executions = 459 - mem: 14610688
459 end
460 start
#executions = 460 - mem: 14612760
460 end
461 start
#executions = 461 - mem: 14614840
461 end
...

This is common to write some method for "Encapsulation", we don't need to know the detail, just call the method to do what we want. But in this case we got memory leak.

Case 5

In this final case we use walkaround to solve the memory leak by "with" magic:

$ php artisan leak_test leak_solve_by_with_weird

Some details:

$albums = Album::with(['songs'])->take(10000)->get();
foreach ($albums as $album) {
    $songs = $album->processSomethingToReturn();
}

// in Album.php
public function processSomethingToReturn()
{
    $songs = $this->songs; // this cause memory leak
    // do something here...
    return $songs;
}

We can see the memory usage is always the same:

...
1240 start
#executions = 1240 - mem: 16015952
1240 end
1241 start
#executions = 1241 - mem: 16015952
1241 end
1242 start
#executions = 1242 - mem: 16015952
1242 end
1243 start
#executions = 1243 - mem: 16015952
1243 end
1244 start
#executions = 1244 - mem: 16015952
1244 end
1245 start
#executions = 1245 - mem: 16015952
1245 end
1246 start
#executions = 1246 - mem: 16015952
1246 end
1247 start
#executions = 1247 - mem: 16015952
1247 end
1248 start
#executions = 1248 - mem: 16015952
1248 end
...

This walkaround solve the memory leak, but really wired, in "Encapsulation" principle, we shoud not to know the detail of method, so we use with(['songs']) in advence is really wired, this should not happend when we write code.

Apparently we shold solve the root cause of memory leak. Why $this->songs in loop cause memory leak but $this->songs()->get() not?

Tobias van Beek replied 2 weeks ago

Case 1 is logic, because you "magic" call a relation that isn't loaded before Laravel will load the relation. It will not automaticly free it during the loop because it doesn't know that it isn't needed anymore.

Case 2 calls the query builder on the object, get the set from it and place that in a variable. After each iteration the $songs variable is cleared and so the memory isn't needed. Because you call the relation function for the query builder but not the relation itself it isn't stored on the $album. That is the reason you don't get an increase in memory.

Case 3, everything is loaded on the start so the memory keeps the same.

Case 4, same idea as case 1. You load the relation and the $album variable isn't removed so the memory is increasing.

Case 5, same as case 3. Everything is loaded on the start so the memory keeps the same.

In basic the difference is that $model->relation will load the relation if that isn't done yet and save the data on the model. While $model->relation()->get() will use the query builder where you get the result while it isn't stored in the $model variable.

Fukuball Lin replied 1 week ago

@tvbeek I try to unset the variables every iteration, the memory still grows, is there any way to free the memory?

$albums = Album::take(10000)->get();
foreach ($albums as $album) {
    $songs = $album->songs; // cause memory leak
    unset($songs);
    unset($album); // can't free the memory, I think in this case is memory leak
}

Sign in to participate in this thread!



We'd like to thank these amazing companies for supporting us