Chunked File Upload with Livewire

Today we upload a file in chunks using Livewire. Upload your files close to your users with Fly.io, you can get your Laravel app running in minutes!

Servers are configured to limit the size of requests they can accept. This is done to avoid long processing times, resulting unavailability, and potential security risks that come with processing large requests in one go

What happens when a user requests to upload a file exceeding the configured limits? Aptly, the upload will fail, either with a custom message we write or the default 413 status code error returned by our server.

We are faced with a predicament dealt to developers before us, and will be dealt to developers long after ourselves: processing large file uploads.

The Problem

An obvious approach to take is to update our configuration limits. We can increase several configuration limitations in our server itself, and PHP's.

The problem is, this is not quite dynamic. File size uploads can increase overtime, and that leaves us to re-configure our limitations. And with increased limitations, come increased request processing times as well.

Isn't there a way we can avoid this perpetual re-configuration and increased request processing time?

The Solution

The solution isn't always about increasing limitations, sometimes simply adjusting an existing approach is all it takes to save the day. Instead of sending the whole file, why not send it in batches?

That's right! Today, we will not alter our configuration under the pressure of evolving limitations. Instead, we will resolve this without change to our configuration.

Today we'll slice, dice, and merge file chunks—with Livewire!

The Plan

We have a three-step plan for slicing and merging:

1. We'll first let Livewire know the expected total $fileSize to receive from all the chunks merged.
2. Then we start slicing, uploading, and merging chunks into a "final file" in our server one after another using Livewire.
3. Once our final file's size reaches the given $fileSize, this means all chunks have been merged. Therefore we feed the final file to Livewire's TemporaryUploadedFile class in order to utilize Livewire's uploaded file features.

For piecing together everything here, you can visit our repository's readme and inspect the relevant files.

The View

Let's start with creating a Livewire Component by running the command php artisan make:livewire chunked-file-upload. Afterwards, update our Livewire view to include a form tag enclosing both a file input element and a button for submission.

<form wire:submit.prevent="submit">
  <input type="file" id="myFile"/>
  <button type="button" onClick="uploadChunks()">Submit</button>
</form>

Every time the user clicks on the submit button, our custom JavaScript function uploadChunks() will slice a selected file into chunks and request Livewire to upload each chunk.

Sharing Expectations

In order to upload a large file, we'll be slicing it into smaller chunks that are within our server's request size limits. We'll upload each chunk one after another so that we can immediately merge an uploaded chunk into a "final file".

But, how exactly will the server know all the chunks have been merged to our final file? It'll need to know the expected final size of our final file of course!

Livewire properties are perfect for sharing information from client to server, so let's include information about our file like its $fileName and $fileSize as public attributes in our Livewire component. Today, we'll separate our file into 1MB chunks, so let's declare a separate attribute for the chunk uploaded $fileChunk and the expected maximum chunk size $chunkSize:

// app/Http/Livewire/ChunkedFileUpload.php
public $chunkSize = 1000000; // 1 MB
public $fileChunk; 

public $fileName;
public $fileSize; 

Let's go back to our Livewire view and revise the uploadChunks() function triggered by our submit button. Every time the user submits a file for upload, we'll set values for $fileName and $fileSize to be later sent to our Livewire component:

// resources/views/livewire/chunked-file-upload.blade.php
function uploadChunks()
{
    const file = document.querySelector('#myFile').files[0];

    // Send the following later at the next available call to component
    @this.set('fileName', file.name, true);
    @this.set('fileSize', file.size, true);

Notice we're using Livewire's set method here. This let's us set a public attribute in our client, but not make an immediate call to the server. The changes to $fileName and $fileSize will be sent to Livewire in the next immediate component request.

Now that our final file details are ready to be shared with our Livewire component, we can take a slice at our first chunk, starting at the file's 0th byte:

    livewireUploadChunk( file, 0 );
}

Slicing A Chunk

How do we slice a chunk out of our file?

Well, we'll need to know where the chunk starts, and where it ends. For the first chunk of our file, the starting point is a given: 0. But how about where the chunk ends?

The end of a chunk is always going to be 1MB( our $chunkSize ) from the starting point of the chunk, or the file size—whichever is smaller between the two:

// resources/views/livewire/chunked-file-upload.blade.php
function livewireUploadChunk( file, start ){
    const chunkEnd  = Math.min( start + @js($chunkSize), file.size );
    const chunk     = file.slice( start, chunkEnd ); 

Now that we have our chunk, we'll have to send it up to our server. We can use Livewire's upload JavaScript function to upload and associate the chunk with our $fileChunk attribute declared above:

    @this.upload('fileChunk', chunk);

After uploading the first chunk, let's also send out the next. We'll need to make sure the current chunk is completely uploaded though, for this, we can hook onto the upload function's event progress callback:

-    @this.upload('fileChunk', chunk);
+    @this.upload('fileChunk', chunk,(uName)=>{}, ()=>{}, (event)=>{
+        if( event.detail.progress == 100 ){
+          // We recursively call livewireUploadChunk from within itself
+          start = chunkEnd;
+          if( start < file.size ){
+            livewireUploadChunk( file, start );
+          }
+        }
+    });
}

The upload is completed when the value of event.detail.progress reaches 100. Once it does, we recursively call the current function livewireUploadChunk() in order to upload our next chunk.

The file.slice method's range is exclusive of the chunkEnd. For example, the range of slice(0,10) actually means 0 until 9, but not 10! This means our next starting point will be chunkEnd.

Saving and Merging

Now that our Livewire view's JavaScript is set up for slicing and uploading chunks, we've come to the final part of our slice and dice journey: saving and merging chunks in our Livewire component!

We'll be using Livewire's WithFileUploads trait to make our file upload a breeze. This trait allows us to declare an attribute that's uploadable—$fileChunk in our case!

// app/Http/Livewire/ChunkedFileUpload.php

+ use WithFileUploads;

// Chunks info
public $chunkSize = 1000000; // 1M
public $fileChunk;

// Final file 
public $fileName;
public $fileSize;

+ public $finalFile;

Once a chunk has been uploaded, Livewire must merge it into a "final file". In order to do this, we'll have to intercept Livewire's flow after our chunk has been uploaded.

Luckily for us, Livewire provides "hooks" we can use to intercept Livewire's lifecycle flow for our public attributes. In our specific case, we can hook on to the updated hook for our $fileChunk attribute.

From our updatedFileChunk hook we'll retrieve the file name generated by Livewire for a current chunk using the getFileName() method:

public function updatedFileChunk()
{
    $chunkFileName = $this->fileChunk->getFileName();

Then we'll merge this chunk into our final file, and delete the chunk once merged:

      $finalPath = Storage::path('/livewire-tmp/'.$this->fileName);
      $tmpPath   = Storage::path('/livewire-tmp/'.$chunkFileName);
      $file = fopen($tmpPath, 'rb');
      $buff = fread($file, $this->chunkSize);
      fclose($file);

      $final = fopen($finalPath, 'ab');
      fwrite($final, $buff);
      fclose($final);
      unlink($tmpPath);

Eventually all chunks will arrive one by one and get merged into our final file. To determine whether all chunks have been merged, we simply compare the final file's size with the expected $fileSize.

Of course, this newly generated file is our custom file. We'll need to enclose it in Livewire's TemporaryUploadedFile class in order to utilize Livewire's uploaded file features.

      $curSize = Storage::size('/livewire-tmp/'.$this->fileName);
      if( $curSize == $this->fileSize ){
          $this->finalFile = 
          TemporaryUploadedFile::createFromLivewire('/'.$this->fileName);
      }
}

Don't forget to import the TemporaryUploadedFile class, and declare a new public attribute $finalFile!

Say for example, previewing a new, temporary image in our view like so:

@if ($finalFile)
    Photo Preview:
    <img src="{{ $finalFile->temporaryUrl() }}">
@endif

Implementations are generally just so much more smooth with Livewire, uploading file chunks is no different!