How to solve the problem of large file fragment upload with PHP

Time:2022-5-13

If the uploaded file is only less than 10m, there is no need to consider such an approach. It is directly in PHP Change upload in ini_ max_ filesize = 10m post_ max_ Size = 10m, that’s OK. Let’s talk about PHP uploading super large files

premise

First of all, when uploading super large files, the front end and the back end should cooperate with each other. The file upload should use ajax instead of form submit

thought

The front end divides the file object into files of a certain size (such as 2m or 5m). The divided files are uploaded to the back end one by one. After receiving the fragment files, the back end puts them into a temporary directory first. When receiving the data request completed by the front end, assemble the files in the temporary directory into a new file and save it, Just delete the files in the temporary directory

code

html

<div>
          Upload < input type = "file" name = "myfile" / >
   </div>

Upload here, no instructions for using ajax

javascript

<script>
	$(function(){
		let myfile = document.getElementById("myfile");
		myfile.onchange = function(){
			let file = myfile. files[0]; 		// Here you can get the uploaded file object
			let length = 1024 * 1024 * 5;   // Here is the size of each slice
			let total_ number = Math. Ceil (file. Size / length) // use the forward method to determine the number of slices
			let start = 0; 			// Initial position of slice
			let end = length; 		// End position of slice
			let parr = [];         // Here is promise The all method prepares an array;
			for(let i = 1;i<=total_number;i++){
				//Start slicing here and upload each slice to the server
				let bolb = file. slice(start,end);  // Get a slice
				start = end; 				// Adjust the starting position of the next slice
				end = start+length; 			// Adjust the end position of the next slice
				if(end > file.size){
					end=file. size; 		// Here, the end position of the last slice is adjusted
				}
				let formdata = new FormData();  // Create a formdata object and prepare to transfer data
				formdata. append("file",blob);   // Put the segmented data into formdata
				formdata. append("tempfilename",i+"_"+ file. Name) // at the same time, set a name for this partition, in which I can help the backend sort
				
				//After formdata is assembled, call the pro () function to return a promise object and put it into the parr array for the convenience of the promise All method usage
				parr.push(pro(formadata));
			}
			//After the above for loop ends, the parr array will be full of promise objects uploaded in pieces. At this time, we use promise All method, wait until all uploads are successfully executed, and then send a request to the server, that is, the request for the server to assemble fragments after the upload is completed
			Promise.all(parr).then(res=>{
				If (res.length = = parr. Length) {// if the length of the array returned successfully is equal to that of the parr array, it indicates that all fragments are uploaded successfully
					//At this time, send the request to the upload interface again, and bring the uploaded file name to facilitate the background to find the partition file name to be assembled. Because the same upload interface is requested, we also need to pass a flag = 1 to indicate that this is a data assembly request
 $.ajax({
                    type:"post",
                    url:"http://fastadmin.test/index/upload/getupload",
                    Data: {flag: 1, filename: file. Name}, // here flag = 1 indicates that the upload is completed and the assembly is requested. Filename: indicates which group of file fragments to form
                    success:function(res){
                        if(res.length == parr.length){
                            console.log(111);
                        }
                    },
                    fail: function () {
                        reject()
                    }
                })
				}
			})
		}
	})
	//This function is used to upload fragment files and return a promise object, which is convenient for later use All also determines whether all fragments are uploaded successfully
	//Let me explain here, $ Post () is not allowed to upload files. You can only use $ Ajax () and bring contenttype: false and processdata: false
    function pro(formData){
        return new Promise((resolve,reject)=>{
            $.ajax({
                type:"post",
                url:" http://fastadmin.test/index/upload/getupload ", // address of background upload file
                data:formData,
                Contenttype: false, // this cannot be less. Ajax upload files cannot be less
                processData: false, 	   // This can't be less. Ajax upload files must pass false
                success:function(res){
                    resolve(res)
                },
                fail: function () {
                    reject()
                }
            })
        })
    }		
</script>

The above is the JS core part of the front end, and the notes can be basically understood

php

TP5 framework used

public function getUpload(){
	$tempdir = APP_ PATH. "../public/tempdir"; // The fragmented files here specify a temporary directory, which will be used later
	$flag = input("flag",0);// Receive parameter flag if there is no such parameter, it defaults to 0. If flag = 1, it means to assemble pieces
	if($flag == 0){
		//Here is the upload fragment
		$file = request()->file("file");  // Received this fragment
		$tempfilename = input("tempfilename");  // Received the name of this partition. (note that this name contains sorting information)
		if(!file_exists($tempdir)){
			mkdir($tempdir,0755,true); 		// If the temporary directory does not exist, create a temporary directory
		}
		$fileinfo = $file->move($tempdir,$tmpfilename);
		If ($FileInfo) {// the fragmented files are saved in a temporary directory. The returned results are a little simple. You can return the corresponding data according to your own needs
			return josn(['error'=>0])
		}else{
			return json(['error'=>1])
		}
	}else if($flag == 1){
		//If the flag is 1, it means that the fragment has been uploaded
		$filename = input("filename");
		//Through the string matching of file name, find all fragments and return an array of file paths
		$fileArr = glob($tempdir."/*".$filename);
		//The * here is a wildcard, which can be found in all the documents of $filename contained in the file name
		//To illustrate, the order of arrays in $filearr is not what we want, so we create an array to sort out the order
		$newfileArr = [];
		foreach($fileArr as $f){
		//In the JS front end, we added the serial number + "______________, So we can get the file name, separate it by underlining, and write the order in the key
			$filebasename = basename($f); //$ F is the path one by one. Here, basename is used to get the file name
			$filebasenamesplit = explode("_",$ filebasename); // File names are separated by underscores, 
			$newfileArr[$filebasenamesplit[0]] = $f;    // A new array is constructed, in which the key of the array is the sequence number, and the value of the array is the path of the fragment file
		}
		//When the serial number and path of the pieces are ready, they can be assembled
		$num = count($newfileArr);  // The number of all fragments obtained is the criterion for using the for loop later
		//Start using the for loop to assemble
		$newfilename = "huangjunhui".$ filename; // Here is a name for the assembled document, which can be used at will
		for($i = 1;$i<=$num;$i++){
			file_put_contents($newfilename,file_get_contents($newfileArr[$i]),FILE_APPEND);
			//Here, the fragment files are written into one file by adding, 
		}
		......
		//Delete the partition file in the temporary file. Here, you can use try catch to judge whether there is an error
		foreach($newfileArr as $fi){
			unlink($fi);
		}
		//Finally, return the saved file name to the front end
	}
}

In the above method, I uploaded a 650m file in the local test. It took only 20 seconds and has not been tested on the server. You can try this method.