The core of the project is file block upload. The front and rear ends should be highly coordinated. The two sides need to agree on some data to complete the blocking of large files. We should focus on the following problems in the project.
How to slice;
How to compose a file;
Interrupt which partition to start from.
How to divide and make use of the powerful JS library to reduce our work. There can be wheels for large file blocks in the market, although the nature of programmers once forced me to rebuild the wheels. But because of time and work, I can only give up. Finally, I chose Baidu’s webuploader to realize the needs of the front end.
How to merge? Before merging, we have to solve a problem: how do we distinguish the file to which the partition belongs. At the beginning, I used the unique UUID generated by the front end as the file flag, which is carried on each partition request. But later, when I was doing second transmission, I gave up and used MD5 to maintain the relationship between blocks and files.
In fact, the industry has provided a good solution to the problem of merging files and recording blocks on the server. Referring to Xunlei, you will find that each time you download, there will be two files, one file body and the other is a temporary file. The temporary file stores the status of bytes corresponding to each block.
These can only be done by close contact between the front and rear ends. The front end needs to fragment the file according to the fixed size, and the fragment serial number and size should be brought in the request. After the request sent by the front end reaches the background smoothly, the server only needs to calculate the start position according to the fragment sequence number given in the request data and the block size of each slice (the fragment size is fixed and the same), and write it to the file together with the read file fragment data.
In order to facilitate development, I divide the business logic of the server into initialization, block processing, file upload, etc.
The business logic modules of the server are as follows
Folder generation module
After uploading the folder, the server will scan it. The code is as follows
Block upload and block processing logic should be the simplest logic. UP6 has blocked the file and identified each block data. These identifiers include the index, size, offset, file MD5, file block MD5 (to be opened) and other information of the file block. The server can process these information very conveniently after receiving them. For example, save block data to a distributed storage system
Block upload can be said to be the basis of our whole project, such as breakpoint continuation and pause.
The partition is relatively simple. The front end adopts Web loader, and the basic functions such as blocking have been encapsulated, which is convenient to use.
With the file API provided by webupload, the front end is extremely simple.
Foreground HTML template
When you are apart, you will be together. The large file is divided into pieces, but there is no original file function when it is divided, so we need to synthesize the pieces into the original file. We just need to write the partition to the file according to its original location. Because we have already mentioned the previous principle, we know the block size and block serial number, so I can know the starting position of the block in the file. Therefore, it is wise to use RandomAccessFile here. RandomAccessFile can move back and forth in the file. However, most of the functions of andomaccessfile have been replaced by the “memory mapped files” of NiO in jdk1.4. In this project, I wrote to use RandomAccessFile and mappedbytebuffer to synthesize files. The corresponding methods are uploadfilerandomaccessfile and uploadfilebymappedbytebuffer. The two method codes are as follows.
Second transmission function
I believe everyone has reflected the second transmission function. When uploading the online disk, I found that the uploaded file was transmitted in seconds. In fact, students who have studied the principle slightly should know that it is actually to check the file MD5, record the MD5 of the file uploaded to the system, obtain the MD5 value of the file content or part of the value MD5 before uploading a file, and then match the data on the system.
Breakpoint-http implements the principle of second transmission. After the client selects the file, click on the upload to get the MD5 value of the file. After getting the MD5, call the system interface (/index/checkFileMd5) to check whether the MD5 is already there (I use redis to store data in the project, use key to make key, value is the address of file storage.) the interface returns the check state. Then proceed to the next step. I believe you can understand it by looking at the code.
Well, the MD5 value of the front end also uses the functions of webuploader, which is a good tool.
Control will trigger MD5 after calculating the file MD5_ Complete event and pass the value MD5. The developer only needs to handle this event,
UP6 has automatically processed the breakpoint continuation, which does not need to be processed separately.
In F_ These parameters are received and processed in post.jsp. Developers only need to pay attention to business logic and do not need to pay attention to other aspects.
Breakpoint continuation means that the file upload process is interrupted. Human factors (suspension) or force majeure (disconnection or poor network) cause the file upload to fail in half. Then, when the environment is restored, re upload the file instead of starting from scratch.
As mentioned earlier, the function of breakpoint continuation is realized based on block upload. A large file is divided into many small blocks. The server can land each block successfully uploaded. The client calls the interface for rapid verification at the beginning of file upload, and the condition is to skip a block.
The implementation principle is to get the value of the file MD5 before uploading each file, and call the interface before the upload of the file (/index/checkFileMd5, which is also the test interface of seconds). If the status of the file is not completed, then the block number is not returned, and then the front end performs the conditional screen to calculate which unuploaded blocks and then uploads.
After receiving the file block, it can be written directly to the file of the server
This is the effect of file block upload
This is the effect after the folder is uploaded
This is the storage structure on the server after the folder is uploaded
Welcome to join the group to discuss: 374992201
This work adoptsCC agreement, reprint must indicate the author and the link to this article