After using HTTP for so long, do you know the content length?

Time:2019-12-5

Abstract:Understand HTTP protocol

  • Original text: after using HTTP for so long, do you know content length and transfer encoding?
  • Author: Park Ruiqing’s blog

A series of thinking caused by content length problems:
Some time ago, API gateway was developed. There was a timeout when using postman for debugging. It was determined that the request data was processed after troubleshootingContent-LengthThis article is about the problems caused by the inconsistency with the actual situation

Content-Length, HTTP message length, usingDecimal numberExpressedNumber of octetsIn general, a lot of work is done by the framework. We seldom pay attention to this part, but in a few cases, it happensContent-LengthIn case of inconsistency with the actual message length, the program may have some strange exceptions, such as:

  • No response until timeout
  • The request was truncated and the next request resolution was scrambled

Content-LengthIs the length of the HTTP message, usingDecimal numberExpressedNumber of octets, is a common field in headersContent-LengthIt should be precise, otherwise it will cause exception (especially, this field is optional in HTTP 1.0)

Content-LengthThe first part indicates the byte size of the entity body in the message. This size includes all content coding. For example, the text filegzipCompressed words,Content-LengthThe first part refers to the compressed size rather than the original size

How content length works

Content-LengthDecimal number is used to indicate the length of the message, through which the server / client can know the length of the message to be read later

After using HTTP for so long, do you know the content length?

After using HTTP for so long, do you know the content length?

If this length is not correct, the following occurs:

Content length > actual length

If the content length is larger than the actual length, the server / client will wait for the next byte after reading the end of the message, and there will be no response until the timeout

After using HTTP for so long, do you know the content length?

Again, in the response messageContent-LengthExceeding the actual length also has the same effect:

After using HTTP for so long, do you know the content length?

After using HTTP for so long, do you know the content length?

Content length < actual length

If the length is less than the actual length, the first requested message will be intercepted. For example, the parameter isparam=piaoruiqing, Content-LengthIf it is 10, the requested message will be intercepted as:param=piao, as shown in the figure:

After using HTTP for so long, do you know the content length?

After using HTTP for so long, do you know the content length?

But is it just so? Of course not. Let’s see what happens to the second request, as shown in the figure:

After using HTTP for so long, do you know the content length?

For two consecutive requests, the first message is truncated, and the second one is not truncated as expected. Instead, the server throws an exception:Request method 'ruiqingPOST' not supported. no stimulation (ノ) ゚ Д゚ ()

thatruiqingPOSTWhat kind of immortal method is it??? At this time, with the sensitivity of many years of development (debug) experience, we can roughly guess that the last request was intercepted and the rest of the message appeared in this request. Take out Wireshark to verify it, as shown in the figure:

After using HTTP for so long, do you know the content length?

The reason for this is that it’s turned onConnection:keep-alive, if usingConnection:close, the phenomenon is that every request is truncated, but there is no parsing confusion (such as splicing the last remaining message into the subsequent request message)

Not sure what to do with the content length value

Content-LengthThe first part indicates the byte size of the entity body in the message. However, if the message length cannot be obtained before the request processing is completed, we cannot specify it explicitlyContent-Length, you should useTransfer-Encoding: chunked

What is transfer encoding: chunked

Data is sent in a series of chunksContent-LengthIn this case, the header is not sent. At the beginning of each block, you need to add the length of the current block, which is expressed in hexadecimal form, followed by\r\n, followed by the block itself, followed by\r\n. the termination block is a conventional block, the difference is that its length is 0

Transfer encoding: how chunked works

Let’s use an example of downloading files to exploreTransfer-Encoding: chunkedHow does it work? The server code is as follows:

After using HTTP for so long, do you know the content length?

Use postman to initiate the request and Wireshark to view the packet, as shown in the figure:

After using HTTP for so long, do you know the content length?

The chunked data can be seen clearly in Wireshark. Its structure is roughly as follows: the returned message is divided into multiple data blocks, each of which has two parts,length + data, both of which use CRLF (i.e\r\n)The ending block is a special data block with a length of 0, as shown in the figure:

After using HTTP for so long, do you know the content length?

In this way, block coding is completed. It is mainly used in the following scenarios, that is, to transmit a large amount of data, but the length of response can not be obtained before the request is processed. For example, when it is necessary to generate a large HTML table with the data obtained from the database query, it needs to transmit a large number of pictures, etc

  • Content-LengthIf it exists and takes effect, it must be correct, otherwise an exception will occur. (if it is greater than the actual value, it will time out, if it is smaller than the actual value, it will be truncated and may cause subsequent data analysis confusion.)
  • If the message containsTransfer-Encoding: chunkedFirst, thenContent-LengthWill be ignored

Reference resources

  • https://developer.mozilla.org
  • HTTP authority Guide

Copyright notice

This article is published on Park Ruiqing’s blog. It is allowed to be reprinted for non-commercial purposes. However, the reprint must retain Park Ruiqing, the original author, and the link: https://blog.piaoruiqing.com. If there is any consultation or cooperation on authorization, please contact [email protected]

About fundebug

Fundebug focuses on real-time BUG monitoring of JavaScript, WeChat applets, WeChat games, Alipay applets, React Native, Node.js and Java online applications. Since the official launch on the double 11 of 2016, fundebug has handled 2 billion + error events in total. Paying customers include sunshine insurance, walnut programming, litchi FM, 1-on-1 leader, micro pulse, qingtuan club and many other brand enterprises. Welcome to try it for free!

Recommended Today

Sharing 10 useful methods of laravel 5.8 sets

This article is forwarded from the professional laravel developer community, original link: https://learnku.com/laravel/t… In laravel, there is a very useful class for manipulating arrays, called collections. I believe that every developer of laravel has used the collection more or less, especially when operating eloquent. In this article, I will list 10 common methods. 1. Weight […]