A network request process under Browser

Time:2020-10-23

A network request process under Browser

To input http://github.com/appleguardu… Web request is essentially a host to another host to obtain the corresponding resources

1. Process switching

When the navigation bar enters http://github.com/appleguardu… The browser process gives the URL to the network process to process and give me to go to this host to get resources;

2. Resolve the requested URL, DNS query IP, and prepare port

  • URL: Uniform Resource Locator
Protocol: protocol header, such as HTTP, FTP, etc
Host: host domain name or IP address
Port: port number
Path: directory path
Query: query parameter
Fragment: the hash value after ා, which is generally used to locate a certain position
  • DNS (mapping system between domain name and IP)
Because different hosts are determined by IP address, domain name resolution and query will be carried out first; 
The browser will query the domain name server http://github.com Corresponding IP

//IP query also has a set of optimization mechanism: caching mechanism (can reduce network requests)
Browser cache: the browser will cache DNS records at a certain frequency;
Operating system cache: if the required DNS records cannot be found in the browser cache, go to the operating system;
Route cache: router also has DNS cache;
ISP (ISP) has DNS server for Internet inquiry;
Root server: if the DNS server of ISP cannot be found, it will send a request to the root server for recursive query;
(the DNS server first asks the IP address of the root domain name server. COM domain name server, then the. GitHub domain name server, and so on.)
  • Port
If not specified, the default port of HTTP protocol is port 80, which is HTTPS: 443

3. Wait for TCP queue (browser’s restrictions on TCP)

There is a mechanism for choose browser. Under the unified domain name, only six TCP connections can be established at the same time (the restrictions of the browser may be different). If there are more than six requests, the rest will be queued and wait for the previous one to complete before establishing the connection

4. Establish TCP connection

TCP protocol

TCP operation: connection creation, data transmission and connection termination

  • Connection creation: three handshakes (guarantee reliable transmission)
1. The client creates an active open by sending a syn to the server as part of the three handshakes;
The client sets the serial number of this connection as a random number.

2. The server should send back a syn / ack for a legitimate syn.
The ACK confirmation code should be a + 1, and syn / ACK packet itself has a random number B.

3. Finally, the client sends another ack. At this time, the serial number of the packet is set to a + 1, and the ACK's confirmation code is B + 1.
When the server receives the ACK, it completes three handshakes and enters the connection creation state.

Client side: syn ACK
Server side: syn + ACK
  • Data transmission (some mechanisms)
The serial number is used to sort the received TCP message segments and detect duplicate data;
Check sum is used to detect the error of message segment, that is, error free transmission;
Use acknowledgement and timer to detect and correct packet loss or delay;
Flow control; congestion control; retransmission of lost packets.
  • End of connection (four waves)
Connection termination uses a four-way handshake process, in which each side of the connection is terminated independently.
When an endpoint wants to stop its connection on this side, it sends fin to the opposite side, and the opposite side replies ack to confirm.
Therefore, the process of removing one side of the connection requires a pair of fin and ACK, which are sent out by the endpoints on both sides.

Client side: fin ACK
Server side: ack fin

5. Send HTTP request

When the TCP connection is established, the browser and the server start to communicate and transmit data

Build request message

  • Request line
GET /appleguardu HTTP/1.1
//Request method: (get, post, options, put, head, delete, connect, trace
//Resource path URI: / applegardu
//Protocol version: http 1.1
  • Request header (common)
Accept: receive type, indicating the MIME type supported by the browser (content type returned by the benchmarking server)
Accept encoding: the compression type supported by the browser, such as gzip, cannot be received if it exceeds the type
Content type: the type of entity content sent out by the client
Cache control: Specifies the caching mechanism for requests and responses, such as no cache
If modified since: corresponds to the last modified since of the server. It is used to match and see whether the file changes. It can only be accurate to within 1 s, in http1.0
Expires: cache control, no request will be made during this time. The cache is used directly, http1.0, and the server time
Max age: represents the number of seconds that the resource is cached locally. It will not be requested within the valid time, but will be used in HTTP1.1
If none match: corresponds to the Etag of the server, which is used to match whether the content of the file has changed (very accurate), in HTTP1.1
Cookie: when there is a cookie, it will be brought automatically when accessing the same domain
Connection: how to handle long connections when the browser communicates with the server, such as keep alive
Host: requested server URL
Origin: where is the initial request initiated (only accurate to the port), origin respects privacy more than referer
Referer: the source URL of the page (applicable to all types of requests, and will be accurate to the detailed page address. This field is commonly used in CSRF interception)
User agent: some necessary information of user client, such as UA header
  • Request body
The parameters in post request are usually placed in the body of the request
Such as the serialization form of parameters (a = 1 & B = 2;
Or directly put the form object (form data object, which can be mixed with parameters and files, etc.) when uploading

Note: if a cache resource is requested, theTermination request

The browser will check the requested file resources. If the requested resources exist in the browser cache, the request will be intercepted. The browser will directly get the request from the local cache and send it to the network process, and end the request process here
About HTTP caching

Cache classification

Cache can be divided into two types: strong cache (200 from cache) and negotiation cache (304 returned by server);
Strong cache: if the browser judges that the local cache is not expired, it can be used directly without sending an HTTP request;
Negotiation cache 304:When browseragainSend an HTTP request to the server, and then the server (304) tells the browser that the file has not changed, so that the browser can use the local cache

  • How to distinguish the two types of cache? By setting the HTTP header information
#Several header cache properties
Strong cache (HTTP1.1): cache control / max age 
0.0 / expectma: strong cache 

Negotiation cache (HTTP1.1): if none match / e-tag
Negotiation cache (http1.0): if modified since / last modified

<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
There is also a meta tag in HTML pages to control the caching scheme pragma, but it is usually set on the server side
  • Cache control under http1.0
Pragma: strictly speaking, it does not belong to the special cache control header, but it can disable the local strong cache when setting no cache (it belongs to compile control to implement specific instructions, mainly because it is compatible with http1.0, so it has been widely used before)

Expires: the server-side configuration belongs to the strong cache, which is used to control that the browser will not issue requests before the specified time, but will directly use the local cache,
Note: expires generally corresponds to the server-side time, such as: Fri, 30 OCT, 1998, 14:19:41
If the client time is not synchronized with the server, the local cache of the browser may be useless or cannot be expired

If modified since / last modified: these two appear in pairs and belong to the negotiation cache content;
The browser head is if modified since, while the server side is last modified;
Its function is that if modified since and last modified match when a request is initiated,
Therefore, the server will not return the resource entity, but only the header to inform the browser that the local cache can be used
Last modified, as the name implies, refers to the last modification time of the file, and can only be accurate to less than 1s
Note: if the server files will change periodically, the cache will be invalid
  • Cache control under HTTP1.1
There are many cache values, such as control cache, no cache and so on

Max age: it is configured on the server side to control the strong cache. Within a specified period of time, the browser does not need to send a request and directly uses the local cache,
Note that Max age is the value of the cache control header, not a separate header,
For example, cache control: Max age = 3600, and it is worth absolute time, which is calculated by the browser itself

If none match / e-tag: these two appear in pairs and belong to the negotiation cache content;
The head of the browser is if none match, while the server side is e-tag,
Similarly, if the if none match matches e-tag after the request is sent, the browser is informed to use the local cache;
Unlike last modified, e-tag is more accurate. It is similar to fingerprint, so it has higher priority
Based on fileetag inode mtime size generation, that is, as long as the file changes, the fingerprint will change, and there is no limit of 1s accuracy.

6. The server processes HTTP requests

When the server receives the request message from the browser, it will be handed over to the corresponding processing program for processing (omit… Here), and then the server will return the response message

response message

  • Response line
HTTP/1.1 200 OK
//Protocol version status code status code information
//200 -- indicates that the request was successfully completed and the requested resource is sent back to the client
//304 -- since the last request, the requested web page has not been modified. Please use the local cache
//400 -- error in client request (for example, it can be intercepted by security module)
//401 - unauthorized request
//403 -- prohibit access (for example, when you are not logged in)
//404 - resource not found
//500 - server internal error
//503 - service not available
  • Response header (common)
Access control allow headers: server side allowed headers
Access control allow methods: server side allowed request methods
Access control allow origin: the request origin header allowed on the server side (for example, *)
Content type: the type of entity content returned by the server
Date: the time when the data was sent from the server
Cache control: tells browsers or other customers what environment can cache documents safely
Last modified: the last modification time of the requested resource
Expires: when should a document be considered out of date so that it is no longer cached
Max age: how many seconds should the client's local resources be cached, which will be effective when cache control is enabled
Etag: the current value of the entity tag of the request variable
Set Cookie: sets the cookie associated with the page. The server passes the cookie to the client through this header
Keep alive: if the client has keep alive, the server will respond (for example, timeout = 38)
Server: some information about the server
  • Response body
The response body is generally the content that the server needs to pass to the client;
For example, when the interface is requested, the entity is the JSON format object of the corresponding information
Such as page request: the corresponding HTML string is in the entity

be careful

In general, the request header and the response header match each other
If the accept of the request header should match the content type of the response header, otherwise an error will be reported;
In cross domain request, the origin of the request header should match the access control allow origin of the response header, otherwise a cross domain error will be reported
And when using caching,
If modified since and if none match of the request header are
The last modified and Etag of the response header correspond to each other

The above is basically through the server-side program to control

redirect

When the response status code is 301, redirection will occur. The browser process will let the navigation work again, modify the address in the location field, and continue the above request process

8. The browser parses and renders the page

Browser process starts scheduling, rendering process receives HTML data of network process through pipeline, and starts to parse HTML

Rendering process workflow

  • 1. DOM tree construction
When the rendering process receives the confirmation information of navigation and starts to accept HTML data, the main thread will parse the text string as dom;
Here we rely on the HTML parser: 
Accept byte stream > maintain token stack > generate node node > form dom;

When embedded script is encountered, DOM parsing stops; JS engine intervenes (DOM structure may be modified);
After JS is executed, the parsing will resume, so JS will block DOM parsing

When encountering other inline resources (CSS, IMG), it will inform the network process to download, especially CSS;
JS will rely on cssom when operating DOM style, and cssom is also needed to generate layouttree; 
So CSS will block the execution of JS
  • 2. Style calculation to construct cssom (CSS rule tree)
Here, CSS will be parsed based on CSS selector to obtain the final calculation style value of each node;
The corresponding is stylesheets
  • 3. Calculate the layout and generate the layout tree
If you want to render a complete page, you need to know not only the specific style of each node, but also the location of each node on the page,
Layout is actually the process of finding the geometric relationships of all elements.

Here, by traversing the calculation style of DOM and related elements, the main thread will build a layout tree containing the coordinate information and box size of each element.
The layout tree is similar to the DOM tree, but it only contains elements visible to the page, if an element is set to` display:none `
This element does not appear in the layout tree. Although the pseudo element is not visible in the DOM tree, it is visible in the layout tree.
  • 4. Layer, paint
Generate a special layer (will change attribute) for a specific node, and generate a layer tree;
The drawing table is generated for the layer (the drawing instruction and order are recorded) and submitted to the composition thread
  • 5. Block and rasterize
The composition thread divides the layer into blocks and generates bitmap by rasterization (GPU process)
  • 6. Synthesis, display
After the block is rasterized, a drawing command is generated and submitted to the browser process for execution through IPC,
Draw it into memory and display it on the monitor

9. Disconnect TCP connection

When the data transmission is completed, the TCP connection needs to be disconnected, and the TCP wave is initiated four times;
Usually, once the server returns the request data to the client, it will close the TCP connection.
However, if the browser or server adds

Connection: keep alive (enabled by default under http / 1.1, closed by default in http / 1.0)
//Keep alive will not be maintained forever. It has a duration, which is generally configured in the server (for example, Apache,
In addition, long connection is valid only when both client and server support it

The TCP connection remains open after it is sent so that the browser can continue to send requests over the same TCP connection.
Keeping TCP connection can save the time needed to establish a connection for the next request and improve the resource loading speed. For example, the images embedded in a web page are all from the same web site. If a persistent connection is initialized, you can reuse the connection to request other resources, without having to re-establish a new TCP connection.

reference resources

Encyclopedia of TCP protocol
HTTP headers

  • Understanding http2.0
-Multiplexing (i.e. a TCP / IP connection can request multiple resources)
-Head compression (HTTP header compression, reduced volume)
-Binary framing (a binary framing layer is added between the application layer and the transport layer to improve the transmission performance and achieve low latency and high throughput)
-Server side push (the server can send multiple responses to a request from the client, and can actively notify the client)
-Request priority (if the flow is given a priority, it will process it based on this priority, and the server determines how much resources are needed to process the request.)