Browser caching mechanism and Implementation


What is caching

Caching is a technology that saves a copy of a resource and uses it directly on the next request( MDN)

The web browser cache here mainly refers to HTTP cache.

The role of caching

  • Reduce network delay and speed up page opening
  • Reduce network bandwidth consumption
  • Reduce server stress

Cache classification

By permission

  • Private cache, private cache can only be used for individual users
  • Shared cache. The response stored in shared cache can be used by multiple users. For example, ISP or your company may set up a web proxy to provide it to users as part of the local network infrastructure. In this way, popular resources will be reused to reduce network congestion and delay.

By storage location

Browser local cache

From memory cache

Memory cache has two characteristics: fast read and timeliness: fast read: memory cache will directly store the compiled and parsed files into the memory of the process, occupying certain memory resources of the process, so as to facilitate the fast read of the next run. Timeliness: once the process is shut down, the memory of the process will be cleared.

From disk cache

The hard disk cache is to write the cache to the hard disk file directly. To read the cache, I / O operation should be performed on the hard disk file stored in the cache, and then the cache content should be re parsed. The reading speed is slower than that of the memory cache.

Take the description displayed in the browser as the column:

Browser caching mechanism and Implementation

  • From memory cache means to use the cache in memory
  • From disk cache means that the cache in the hard disk is used
  • From prefetch cache: when the resource of preload or prefetch is loaded, both are stored in the HTTP cache. After the resource is loaded, if the resource can be cached, it will be stored in the HTTP cache for subsequent use; If resources are not cacheable, they are stored in the memory cache before they are used.

Browser resource cache access priority

When the browser requests a resource, it will find the cache in order of priority (service worker, memory cache, disk cache, push cache). If it hits, it will use the cache, otherwise it will initiate the request.

Take the service access without service worker as an example:–>200 – > close tab of page – > reopen –>200 (from disk cache) – > refresh – > 200 (from memory cache).

Proxy / server caching

Gateway cache, CDN, reverse proxy cache, load balancer and other caches deployed on the server

Take Tencent as an example   CDN as an example: x-cache-l in request header ookup:Hit From Memcache represents the memory that hits the CDN node; X-Cache-L ookup:Hit From Disktank represents the disk that hits the CDN node; X-Cache-L ookup:Hit From Upstream indicates that the CDN has not been hit.

Browser resource access process

Browser caching mechanism and Implementation

The key steps are as follows:

  1. Before sending a request, the browser will go to the cache to check whether it hits the strong cache. If it hits, it will read the resource directly from the cache and will not send the request to the server. Otherwise, go to the next step.
  2. When the strong cache fails to hit, the browser makes a request to the server.
  3. The server will determine whether to hit the negotiation cache according to some fields in the request header. If it hits, the server will return 304 response, but will not carry any response entity. It just tells the browser that it can get this resource directly from the browser cache.
  4. If both the local cache and negotiation cache fail to hit, the resource will be loaded directly from the server, and the server will put the cache rules into the HTTP header of the HTTP response message and return the request result to the browser together

HTTP header cache related fields

Strong cache (local cache) control expires & cache control

The fields that control the mandatory cache are expires and expires in the request header, where cache control has higher priority than expires.

Expires + date

Expires is the response header field of HTTP / 1.0 control web page cache, and its value is the expiration time of the request result cache returned by the server. That is, when the request is initiated again, if the client’s time is less than the value of expires, the cache result will be used directly.

Expires is a field of HTTP / 1.0, but now the browser uses http / 1.1 by default. Is web page caching controlled by expires in http / 1.1?

When it comes to http / 1.1, expiration has been replaced by cache control, because the principle of cache control is to use the time of the client to compare with the time returned by the server. If the time of the client and the server is different for some reasons (for example, the time zone is different; If there is an error between the client and the server (the time is not accurate), then the forced cache will be directly invalid, so the existence of the forced cache is meaningless.

Cache control (both request header and response header support this property)

Defined by HTTP / 1.1   Cache-Control   Header is used to distinguish the support for caching mechanism. Both request header and response header support this attribute. The cache policy is defined by the different values it provides.

  • Cache-Control: no-store   // The cache must not store any content about client request and server response. Each request initiated by the client will download the complete response content.
  • Cache control: no cache / / the cache is required to submit the request to the original server for verification (negotiation cache verification). Every time a request is sent, the cache will send the request to the server (the request should have a verification field related to the local cache). The server will verify whether the cache described in the request has expired. If it has not expired (Note: actually 304 is returned), the cache will use the local cache copy.
  • Cache-Control: private //  ” “Private” means that the response is dedicated to a single user. The man in the middle cannot cache the response. The response can only be applied to the browser private cache.
  • Cache-Control: public   //” The “public” instruction indicates that the response can be cached by any middleman. If “public” is specified, some pages that are not usually cached by middleman (such as pages with HTTP authentication information (account password) or some specific status codes) will be cached.
  • Cache-Control: max-age=31536000    // Represents the maximum time that a resource can be cached (kept fresh). Relative to expires, Max age is the number of seconds from the time the request was initiated. For those files that will not change in the application, you can usually manually set a certain length of time to ensure that the cache is effective, such as images, CSS, JS and other static resources. Use with age
  • Cache-Control: must-revalidate     // When considering using an obsolete resource, the cache must first verify its state, and the expired cache will not be used

Pragma (request header)

Pragma   It is a header attribute defined in http / 1.0 standard. The effect of containing pragma in the request is the same as that of defining cache control: no cache in the header information. However, this attribute is not explicitly defined in the HTTP response header, so it cannot be used to completely replace the cache control header defined in http / 1.1. Pragma is usually defined to be backward compatible with HTTP / 1.0-based clients.

Negotiation cache control

The identity of the negotiation cache is also returned to the browser together with the request result in the HTTP header of the response message. The fields controlling the negotiation cache are: last modified / if modified since and Etag / if non match. Etag / if non match has higher priority than last modified / if modified since.

Last modified (response header) and if modified since (request header)

Belongs to HTTP 1.0. When the if modified since header is used to access the resource requested by the server, the server will check the last modified. If the last modified time is earlier than or equal to if modified since, a 304 response without the principal will be returned. Otherwise, the resource will be returned again.

Last modified can only be accurate to one second and can be used as a weak verifier.

Etag (response header) and if none match (request header)

Belongs to HTTP 1.1. Etag is a response header field and a strong verifier. It is a hash string generated according to the content of the entity, which identifies the status of the resource and is generated by the server. If none match is a conditional request header. If this field is added to the request header when a resource is requested, and the value is the Etag of the resource returned by the server before, then the server will return a 200 response with the requested resource entity only if and only if there is no Etag attribute value of any resource on the server that is listed in the header, otherwise the server will return a 304 response without the requested resource entity.

ETag  VS  Last-Modified 

The final modification of last modified annotation can only be accurate to seconds. If some files are modified several times within one second, it will not be able to accurately mark the freshness of files;
Some files may be changed periodically, but their contents do not change (only the modification time is changed), but last modified has changed, which makes the files unable to use the cache;
It is possible that the server does not get the file modification time accurately, or the time is inconsistent with the proxy server.
Etag has higher priority than last modified, and Etag will prevail when it exists at the same time.

Vary  ( Response header)

It is an HTTP response header information, which determines whether to use a cached response or request a new response from the source server for a future request header. It is used by the server to indicate that the   content negotiation   Which headers should be used when selecting a resource representative in content negotiation algorithm. It means that a response is different from a response header.

such as   Vary: Accept   That is to say, the response is different because of the request resource format header, then the resource accessed through the same URI can know its content format is different according to the header.

The same URL can provide multiple different documents, which requires a mechanism between the server and the client to select the most appropriate version, which is called content negotiation. The server automatically sends the most appropriate version according to some fields in the request header sent by the client. There are two types of request header fields that can be used for this mechanism: the accept field and other fields.

For example, accept encoding is a special field for content negotiation, and the server only needs to add a content encoding field in the response header to indicate the content compression format; Or do not output content encoding, indicating that the content is not compressed. The cache server caches different contents for different content encoding, and then returns the most appropriate version according to the accept encoding field in the specific request. The variable: accept encoding response header is added to explicitly inform the cache server to cache different versions according to the content of the accept encoding field;

Different resource caching strategies

  • Different resources may have different update requirements. Review and determine the appropriate Max age for each resource;
  • Some resources are updated more frequently than others. If specific parts of a resource, such as JS functions or a set of CSS styles, are updated frequently, consider providing its code as a separate file. In this way, the remaining content (such as library code that will not be updated frequently) can be obtained from the cache every time the update is obtained, so as to ensure the minimum download capacity;
  • The combination of resource URL with content characteristic code and short time or no cache life cycle for HTML documents can control the update speed of the client. The low-frequency update resource (JS / CSS) changes, and only the entry changes in the high-frequency change resource file (HTML).

Special instructions

F5 / click refresh button in the toolbar / right click menu to reload

The function of F5 is different from that of typing directly in the URI input field and then pressing enter. F5 will make the browser send an HTTP request to the server in any case, even if the previous response has an expires header. So, when I’m here   When you press F5 in the web page, the browser will send an HTTP request to the server, but it contains such headers:

Cache-Control: max-age=0
If-Modified-Since: Fri, 15 Jul 2016 04:11:51 GMT

Among them, cache control is imposed by chrome, and if modified since the last modified header is included when obtaining the resource, the browser will use the if modified since header information to resend the time to confirm whether the resource needs to be resend. In fact, the server did not modify the index.css file, so it returned a 304 (not modified). The response information is very small, the route trip consumed is not much, and the web page is refreshed quickly.
Browser caching mechanism and Implementation

There is no Etag in the above example. If the response contains Etag, the HTTP request triggered by F5 will also contain if none match.

Implementation of cache rules

The client browser cache described above means that the storage location is in the client browser, but the actual setting of the client browser cache is completed in the resources on the server. Although the above describes the properties of the client browser cache, the actual setting of these properties needs to be done in the resources on the server   There are two ways to set the browser cache, one is through instruction declaration, the other is through programming

Ngnix instruction setting

Example 1
Configure the command expires to control the values of expires and cache control in the HTTP response. The default value is off

location ~ .*.(js|css)?$ {    
    expires 1y; }

Example 2:
For all requests with suffix. HTML, the return header cache control uses the no cache instruction

location ~ .*\.(html)$ {
         try_files $uri $uri/ =404;
         root /data/web/default;
         add_header Cache-Control no-cache;

Example 3:
Etag settings

http {
        etag off;

Example 4

Configure last modified (on by default)

Programming mode

We won’t describe it in detail here. For server services made in different languages, please see the description of relevant modules
Take the implementation of KOA as an example

var koa = require('koa');
var app = new koa();
// response
app.use(function *(){
  this.body = 'Hello World';
  var etag = this.get('ETag');
  var date = new Date;
  var hashStr = this.body;
  var hash = require("crypto").createHash('sha1').update(hashStr).digest('base64');
    'Etag': hash,
    'Last-Modified': new Date

Reference documents

Mdn-http cache:… ;
HTTP cache control:… ; ;… ;… ;