ASP.NET Implementation of response compression in core

Time:2021-2-12

Introduction#

Response compression technology is a common technology in the field of web development. In the case of limited bandwidth resources, using compression technology is the first choice to improve the bandwidth load. Web servers we are familiar with, such as IIS, Tomcat, nginx, Apache, etc., can use compression technology. The commonly used compression types include brotli, gzip, deflate. Their effects on CSS, JavaScript, HTML, XML, JSON and other types are obvious, but there are certain limitations. The effect on images may not be so good, because images themselves are compression formats. Secondly, for files less than about 150-1000 bytes (depending on the content of the file and the efficiency of compression), the cost of compressing small files may produce larger compressed files than uncompressed files. stay ASP.NET In the core, we can use response compression in a very simple way.

Usage#

stay ASP.NET The method of using response compression in core is relatively simple. First, add the services.AddResponseCompression Injection response compression related settings, such as compression type, compression level, compression target type, etc. Secondly, add the app.UseResponseCompression Intercept requests to determine whether compression is needed. The general usage is as follows


public class Startup
{
  public void ConfigureServices(IServiceCollection services)
  {
    services.AddResponseCompression();
  }

  public void Configure(IApplicationBuilder app, IHostingEnvironment env)
  {
    app.UseResponseCompression();
  }
}

If you need to customize some configuration, you can also manually set compression related parameters

public void ConfigureServices(IServiceCollection services)
{
  services.AddResponseCompression(options =>
  {
    //Can add a variety of compression types, the program will automatically get the best way according to the level
    options.Providers.Add<BrotliCompressionProvider>();
    options.Providers.Add<GzipCompressionProvider>();
    //Add custom compression policy
    options.Providers.Add<MyCompressionProvider>();
    //Use the compression policy for the specified mimeType
    options.MimeTypes = 
      ResponseCompressionDefaults.MimeTypes.Concat(
        new[] { "application/json" });
  });
  //Set the corresponding compression level for different compression types
  services.Configure<GzipCompressionProviderOptions>(options => 
  {
    //Using the fastest way to compress is not necessarily the best way to compress
    options.Level = CompressionLevel.Fastest;

    //No compression
    //options.Level = CompressionLevel.NoCompression;

    //Even if it takes a long time, use the best compression method
    //options.Level = CompressionLevel.Optimal;
  });
}

The general working method of response compression is to add accept in the request header when launching an HTTP request- Encoding:gzip Or any other compression type you want, you can pass multiple types. The server receives the request to obtain the accept encoding to determine whether this type of compression method is supported. If it is supported, the compressed output content is related, and the content encoding is set to the current compression method to return together. After the client gets the response, it gets the content encoding to judge whether the server adopts the compression technology, and according to the corresponding value to judge which compression type is used, and then uses the corresponding decompression algorithm to get the original data.

Research on source code#

Through the above introduction, I believe you have a certain understanding of response compression. Next, let’s understand its general working principle by looking at the source code.

AddResponseCompression#

First, let’s look at the injection related code, which is hosted in the responsecompressionservicesextension extension class[Click to view the source code]


public static class ResponseCompressionServicesExtensions
{
  public static IServiceCollection AddResponseCompression(this IServiceCollection services)
  {
    services.TryAddSingleton<IResponseCompressionProvider, ResponseCompressionProvider>();
    return services;
  }

  public static IServiceCollection AddResponseCompression(this IServiceCollection services, Action<ResponseCompressionOptions> configureOptions)
  {
    services.Configure(configureOptions);
    services.TryAddSingleton<IResponseCompressionProvider, ResponseCompressionProvider>();
    return services;
  }
}

The main thing is to inject responsecompressionprovider and responsecompressionoptions. First, let’s look at responsecompressionoptions[Click to view the source code]

public class ResponseCompressionOptions
{
  //Set the type of compression required
  public IEnumerable<string> MimeTypes { get; set; }

  //Set types that do not require compression
  public IEnumerable<string> ExcludedMimeTypes { get; set; }

  //Do you want to enable HTTPS support
  public bool EnableForHttps { get; set; } = false;

  //Compact type collection
  public CompressionProviderCollection Providers { get; } = new CompressionProviderCollection();
}

I won’t introduce this class too much. It’s relatively simple. Responsecompressionprovider is the core class that we provide the response compression algorithm. How to select the compression algorithm automatically is provided by it. There are many codes in this class, so we won’t explain them one by one. The specific source code can be consulted by ourselves[Click to view the source code]First, let’s look at the constructor of responsecompressionprovider

public ResponseCompressionProvider(IServiceProvider services, IOptions<ResponseCompressionOptions> options)
{
  var responseCompressionOptions = options.Value;
  _providers = responseCompressionOptions.Providers.ToArray();
  //If no compression type is set, Br and gzip compression algorithms are adopted by default
  if (_providers.Length == 0)
  {
    _providers = new ICompressionProvider[]
    {
      new CompressionProviderFactory(typeof(BrotliCompressionProvider)),
      new CompressionProviderFactory(typeof(GzipCompressionProvider)),
    };
  }
  //Create the corresponding compression algorithm provider according to the compression providerfactory, such as gzip compression provider
  for (var i = 0; i < _providers.Length; i++)
  {
    var factory = _providers[i] as CompressionProviderFactory;
    if (factory != null)
    {
      _providers[i] = factory.CreateInstance(services);
    }
  }
  //The default compression target types are set as text / plain, text / CSS, text / HTML, application / JavaScript and application / XML
  //text/xml、application/json、text/json、application/was
  var mimeTypes = responseCompressionOptions.MimeTypes;
  if (mimeTypes == null || !mimeTypes.Any())
  {
    mimeTypes = ResponseCompressionDefaults.MimeTypes;
  }
  //Put the default mimeType into the HashSet
  _mimeTypes = new HashSet<string>(mimeTypes, StringComparer.OrdinalIgnoreCase);
  _excludedMimeTypes = new HashSet<string>(
    responseCompressionOptions.ExcludedMimeTypes ?? Enumerable.Empty<string>(),
    StringComparer.OrdinalIgnoreCase
  );
  _enableForHttps = responseCompressionOptions.EnableForHttps;
}

Among them, brotli compression provider and gzip compression provider provide compression methods. Let’s see the general implementation of gzip providers[Click to view the source code]

public class GzipCompressionProvider : ICompressionProvider
{
  public GzipCompressionProvider(IOptions<GzipCompressionProviderOptions> options)
  {
    Options = options.Value;
  }

  private GzipCompressionProviderOptions Options { get; }

  //Corresponding encoding name
  public string EncodingName { get; } = "gzip";

  public bool SupportsFlush => true;

  //The core code is to convert the original output stream into a compressed gzipstream
  //The compression level we set will determine the performance and quality of compression
  public Stream CreateStream(Stream outputStream)
    => new GZipStream(outputStream, Options.Level, leaveOpen: true);
}

As for other related methods of responsecompression provider, when we explain the useresponsecompression middleware, we will see the specific methods used, because this class is the core class of response compression. Now I have said it in advance, I may forget where the middleware is used. Next, let’s look at the general implementation of useresponsecompression.

UseResponseCompression#

Useresponsecompression is a parameterless extension method, which is also relatively simple. Because the configuration and work are completed by the injection place, we can directly check the implementation in the middleware and find the middleware location responsecompression middleware[Click to view the source code]

public class ResponseCompressionMiddleware
{
  private readonly RequestDelegate _next;
  private readonly IResponseCompressionProvider _provider;

  public ResponseCompressionMiddleware(RequestDelegate next, IResponseCompressionProvider provider)
  {
    _next = next;
    _provider = provider;
  }

  public async Task Invoke(HttpContext context)
  {
    //Judge whether it contains the accept encoding header information, not including a direct shout "carry the next"
    if (!_provider.CheckRequestAcceptsCompression(context))
    {
      await _next(context);
      return;
    }
    //Get the original output body
    var originalBodyFeature = context.Features.Get<IHttpResponseBodyFeature>();
    var originalCompressionFeature = context.Features.Get<IHttpsCompressionFeature>();
    //Initialize response compression body
    var compressionBody = new ResponseCompressionBody(context, _provider, originalBodyFeature);
    //Set to compression body
    context.Features.Set<IHttpResponseBodyFeature>(compressionBody);
    context.Features.Set<IHttpsCompressionFeature>(compressionBody);

    try
    {
      await _next(context);
      await compressionBody.FinishCompressionAsync();
    }
    finally
    {
      //Restore the original body
      context.Features.Set(originalBodyFeature);
      context.Features.Set(originalCompressionFeature);
    }
  }
}

This middleware is very simple. It initializes the response compression body. You may be curious to see that there is no code related to calling compression. Responsecompressionbody only calls finishcompressionaasync, which is related to release. Don’t worry. Let’s look at the structure of responsecompressionbody class


internal class ResponseCompressionBody : Stream, IHttpResponseBodyFeature, IHttpsCompressionFeature
{
}

This class implements the ihttppresponsebodyfeature, which we use Response.Body In fact, it is the htt obtained pResponseBodyFeature.Stream Property. We use Response.WriteAsync In fact, the related methods call pipewriter internally to write, and pipewriter is from htt pResponseBodyFeature.Writer Property. Generally speaking, the core of output related operations is to operate ihttppresponsebody feature. Interested can check the httpresponse related source code, you can understand the relevant information. So our responsecompressionbody actually rewrites methods related to output operations. In other words, as long as you call response related write or body related, you are actually operating the ihttppresponsebodyfeature. Since we have enabled the middleware related to response output, we will call the method related to the implementation class responsecompressionbody of ihttppresponsebodyfeature to complete the output. In general, we think that we only need to operate on the output stream, but the response compression middleware rewrites the output related operations.

After understanding this, I believe you will not have too many questions. Because the responsecompressionbody rewrites the operation related to the output, there are relatively many codes, so we don’t paste them one by one. We only look at the code related to the design of the response compression core. For details about the source code of the responsecompressionbody, you can refer to them if you are interested[Click to view the source code]The essence of output is actually calling the write method. Let’s take a look at the implementation of the write method

public override void Write(byte[] buffer, int offset, int count)
{
  //This is the core method. The compression related output is here
  OnWrite();
  //_ Compressionstream is initialized in the onwrite method
  if (_compressionStream != null)
  {
    _compressionStream.Write(buffer, offset, count);
    if (_autoFlush)
    {
      _compressionStream.Flush();
    }
  }
  else
  {
    _innerStream.Write(buffer, offset, count);
  }
}

From the above code, we can see that the onwrite method is the core operation. Let’s directly look at the implementation of the onwrite method

private void OnWrite()
{
  if (!_compressionChecked)
  {
    _compressionChecked = true;
    //Determine whether the logic related to compression is satisfied
    if (_provider.ShouldCompressResponse(_context))
    {
      //Match the value corresponding to the vary header information
      var varyValues = _context.Response.Headers.GetCommaSeparatedValues(HeaderNames.Vary);
      var varyByAcceptEncoding = false;
      //Judge whether the value of variety is accept encoding
      for (var i = 0; i < varyValues.Length; i++)
      {
        if (string.Equals(varyValues[i], HeaderNames.AcceptEncoding, StringComparison.OrdinalIgnoreCase))
        {
          varyByAcceptEncoding = true;
          break;
        }
      }
      if (!varyByAcceptEncoding)
      {
        _context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);
      }
      //Get the best icompressionprovider, that is, the best compression method
      var compressionProvider = ResolveCompressionProvider();
      if (compressionProvider != null)
      {
        //Set the selected compression algorithm and put it into the value of content encoding header
        //The client can judge which compression algorithm the server adopts through the content encoding header information
        _context.Response.Headers.Append(HeaderNames.ContentEncoding, compressionProvider.EncodingName);
        //When compressing, content-md5 removes the header because the body content has changed and the hash is no longer valid.
        _context.Response.Headers.Remove(HeaderNames.ContentMD5); 
        //When compressing, the content length is removed from the header because the body content changes when the response is compressed.
        _context.Response.Headers.Remove(HeaderNames.ContentLength);
        //Returns the compressed related output stream
        _compressionStream = compressionProvider.CreateStream(_innerStream);
      }
    }
  }
}

private ICompressionProvider ResolveCompressionProvider()
{
  if (!_providerCreated)
  {
    _providerCreated = true;
    //Call the method of responsecompressionprovider to return the most suitable compression algorithm
    _compressionProvider = _provider.GetCompressionProvider(_context);
  }
  return _compressionProvider;
}

From the above logic, we can see that before executing compression related logic, we need to judge whether the compression related method shouldcompressresponse is satisfied. This method is a method in responsecompressionprovider, so there is no need to paste code here. It is the judgment logic originally. I sort it out directly, which is about the following situations

  • If the request is in the case of HTTP, whether the setting of compression in the case of HTTP has been set, that is, the property setting of enable for HTTP in responsecompression options
  • Response.Head Cannot contain content range header information in
  • Response.Head Content encoding header information cannot be included before
  • Response.Head Content type header information must be included before
  • The returned mimeType cannot contain the configured types that do not need to be compressed, that is, the excludedmimetypes of responsecompression options
  • The returned mimeType should contain the configured types to be compressed, that is, mimetypes of responsecompression options
  • If the above two conditions are not met, the returned mimeType contains * / * and response compression can also be performed

Next, let’s look at the getcompressionprovider method of responsecompressionprovider to see how it determines which compression type to return

public virtual ICompressionProvider GetCompressionProvider(HttpContext context)
{
  var accept = context.Request.Headers[HeaderNames.AcceptEncoding];
  //Determine whether the request header contains accept encoding
  if (StringValues.IsNullOrEmpty(accept))
  {
    Debug.Assert(false, "Duplicate check failed.");
    return null;
  }
  //Get the value in accept encoding, judge whether it contains gzip, Br, identity, etc., and return the matching information
  if (!StringWithQualityHeaderValue.TryParseList(accept, out var encodings) || !encodings.Any())
  {
    return null;
  }
  //The matching priority is calculated according to the request information and setting information
  var candidates = new HashSet<ProviderCandidate>();
  foreach (var encoding in encodings)
  {
    var encodingName = encoding.Value;
    //Quality involves a very complex algorithm. You can refer to it if you are interested
    var quality = encoding.Quality.GetValueOrDefault(1);
    //Quality must be greater than 0
    if (quality < double.Epsilon)
    {
      continue;
    }
    //Match the encoding name in the request header with the encoding name in the providers compression algorithm
    //It can be seen from this that the priority of matching has something to do with the order of registering providers
    for (int i = 0; i < _providers.Length; i++)
    {
      var provider = _providers[i];
      if (StringSegment.Equals(provider.EncodingName, encodingName, StringComparison.OrdinalIgnoreCase))
      {
        candidates.Add(new ProviderCandidate(provider.EncodingName, quality, i, provider));
      }
    }
    //If the encoding name in the request header is *, the matching is performed in all registered providers
    if (StringSegment.Equals("*", encodingName, StringComparison.Ordinal))
    {
      for (int i = 0; i < _providers.Length; i++)
      {
        var provider = _providers[i];
        candidates.Add(new ProviderCandidate(provider.EncodingName, quality, i, provider));
      }
      break;
    }
    //If the encoding name in the request header is identity, the response is not encoded
    if (StringSegment.Equals("identity", encodingName, StringComparison.OrdinalIgnoreCase))
    {
      candidates.Add(new ProviderCandidate(encodingName.Value, quality, priority: int.MaxValue, provider: null));
    }
  }

  ICompressionProvider selectedProvider = null;
  //If there is only one match, return it directly
  if (candidates.Count <= 1)
  {
    selectedProvider = candidates.FirstOrDefault().Provider;
  }
  else
  {
    //If there are multiple matches, the first one will be matched according to the reverse quality order and the positive priority order
    selectedProvider = candidates
      .OrderByDescending(x => x.Quality)
      .ThenBy(x => x.Priority)
      .First().Provider;
  }
  //If there is no match to selectedprovider or identity, null will be returned directly
  if (selectedProvider == null)
  {
    return null;
  }
  return selectedProvider;
}

Through the above introduction, we can roughly understand the general working mode of response compression, and briefly summarize it

  • First, set the compression related algorithm type or the mimeType of the compression target
  • Secondly, we can set the compression level, which will determine the quality and performance of compression
  • Through the response compression middleware, we can obtain a compression algorithm with the highest priority for compression, which is mainly for the case of multiple compression types. This compression algorithm has a certain relationship with the internal mechanism and the order of registration compression algorithm, and will ultimately choose the return with the largest weight.
  • The core work class of response compression middleware, responsecompressionbody, implements the ihttppresponsebody feature and rewrites the output related methods to compress the response. Instead of calling the related methods manually, we replace the default output method. As long as the response compression is set and the request satisfies the response compression, the compression related methods in the responsecompression body are executed by default where the output is called, instead of intercepting the specific output for unified processing. As for why, I don’t understand what the designer really thinks.

Summary#

Before looking at the relevant code, I thought that the logic related to response compression would be very simple. After reading the source code, I knew that I thought it was too simple. One of the biggest differences with my own ideas is that in the response compression middleware, I thought that I would compress the output stream by intercepting it uniformly, but I didn’t expect that I would rewrite the overall output operation. Because before we used Asp.Net When related frameworks are written, filter or HttpModule is used for processing, so there is a thinking pattern. It could be Asp.Net Core designers have a deeper understanding. Maybe I don’t understand it thoroughly enough. I can’t understand the benefits of doing so. If you have a better understanding or answer, please leave a message in the comments area.

This is about ASP.NET This is the article about the implementation of response compression in core, and more about it ASP.NET Core response compressed content, please search previous articles of developer or continue to browse the following related articles. I hope you can support developer more in the future!