. Net core httpclient processing response compression details

preface:

In previous articlesASP. Implementation of response compression in net coreAs mentioned, the main work of the server is based onContent-EncodingThe header information determines which method to compress and return. Someone in the group asked before, is it still necessary to compress requests on the server when the network bandwidth is so high? Indeed, today’s distributed and load balancing technologies are so mature that many scenarios that need to handle high concurrency big data can be implemented by adding server nodes. However, when resources are limited or it is not necessary to add new server nodes for a certain point, we still need to use some routine processing methods of the program itself. The author personally believes that the use scenario of response compression is as follows. In the case of tight bandwidth pressure and sufficient CPU resources, the overall effect of response compression is relatively obvious.
If there is compression, there is decompression, and the decompression is processed at the request client. For example, browsers, which are the most commonly used HTTP clients, are used by many browsers by default when we send a request (such as when we browse a web page)Request HeadAdd inContent-EncodingAnd then process the relevant decompression according to the response information. All of these come from the fact that the browser has built-in mechanisms for request compression and decompression. There are many similar tools, such as the commonly used proxy packet capturing toolsFilderThis mechanism is also built in. It just needs to be handled manually, but the implementation method is the same. Sometimes we need to use this mechanism in the process of writing our own programs.Net HttpWebRequestThere is no such mechanism in the class library, and it was added in later versionsHttpClientThere is a built-in mechanism to handle this operation,

1、 Usage

First, let’s take a look atHttpClientHow to handle response compression in

//Custom httpclienthandler instance
HttpClientHandler httpClientHandler = new HttpClientHandler
{
    AutomaticDecompression = DecompressionMethods.GZip
};
//Use the constructor that passes a custom httpclienthandler instance
using (HttpClient client = new HttpClient(httpClientHandler))
{
    var response = await client.GetAsync($"http://MyDemo/Home/GetPerson?userId={userId}");
}

This operation is still very simple. What we do is notHttpClientInstead ofHttpClientHandlerIn the previous article[.NET Core HttpClientSource code exploration],HttpClientThe essence of isHttpMessageHandler, andHttpClientWhat is really used isHttpMessageHandlerThe most important subclassHttpClientHandler, all request operations are performed throughHttpMessageHandlerConducted. We can seeAutomaticDecompressionAccept thatDecompressionMethodsEnumeration. Since it is enumeration, it means that it contains more than one value. Next, let’s seeDecompressionMethodsSource code in

[Flags]
public enum DecompressionMethods
{
    //Use all compression and decompression algorithms.
    All = -1,
    //Do not use decompression
    None = 0x0,
    //Using gzip decompression algorithm
    GZip = 0x1,
    //Using deflate decompression algorithm
    Deflate = 0x2,
    //Using the brotli decompression algorithm
    Brotli = 0x4
}

This enumeration is for common output decompression algorithms by default. Next, let’s take a look at theHttpClientFactoryHow to handle response compression in. In the previous article [.netCore HttpClientFactory+ConsulImplement service discovery]HttpClientFactoryGeneral working mode of defaultPrimaryHandlerThe httpclienthandler instance is passed, and it is registeredHttpClientFactoryYou can pass theConfigurePrimaryHttpMessageHandlercustomPrimaryHandlerNext, we will implement the default value of

services.AddHttpClient("mydemo", c =>
{
    c.BaseAddress = new Uri("http://MyDemo/");
}).ConfigurePrimaryHttpMessageHandler(provider=> new HttpClientHandler
{
    AutomaticDecompression = DecompressionMethods.GZip
});

Actually registeredHttpClientFactoryYou can also use the customizedHttpClientThe specific usage is as follows

services.AddHttpClient("mydemo", c =>
{
    c.BaseAddress = new Uri("http://MyDemo/");
}).ConfigureHttpClient(provider => new HttpClient(new HttpClientHandler
{
    AutomaticDecompression = DecompressionMethods.GZip
}));

HttpClientIndeed, it has helped us do a lot of things. It only needs a simple configuration to start the processing of response compression. It’s even more intriguingHttpClientNext, we will view how it initiates a responsive compression request and decompresses the response results through the source code.

2、 Source code exploration

From the above usage, we know that no matter which form is used, it is aimed atHttpClientHandlerDo the configuration operation. Next, let’s checkHttpClientHandlerIn classAutomaticDecompressionCode for attribute

public DecompressionMethods AutomaticDecompression
{
    get => _underlyingHandler.AutomaticDecompression;
    set => _underlyingHandler.AutomaticDecompression = value;
}

Its own value operation comes from_underlyingHandlerThis object, that is, both reading and setting are in operation_underlyingHandler.AutomaticDecompression, we found_underlyingHandlerObject declaration location

private readonly SocketsHttpHandler _underlyingHandler;

Here is an explanation,HttpClientThe substantive work category isHttpClientHandler, andHttpClientHandlerThe real request depends onSocketsHttpHandlerThis class, that isSocketsHttpHandlerIs the class that originally initiated the request.HttpClientHandlerEssence or passSocketsHttpHandlerHTTP requests initiated. Next, we will checkSocketsHttpHandlerHow classes are handledAutomaticDecompressionOf this attribute

public DecompressionMethods AutomaticDecompression
{
    get => _settings._automaticDecompression;
    set
    {
        CheckDisposedOrStarted();
        _settings._automaticDecompression = value;
    }
}

there_settingsIt is no longer a specific function class, but used for initialization or savingSocketsHttpHandlerConfiguration class for partial attribute values of

private readonly HttpConnectionSettings _settings = new HttpConnectionSettings();

We are not analyzing hereSocketsHttpHandlerThere are other codes except for processing response compression, so I won’t look at these for details. I can find them directly_settings._automaticDecompressionProperty, and finally found this code

if (settings._automaticDecompression != DecompressionMethods.None)
{
    handler = new DecompressionHandler(settings._automaticDecompression, handler);
}

It is clear here that the real processing related to request response compression is in theDecompressionHandlerMedium. As we said before,HttpClientThe real way to work is to implement selfHttpMessageHandler, which encapsulates the implementation modules of different functions into concreteHandlerMedium. When you need to use the function of which module, you can directly use the correspondingHandlerThe operation class can send the processing request. This design idea isASP.NET CoreIt is also reflected incisively and vividly,ASP.NET CoreThe approach is to build different endpoints to process and output requests. From these we can knowDecompressionHandlerThat’s the topic of today. Next, let’s check it outDecompressionHandlerClass, let’s first look at the coreSendAsyncMethod, which is the execution method for sending the request

internal override async ValueTask<HttpResponseMessage> SendAsync(HttpRequestMessage request, bool async, CancellationToken cancellationToken)
{
    //Judge whether it is a gzip compression request. If yes, add the request header accept encoding header as gzip
    if (GZipEnabled && !request.Headers.AcceptEncoding.Contains(s_gzipHeaderValue))
    {
        request.Headers.AcceptEncoding.Add(s_gzipHeaderValue);
    }
    //Judge whether it is a deflate compression request. If yes, add the request header accept encoding header as deflate
    if (DeflateEnabled && !request.Headers.AcceptEncoding.Contains(s_deflateHeaderValue))
    {
        request.Headers.AcceptEncoding.Add(s_deflateHeaderValue);
    }
    //Judge whether it is a brotli compression request. If yes, add the request header accept encoding as brotli
    if (BrotliEnabled && !request.Headers.AcceptEncoding.Contains(s_brotliHeaderValue))
    {
        request.Headers.AcceptEncoding.Add(s_brotliHeaderValue);
    }
    //Send request
    HttpResponseMessage response = await _innerHandler.SendAsync(request, async, cancellationToken).ConfigureAwait(false);

    Debug.Assert(response.Content != null);
    //Get the returned content encoding output header information
    ICollection<string> contentEncodings = response.Content.Headers.ContentEncoding;
    if (contentEncodings.Count > 0)
    {
        string? last = null;
        //Get last value
        foreach (string encoding in contentEncodings)
        {
            last = encoding;
        }
        //Judge whether the server adopts gzip compression according to the response header
        if (GZipEnabled && last == Gzip)
        {
            //Use the gzip decompression algorithm to decompress the returned content, and assign a new value to the response Content
            response.Content = new GZipDecompressedContent(response.Content);
        }
        //Judge whether the server adopts deflate compression according to the response header
        else if (DeflateEnabled && last == Deflate)
        {
            //Decompress the returned content using the deflate decompression algorithm, and assign a new value to the response Content
            response.Content = new DeflateDecompressedContent(response.Content);
        }
        //Judge whether the server adopts brotli compression according to the response header
        else if (BrotliEnabled && last == Brotli)
        {
            //Extract the returned content using the brotli decompression algorithm, and assign a new value to the response Content
            response.Content = new BrotliDecompressedContent(response.Content);
        }
    }
    return response;
}

From the above logic, we can see thatGZipEnabled、DeflateEnabled、BrotliEnabledThree variables of bool type determine which request compression method is used. The main implementation methods are

internal bool GZipEnabled => (_decompressionMethods & DecompressionMethods.GZip) != 0;
internal bool DeflateEnabled => (_decompressionMethods & DecompressionMethods.Deflate) != 0;
internal bool BrotliEnabled => (_decompressionMethods & DecompressionMethods.Brotli) != 0;

Mainly according to our configurationDecompressionMethodsThe enumeration value determines which compression result you want to obtain. The decompression implementation logic is encapsulated inGZipDecompressedContent、DeflateDecompressedContent、BrotliDecompressedContentMedium,Let’s take a look at their specific code

private sealed class GZipDecompressedContent : DecompressedContent
{
        public GZipDecompressedContent(HttpContent originalContent)
            : base(originalContent)
        { }
        //Decompress the returned stream using the gzipstream class
        protected override Stream GetDecompressedStream(Stream originalStream) =>
            new GZipStream(originalStream, CompressionMode.Decompress);
    }

    private sealed class DeflateDecompressedContent : DecompressedContent
    {
        public DeflateDecompressedContent(HttpContent originalContent)
            : base(originalContent)
        { }
        //Decompress the returned stream using the deflatstream class
        protected override Stream GetDecompressedStream(Stream originalStream) =>
            new DeflateStream(originalStream, CompressionMode.Decompress);
    }

    private sealed class BrotliDecompressedContent : DecompressedContent
    {
        public BrotliDecompressedContent(HttpContent originalContent) :
            base(originalContent)
        { }
        //Decompress the returned stream using the brotlistream class
        protected override Stream GetDecompressedStream(Stream originalStream) =>
            new BrotliStream(originalStream, CompressionMode.Decompress);
    }
}

Its main working mode is to use the decompression method of the corresponding compression algorithm to get the original information. To sum up briefly,HttpClientThe compression related processing mechanism is based on your configurationDecompressionMethodsDetermine which compression algorithm you want to use. Then add accept after matching to the corresponding compression algorithm-EncodingThe request header is the desired compression algorithm. Finally, it is obtained according to the response resultsContent-EncodingOutput header information, determine which compression algorithm the server uses, and use the corresponding decompression method to extract the original data.

Summary:

Through this discussionHttpClientWe can learn about the processing of response compression,HttpClientIt is highly flexible and extensible both in design and implementation, which is why.Net CoreThe official only recommends thatUsing httpclientAn HTTP request mode. Since the use is relatively simple and the implementation method is relatively clear, I will not elaborate here. Mainly to tell youHttpClientBy default, response compression can be handled directly, instead of the same as that we used beforeHttpWebRequestIt also needs to be manually coded to implement.