A detailed analysis of the serialization of redis and memcached in. Net core

Time:2020-2-3

Preface

When using distributed cache, it is inevitable to do such a step, serialize the data and then store it in the cache.

The serialization operation may be explicit or implicit, depending on whether the package used helps us do such a thing.

This article will use redis and memcached in the. Net core environment as examples. Redis mainly usesStackExchange.Redis, memcached mainly uses enyimemcachedcore.

Let’s take a look at some of our common serialization methods.

Common serialization methods

Perhaps, a common way is to serialize an object into a byte array, and then use the array to interact with the cache server.

As for serialization, there are many algorithms in the industry. In a sense, the result of these algorithms is speed and volume.

In fact, when operating distributed cache, we pay more attention to these two issues!

Under the same conditions, the speed of serialization and deserialization can determine whether the execution speed can be faster.

The result of serialization, that is, what we need to plug into memory, if we can make it smaller, it can save a lot of valuable memory space.

Of course, the focus of this article is not to compare the serialization methods, but to introduce how to use them in combination with cache, and also to mention some points that serialization can consider when using cache.

Here are some common serialization libraries:

  • System.Runtime.Serialization.Formatters.Binary
  • Newtonsoft.Json
  • protobuf-net
  • MessagePack-CSharp
  • ….

In these libraries

System.Runtime.Serialization.Formatters.BinaryIt’s in the. Net class library itself, so it’s a good choice when you don’t rely on third-party packages.

Newtonsoft. Jason should not have said much.

Protobuf net is the protocol buffers implemented by. Net.

Message pack CSharp is a very fast message pack serialization tool.

These serialization of the library is also the author usually involved, there are some unfamiliar did not list out!

Before we start, we define a product class, and the subsequent operations are based on this class.


public class Product
{
 public int Id { get; set; }
 public string Name { get; set; }
}

Let’s take a look at the use of redis.

Redis

Before introducing serialization, we need to know that in stackexchange.redis, the data we want to store is in the form of redisvalue. And redisvalue supports string, byte [] and other data types.

In other words, when we use stackexchange.redis, the data stored in redis needs to be serialized into the type supported by redisvalue.

This is what we said earlier, which requires explicit serialization.

Let’s take a look at the BinaryFormatter provided by the. Net class library.

Serialized operation


using (var ms = new MemoryStream())
{
 formatter.Serialize(ms, product);  
 db.StringSet("binaryformatter", ms.ToArray(), TimeSpan.FromMinutes(1));
}

Deserialization operation


var value = db.StringGet("binaryformatter");
using (var ms = new MemoryStream(value))
{
 var desValue = (Product)(new BinaryFormatter().Deserialize(ms));
 Console.WriteLine($"{desValue.Id}-{desValue.Name}");
}

It’s easy to write, but running the code at this time will prompt the following error!

Our product class is not marked serializable. Here is adding [serializable] to the product class.

Run it again, it’s already successful.

Let’s look at newtonsoft.json

Serialized operation


using (var ms = new MemoryStream())
{
 using (var sr = new StreamWriter(ms, Encoding.UTF8))
 using (var jtr = new JsonTextWriter(sr))
 {
 jsonSerializer.Serialize(jtr, product);
 }  
 db.StringSet("json", ms.ToArray(), TimeSpan.FromMinutes(1));
}

Deserialization operation


var bytes = db.StringGet("json");
using (var ms = new MemoryStream(bytes))
using (var sr = new StreamReader(ms, Encoding.UTF8))
using (var jtr = new JsonTextReader(sr))
{
 var desValue = jsonSerializer.Deserialize<Product>(jtr);
 Console.WriteLine($"{desValue.Id}-{desValue.Name}");
}

Since newtonsoft.json has no mandatory requirement for serializable classes, it is OK to remove or retain them.

It runs smoothly.

Of course, it can also be handled in the following ways:


var objStr = JsonConvert.SerializeObject(product);
db.StringSet("json", Encoding.UTF8.GetBytes(objStr), TimeSpan.FromMinutes(1));
var resStr = Encoding.UTF8.GetString(db.StringGet("json"));
var res = JsonConvert.DeserializeObject<Product>(resStr);

Let’s take a look at protobuf

Serialized operation


using (var ms = new MemoryStream())
{
 Serializer.Serialize(ms, product);
 db.StringSet("protobuf", ms.ToArray(), TimeSpan.FromMinutes(1));
}

Deserialization operation


var value = db.StringGet("protobuf");
using (var ms = new MemoryStream(value))
{
 var desValue = Serializer.Deserialize<Product>(ms); 
 Console.WriteLine($"{desValue.Id}-{desValue.Name}");
}

The usage seems to be standard.

But running like this is not so smooth. The error message is as follows:

There are two methods to deal with this problem. One is to add the corresponding attribute on the product class and attribute, and the other is to use protobuf.meta to deal with this problem at runtime. You can refer to the implementation of autoprotobuf.

The first way is to add[ProtoContract]and[ProtoMember]These two attributes.

Running again is what we expect.

Finally, take a look at message pack. According to its description and comparison on GitHub, it seems to be much stronger than other serialized libraries.

By default, it’s like protobufMessagePackObjectandKeyOf these two attributes.

However, it also provides an iformatterresolver parameter, which allows us to choose.

The following is a method that does not need to add attribute.

Serialized operation


var serValue = MessagePackSerializer.Serialize(product, ContractlessStandardResolver.Instance);
db.StringSet("messagepack", serValue, TimeSpan.FromMinutes(1));

Deserialization operation


var value = db.StringGet("messagepack");
var desValue = MessagePackSerializer.Deserialize<Product>(value, ContractlessStandardResolver.Instance);

It is normal to run at this time.

In fact, the serialization step is very simple for redis, because it explicitly lets us handle it and then store the results.

From the perspective of use, the four methods shown above seem to be almost the same, without much difference.

If you compare redis with memcached, you will find that the operation of memcached may be a little more complicated than redis.

Let’s take a look at the use of memcached.

Memcached

Enyimmmemcachedcore has a defaulttranscoder by default
, for general data types (int, string, etc.), this article will not go into details, but specifically describes the object type.

In defaulttranscoder, the serialization of object type data is based on bson.

Another binaryformattertranscoder is the default implementation, which is based on the. Net class librarySystem.Runtime.Serialization.Formatters.Binary

Let’s see how to use these two transcoders.

First, define the methods related to initialization of memcached, as well as the methods for reading and writing cache.

Initialize memcached as follows:


private static void InitMemcached(string transcoder = "")
{
 IServiceCollection services = new ServiceCollection();
 services.AddEnyimMemcached(options =>
 {
  options.AddServer("127.0.0.1", 11211);
  options.Transcoder = transcoder;
 });
 services.AddLogging();
 IServiceProvider serviceProvider = services.BuildServiceProvider();
 _client = serviceProvider.GetService<IMemcachedClient>() as MemcachedClient;
}

The transcoder here is the kind of serialization method we want to choose (for object type). If it’s empty, use bson. If it’s binaryformattertranscoder, use BinaryFormatter.

Note the following two instructions

  • After version 2.1.0, transcoder changed from itranscope type to string type.
  • After version 2.1.0.5, it can be done in the form of dependency injection without specifying a transcoder of string type.

The operations of read-write cache are as follows:


private static void MemcachedTrancode(Product product)
{
 _client.Store(Enyim.Caching.Memcached.StoreMode.Set, "defalut", product, DateTime.Now.AddMinutes(1));

 Console.WriteLine("serialize succeed!");

 var desValue = _client.ExecuteGet<Product>("defalut").Value;

 Console.WriteLine($"{desValue.Id}-{desValue.Name}");
 Console.WriteLine("deserialize succeed!");
}

Our code in the main method is as follows:


static void Main(string[] args)
{
 Product product = new Product
 {
  Id = 999,
  Name = "Product999"
 };
 //Bson
 string transcoder = "";
 //BinaryFormatter
 //string transcoder = "BinaryFormatterTranscoder";   
 InitMemcached(transcoder);
 MemcachedTrancode(product);
 Console.ReadKey();
}

For the two transcoders, they run smoothly and are in useBinaryFormatterTranscoderRemember to add [serializable] to the product class!

Let’s see how to implement memcached transcoder with message pack.

Here, you can inherit defaulttranscoder, and override serializeobject, deserializeobject and deserializeThese three methods.


public class MessagePackTranscoder : DefaultTranscoder
{
 protected override ArraySegment<byte> SerializeObject(object value)
 {
  return MessagePackSerializer.SerializeUnsafe(value, TypelessContractlessStandardResolver.Instance);
 }

 public override T Deserialize<T>(CacheItem item)
 {
  return (T)base.Deserialize(item);
 }

 protected override object DeserializeObject(ArraySegment<byte> value)
 {
  return MessagePackSerializer.Deserialize<object>(value, TypelessContractlessStandardResolver.Instance);
 }
}

Fortunately, messagepack has a way to directly sequence an object into arraysegment, you can also set the arraysegmentDeserialize to an object!!

Compared with Jason and protobuf, it saves a lot of operations!!

At this time, we have two ways to use the newly defined message pack transcoder.

Mode 1: when using, we only need to replace the previously defined transcoder variable (applicable to version > = 2.1.0).


string transcoder = "CachingSerializer.MessagePackTranscoder,CachingSerializer";

Note:If you use this method for processing, remember to spell the transcoder correctly and take the namespace with you, otherwise the transcoder created will always be null, which leads to bsonActivator.CreateInstanceThere should be no more explanation.

Mode 2: handle by dependency injection (applicable to version > = 2.1.0.5)

private static void InitMemcached(string transcoder = "")
{
 IServiceCollection services = new ServiceCollection();
 services.AddEnyimMemcached(options =>
 {
  options.AddServer("127.0.0.1", 11211);
  //If you leave the empty string or no value assigned here, you will go to the following addsingleton
  //If the correct value is assigned here, the following addsingleton will not work
  options.Transcoder = transcoder;
 });
 //Use the newly defined messagepacktranscoder
 services.AddSingleton<ITranscoder, MessagePackTranscoder>();
 //others...
}

Add a breakpoint before running to make sure it’s really in our overridden method.

Final results:

Protobuf and JSON are not introduced here. These two processes are much more complex than message pack. You can refer to memcached transcoder, an open source project written by the author of message pack. Although it was written five years ago, it is also easy to use.

For redis, when calling the set method, we need to explicitly serialize our values first, which is not so concise, so we will encapsulate them in use once.

For memcached, although we don’t need to explicitly serialize when we call the set method, we may have to implement a transcoder ourselves, which is a bit troublesome.

Now I recommend a simple cache library to deal with these problems.

Using easycaching to simplify operations

Easycaching is a simple open source project written by the author in my spare time. The main purpose is to simplify the operation of caching, which is also in constant improvement.

Easycaching provides four serialization methods as mentioned earlier:

  • BinaryFormatter
  • MessagePack
  • Json
  • ProtoBuf

If none of these four types meet the requirements, you can write one yourself, as long as you implement the corresponding method of ieasycacheingserializer interface.

Redis

Before introducing how to use serialization, let’s take a brief look at how to use it (using asp.net core web API for demonstration).

Add redis related nuget package


Install-Package EasyCaching.Redis

Modify startup


public class Startup
{
 //...
 public void ConfigureServices(IServiceCollection services)
 {
  //other services.
  //Important step for Redis Caching  
  services.AddDefaultRedisCache(option=>
  {    
   option.Endpoints.Add(new ServerEndPoint("127.0.0.1", 6379));
   option.Password = "";
  });
 }
}

Then use in the controller:


[Route("api/[controller]")]
public class ValuesController : Controller
{
 private readonly IEasyCachingProvider _provider;
 public ValuesController(IEasyCachingProvider provider)
 {
  this._provider = provider;
 }

 [HttpGet]
 public string Get()
 {
  //Set
  _provider.Set("demo", "123", TimeSpan.FromMinutes(1));
  //Get without data retriever
  var res = _provider.Get<string>("demo");
  _provider.Set("product:1", new Product { Id = 1, Name = "name"}, TimeSpan.FromMinutes(1))

  var product = _provider.Get<Product>("product:1");
  return $"{res.Value}-{product.Value.Id}-{product.Value.Name}"; 
 }
}
  • When it is used, the constructor can inject the dependency of ieasycacheingprovider.
  • Redis uses BinaryFormatter for serialization by default.

How can we replace the new serialization method we want?

Take messagepack as an example, first install the package through nuget


Install-Package EasyCaching.Serialization.MessagePack

Then just add the following sentence to the configureservices method.


public void ConfigureServices(IServiceCollection services)
{
 //others..
 services.AddDefaultMessagePackSerializer();
}

Memcached

Let’s also take a brief look at how to use it (using asp.net core web API for demonstration).

Add nuget package for memcached


Install-Package EasyCaching.Memcached

Modify startup


public class Startup
{
 //...
 public void ConfigureServices(IServiceCollection services)
 {
  services.AddMvc();
  //Important step for Memcached Cache
  services.AddDefaultMemcached(option=>
  {    
   option.AddServer("127.0.0.1",11211);   
  });  
 }

 public void Configure(IApplicationBuilder app, IHostingEnvironment env)
 {
  //Important step for Memcache Cache
  app.UseDefaultMemcached(); 
 }
}

The controller is exactly the same as Redis when used.

What we need to pay attention to here is, in easycaching, the default serialization method is not bson in defaulttranscoder, but BinaryFormatter

How to replace the default serialization operation?

Take the messagepack as an example. First install the package through nuget


Install-Package EasyCaching.Serialization.MessagePack

The rest of the operations are the same as redis!


public void ConfigureServices(IServiceCollection services)
{
 //others..
 services.AddDefaultMemcached(op=>
 {    
  op.AddServer("127.0.0.1",11211);
 });
 //specify the Transcoder use messagepack serializer.
 services.AddDefaultMessagePackSerializer();
}

Because there is a transcoder of its own in easycaching, which injects the ieasycacheingserializer, only the corresponding serializer needs to be specified.

summary

1、 Let’s take a look at the four serialization libraries mentioned in this article

System.Runtime.Serialization.Formatters.BinaryIn use, you need to add [serializable]. The efficiency is the slowest. The advantage is that there is one in the class library, and no additional reference to other packages is required.

Newtonsoft.json is friendly to use. It may be used more, and we don’t need to add some attributes to the defined classes.

Protobuf net may be a little bit cumbersome to use. You can add the corresponding attribute when defining a class, or handle it at runtime (pay attention to subclasses), but its reputation is still good.

Although messagepack CSharp can not add attribute, it will also lose some when it does not add ratio.

As for how to choose, it may depend on the situation!

If you are interested, you can run points with benchmarkdotnet. I also wrote a simple reference: serializerbanchmark

2、 When it comes to cache operations, it may be more inclined to “implicit” operations, which can directly throw an object in or take out an object, at least for the convenience of users.

3、 When serializing, redis is simpler than memcached.

Finally, if you are using easycaching, you can contact me if you have any questions or suggestions!

Example code for the first half: cacheingserializer

Sample code for the second half: Sample

Well, the above is the whole content of this article. I hope that the content of this article has some reference learning value for your study or work. If you have any questions, you can leave a message and exchange. Thank you for your support for developepaer.

Recommended Today

[reading notes] calculation advertising (Part 3)

By logm This article was originally published at https://segmentfault.com/u/logm/articles and is not allowed to be reproduced~ If the mathematical formula in the article cannot be displayed correctly, please refer to: Tips for displaying the mathematical formula correctly This article isComputing advertising (Second Edition)Reading notes. This part introduces the key technology of online advertising, which is […]