Ioredis source code reading [0]

Time:2021-6-23

Recently, because of the need of work, I want to build a node.js terminalRedis ClientComponent out, temporarily select throughioredisAs a fork object.
Because I’ve met them beforeRedisWhen using twemproxy, there will always be the problem of unable to connect to the server. For details, see issues: https://github.com/luin/ioredis/issues/573
So we will modify the source code to modify the problem. However, after the modification, we run the unit test and find that it’s not so simple, it’s not just Info – > Ping, so we have to get familiar with the source code and adjust the logic accordingly.

<!–more–>

Ioredis project structure

From the point of view of the project, the source code is availablelibFolder, is a pure TS project.
libThe files under the directory are mainly provided with some general capabilities, such ascommandpipelineAnd data transmission.

.
ι -- datahandler.ts # data processing
├── ScanStream.ts
├── SubscriptionSet.ts
├── autoPipelining.ts
Implementation of cluster # redis cluster mode
│   ├── ClusterOptions.ts
│   ├── ClusterSubscriber.ts
│   ├── ConnectionPool.ts
│   ├── DelayQueue.ts
│   ├── index.ts
│   └── util.ts
The concrete implementation of command. TS # command
ι -- dispatcher of commander.ts # command
ο - Connectors ා network connection related
│   ├── AbstractConnector.ts
│   ├── SentinelConnector
│   ├── StandaloneConnector.ts
│   └── index.ts
├ -- errors ා abnormal information correlation
│   ├── ClusterAllFailedError.ts
│   ├── MaxRetriesPerRequestError.ts
│   └── index.ts
Θ -- index.ts # entry file
ι -- pipeline.ts # pipeline logic
ι -- a package of promise container.ts # promise
Implementation of redis #'redis instance`
│   ├── RedisOptions.ts
│   ├── event_handler.ts
│   └── index.ts
├── script.ts
├── transaction.ts
├── types.ts
└ -- utils # the realization of some tool functions
    ├── debug.ts
    ├── index.ts
    └── lodash.ts

And the next two folders,redisAndclusterIt’s all concreteredis clientrealization,clusterIt’s correspondingclusterCluster implementation.
So I’m watchingREADMEWe will find that there are two kinds of instances that can be used, https://www.npmjs.com/package/ioredis

new Redis
new Redis.Cluster

Let’s start with the most commonRedisAt first, this note is mainly forRedisWith readme, the logic is smoothed step by step.

const `Redis` = require("ioredis");
const `Redis` = `new Redis`();

redis.set("foo", "bar");
redis.get("foo", function (err, result) {
  if (err) {
    console.error(err);
  } else {
    console.log(result);
  }
});

The most basic use order is to instantiate one firstRedisObject and then callRedisThe corresponding command, ifRedisIf you are not familiar with the command, you can take a look at this website: https://redis.io/commands#

The entry code is located in redis / index. Ts, thoughioredisTS is used, but the implementation of the constructor still uses the very old Es5 method, which inherits theEventEmitterandCommanderThere are two classes, the first one iseventsYes, the second isioredisA class provided by myself is in thecommander.tsFile.

Redis instantiation

RedisThe main tasks are as follows:

  • Set up and maintain communication withRedis ServerNetwork connection for
  • health examination
  • Maintain the queue to ensure that the request will not be lost in case of exception and can be retried

Look backRedisYou’ll see an assignment to this. Connector, leaving out the customConnectorAndSentinelsJust look at the last, the most commonStandaloneConnectorThis is where we build and interactRedis ServerIt’s connected.
Flip throughlib/connectors/StandaloneConnector.tsYou will find that the final call isnet.createConnectionIn fact, this can be the same as what we mentioned aboveRESPCorresponding to, is to use the most basicRedisCommunication protocol to complete the operation.

After the parameters are initialized, theconnectCome withRedis ServerMake a real connection.

netModularcreateConnectionOnly network connection can be established, which is not guaranteed to be what we expectRedisServices.
adoptconnectGot itstreamThe object is actuallysocket client:https://github.com/luin/ioredis/blob/master/lib/redis/index.ts#L321
stayconnectThe main method is to set up and manageRedis ServerAfter establishing the connection, we will callevent_handler.connectHandlermethod.
Two things have been done here

  1. Try checkRedis ServerThat is the pit we mentioned at the beginning. We can go through itRedis.prototype._readyCheckMethod to see the specific implementation,ioredisuseinfoCommand as a probe, but this is in thetwemproxyThere are some problems in cluster mode, because some commands are disabled in this mode, includinginfoThen this leads toRedis ClientServices are always considered unavailable.
  2. Added forsocket clientOfdataEvent monitoring is used to receive the returned data. The main logic isDataHandler.ts, which will be mentioned later.

The logic of readycheck exists in redis / index.ts and redis / event_ In the handler.ts file

Redis.prototype._readyCheck = function (callback) {
  const _this = this;
  this.info(function (err, res) {
    if (err) {
      return callback(err);
    }
    if (typeof res !== "string") {
      return callback(null, res);
    }

    const info: { [key: string]: any } = {};

    const lines = res.split("\r\n");
    for (let i = 0; i < lines.length; ++i) {
      const [fieldName, ...fieldValueParts] = lines[i].split(":");
      const fieldValue = fieldValueParts.join(":");
      if (fieldValue) {
        info[fieldName] = fieldValue;
      }
    }

    if (!info.loading || info.loading === "0") {
      callback(null, info);
    } else {
      const loadingEtaMs = (info.loading_eta_seconds || 1) * 1000;
      const retryTime =
        _this.options.maxLoadingRetryTime &&
        _this.options.maxLoadingRetryTime < loadingEtaMs
          ? _this.options.maxLoadingRetryTime
          : loadingEtaMs;
      debug("Redis server still loading, trying again in " + retryTime + "ms");
      setTimeout(function () {
        _this._readyCheck(callback);
      }, retryTime);
    }
  });
};

In detectionRedisIt will trigger when it is availablecallback, thecallbackI’ll check it outofflineQueueWhether there is a value can be interpreted as yesRedisThose records that were invoked before can be used.ioredisIt will not directly report an error to tell you that the connection has not been established, but it will be temporarily stored in its own queue and sent out in order when it is available.
RedisIn the process of instantiation, we mainly do these things. Next we will seeRedisAfter the command is issued, the logic of specific execution has been changed.

Commander

The function of commander is to implement various functionsRedis ClientBy order of https://www.npmjs.com/package/redis-commands Traversal.
At the same time, it will focus onClientOfReadyState, in theReadyI will do some operations such as temporary command before.
It’s more like an abstract class, becauseRedisandRedis ClusterWill inherit and override some API to complete the work.

commands.forEach(function (commandName) {
  Commander.prototype[commandName] = generateFunction(commandName, "utf8");
  Commander.prototype[commandName + "Buffer"] = generateFunction(
    commandName,
    null
  );
});

function generateFunction(_encoding: string);
function generateFunction(_commandName: string | void, _encoding: string);
function generateFunction(_commandName?: string, _encoding?: string) {
  if (typeof _encoding === "undefined") {
    _encoding = _commandName;
    _commandName = null;
  }

  return function (...args) {
    const commandName = _commandName || args.shift();
    let callback = args[args.length - 1];

    if (typeof callback === "function") {
      args.pop();
    } else {
      callback = undefined;
    }

    const options = {
      errorStack: this.options.showFriendlyErrorStack
        ? new Error().stack
        : undefined,
      keyPrefix: this.options.keyPrefix,
      replyEncoding: _encoding,
    };

    if (this.options.dropBufferSupport && !_encoding) {
      return asCallback(
        PromiseContainer.get().reject(new Error(DROP_BUFFER_SUPPORT_ERROR)),
        callback
      );
    }

    // No auto pipeline, use regular command sending
    if (!shouldUseAutoPipelining(this, commandName)) {
      return this.sendCommand(
        new Command(commandName, args, options, callback)
      );
    }

    // Create a new pipeline and make sure it's scheduled
    return executeWithAutoPipelining(this, commandName, args, callback);
  };
}

In the implementation of all commands, but also to achieve a number ofBufferSuffix API, the main difference between them can be found throughgenerateFunctionFunction is passed into theCommandExample.
andCommandObjects are specific command implementations, so we need to look at command first.

Command

CommandResponsible for things, mainly parameter processing, return value processing, generate the actual value of the command transmission, and so oncallbackIt’s a trigger.

instantiation

stayCommandIn the process of instantiation of, in addition to the assignment of some properties, ainitPromiseMethod, anPromiseObject.
There are two important processing, one is about parameter conversion, and the other is about return value.

private initPromise() {
  const Promise = getPromise();
  const promise = new Promise((resolve, reject) => {
    if (!this.transformed) {
      this.transformed = true;
      const transformer = Command._transformer.argument[this.name];
      if (transformer) {
        this.args = transformer(this.args);
      }
      this.stringifyArguments();
    }

    this.resolve = this._convertValue(resolve);
    if (this.errorStack) {
      this.reject = (err) => {
        reject(optimizeErrorStack(err, this.errorStack, __dirname));
      };
    } else {
      this.reject = reject;
    }
  });

  this.promise = asCallback(promise, this.callback);
}

Special treatment of parameters and return values

If searchingCommand.tsFile, you’ll findCommand._transformer.argumentadoptsetArgumentTransformerMethod.
And then look at the codesetArgumentTransformerThere are only a fewhsetOrders, andmsetOrders.

Command.setArgumentTransformer("hmset", function (args) {
  if (args.length === 2) {
    if (typeof Map !== "undefined" && args[1] instanceof Map) {
      return [args[0]].concat(convertMapToArray(args[1]));
    }
    if (typeof args[1] === "object" && args[1] !== null) {
      return [args[0]].concat(convertObjectToArray(args[1]));
    }
  }
  return args;
});

If you have used itRedisOfhash setYou should know that multiple key values are operated by appending parameters

> HMSET key field value [field value ...]

In this way, you need to pass in an array to use in JS, and the key value of the array is maintained by the user himself. Such a sequential operation mode must not be used to writing JSObjectIt’s comfortable, soioredisProvides a logic for parameter conversion, which is used toObjectConvert to one dimensional array:

export function convertObjectToArray(obj) {
  const result = [];
  const keys = Object.keys(obj);

  for (let i = 0, l = keys.length; i < l; i++) {
    result.push(keys[i], obj[keys[i]]);
  }
  return result;
}

export function convertMapToArray<K, V>(map: Map<K, V>): Array<K | V> {
  const result = [];
  let pos = 0;
  map.forEach(function (value, key) {
    result[pos] = key;
    result[pos + 1] = value;
    pos += 2;
  });
  return result;
}

If you look carefullyCommand._transformerYou’ll find another onereplyThe logic here is mainly in_convertValueAfter receiving the return value, we will call the user-defined function we passed in to process the return value.
At present, the only place to use the code ishgetallThe processing logic of,hmgetAndhgetallstayRedisAll of them return an array of dataioredisThe array is spliced into one in kV formatObjectIt is convenient for users to operate.

Command.setReplyTransformer("hgetall", function (result) {
  if (Array.isArray(result)) {
    const obj = {};
    for (let i = 0; i < result.length; i += 2) {
      obj[result[i]] = result[i + 1];
    }
    return obj;
  }
  return result;
});

set upkeyprefix

If you look at itCommandIn the process of instantiation, you will also find_iterateKeysSuch a function call has two functions:

  1. Extract all the keys in the parameter
  2. Optionally, add a prefix to the key

The function uses theredis-commandsThe two APIs of,existsandgetKeyIndexesTo get the subscripts of all the keys in the parameter array.
Because this function does two things, when you first see the usage of the constructor, and then look at the specific implementation of the function, you will find thatthis.keysI’m confused, but when I seeCommandIt also provides agetKeysThe API can understand the logic.

If setkeyPrefix, will trigger_iterateKeysIt is used to adjust the key name and store it in keys to return the value.
When callinggetKeysIf it is not setkeyPrefix, the default null processing function will be used to perform the same logic, that is, get all the keys and return them; If keyprefix has been set before, this.keys will be returned directly, and the logic will not be repeated.

Ioredis source code reading [0]

//Constructor inner logic
if (options.keyPrefix) {
  this._iterateKeys((key) => options.keyPrefix + key);
}

//The location of another call
public getKeys(): Array<string | Buffer> {
  return this._iterateKeys();
}

private _iterateKeys(
  transform: Function = (key) => key
): Array<string | Buffer> {
  if (typeof this.keys === "undefined") {
    this.keys = [];
    if (commands.exists(this.name)) {
      const keyIndexes = commands.getKeyIndexes(this.name, this.args);
      for (const index of keyIndexes) {
        this.args[index] = transform(this.args[index]);
        this.keys.push(this.args[index] as string | Buffer);
      }
    }
  }
  return this.keys;
}

Generation of sending command data

Everybody use itRedisIt should be more through the codeClientCall a variety of commands to do, occasionally through redis cli direct command line operation.
But actuallyRedisWe used a tool calledRESP(redis serialization protocol).
If the machine hasRedisIf so, we can simply demonstrate it locally.

> echo -e '*1\r\n$4\r\nPING\r\n' | nc 127.0.0.1 6379
+PONG

We’ll get one+PONGcharacter string. Such an interaction is actually the vast majorityClientAndRedis ServerThe format used when interacting.

P.S. RESPThere are human readable versions for interaction, but the performance is relatively low.

For example, if we want to execute a set and a get, how should we write this command

#The beginning represents a comment

# SET hello world
#Number of parameters
*3
#The length of the command value on this line (set command)
$3
#The value corresponding to the command (set command)
SET
#The length of the command value of this line (specific key: Hello)
$5
#The value corresponding to the command (specific key: Hello)
hello
#The length of the command value on this line (the length of value)
$5
#The value corresponding to the command (value ontology)
world

# GET hello
#Number of parameters
*2
#The length of the command value on this line (get command)
$3
#The value corresponding to the command (get command)
GET
#The length of the command value of this line (specific key: Hello)
$5
#The value corresponding to the command (specific key: Hello)
hello

setThere’s nothing unexpected about the return value of. It’s just a+OK, andgetThe return value of has two lines, the first line$5Indicates the length of the return value. The second line is the real return valueworld
So if you go to see itCommandThe towritable function of realizes this logic. Because it is relatively long, it is not pasted https://github.com/luin/ioredis/blob/master/lib/command.ts#L269

CommandThe main implementation is these logic, we are in theCommanderYou can see that all command calls are executed at the endthis.sendCommandThe specific scheduling is in theRedisRedis ClusterAnd so on. So we can go backRedisTake a look at the implementation logic.

Redis send command

sendCommandIn the implementation of, will be carried outRedisStatus check, if yeswaitperhapsendAnd so on, will carry on the corresponding processing.
Then we will check whether it is a status that can send commands

let writable =
    this.status === "ready" ||
    (!stream &&
      this.status === "connect" &&
      commands.exists(command.name) &&
      commands.hasFlag(command.name, "loading"));
  if (!this.stream) {
    writable = false;
  } else if (!this.stream.writable) {
    writable = false;
  } else if (this.stream._writableState && this.stream._writableState.ended) {
    writable = false;
  }

The code is fairly clear. Here’s another point that we’re dealing withinfoThe problem with the command is, usingpingOrder insteadinfo, which was stuck here at first, and the subsequent debugging found that,pingCommand does not haveloadingThis oneflagFeatures, sopingAll the orders were put inofflineQueueIn view of this situation, we willpingAdd an additional judgment logic to ensure thatwriteThe value of is true.

Next, ifwriteIf it’s true, then we’ll use itstreamThat is to say, the socket connection established in the front will send our real command, which is called at this timewriteAnd willCommand#toWritableThe return value of is passed in as data, which is based onRESPSerialization of the format.
At the same time, some information will be put in thecommandQueueIn the middle, it’s the same asofflineQueueThey are all examples of the same type, and the specific functions will be mentioned later.

this.commandQueue.push({
  Command: command, // command instance
  Stream: stream, // socket client
  Select: this.condition.select, // this is also not used
});

Another open source module, denque: https://www.npmjs.com/package…

IfwriteIf it is false, then the order will be placedofflineQueueIn the middle.

When the logic is over, thecommand.promiseGo back, we’re inCommandIn the process of instantiation, you can see that aPromiseObject, andresolveAndrejectMade a reference, after the data will be used to return.
After the command has been sent, the next step is to wait for the data to return. Here we will talk about the introductionReactAfter instantiationconnectCalledDataHandlerExamples of what has been done.

DataHandler

DataHandlerIt’s an alternative way of writing, because it’s direct when usednewIt does not receive a return value.
In the constructor, we do two things, one is to instantiate the otherRedisParserObject, the other is listeningredis.stream.on('data')Event, that is, we are instantiatingRedisIt’s passed on when you’re heresocket client, indataCalled when the event is triggeredRedisParser.executeTo complete the parsing.
RedisParserIt’s another open source module. If you are interested, you can see it here: https://www.npmjs.com/package/redis-parser
At present, it can be considered that theexecuteMethod will call thereturn ReplyIt’s OK. This is a result of analysisresponseWe’ll get thisresponseAfter that, I’ll start fromcommandQueueIn turn, the objects passed in before are taken out.
The method of fetching is in accordance with the way of queueshiftTo retrieve the first element in the queue each time.
And then calling the elementcommandAttributeresolveMethod, which is passed in when we call various redis commandscallbackIt’s too late.

Here is something to addRedisWe can see from the whole logical link that the relevant knowledge is roughly as follows:

  1. The user executes the command
  2. Redisinstantiation CommandAnd put it in the queue
  3. After receiving the data response, parse the data, get the first element in the queue, and call the correspondingcallback

There could be many at the same timeRedisThe request is sent out, but after receiving the data, it is not necessary to determine which response this time corresponds tocommandBecauseRedisIt is also a single process working mode, and the command processing will also be processed according to the order of receiving data, because it is a single processioredisThe same socket connection is used, so there is no saying that the order in which commands are sent to the remote end will change.
So we can rest assured that through the simplest way,push + shiftTo process the data.

This is also why some big key operations slow down the response of the entire redis service( In the case of no fragmentation or other processing)

Summary

So far, in normal modeRedis ClientWe have sorted out the whole logic, from creating to sending commands to receiving return values.
We’ll focus on it laterRedis ClusterOutput another note, let’s take a look at theClusterThere will be different processing logic in this mode.

reference material

  • ioredis
  • redis commands
  • Node.js | net
  • Why Redis is single-threaded and why is Redis so fast!
  • Redis is single-threaded, then how does it do concurrent I/O?