Creation and use of k-k-row cache module in DCache distributed storage system

Time：2021-3-2

Creation and use of k-k-row cache module in DCache distributed storage system

By Eaton

Introduction|With the development of micro services and cloud, the demand for distributed architecture is becoming more and more common. The traditional SQL structured storage solution has been unable to keep up with the pace, so NoSQL appeared. As a distributed NoSQL caching system based on tars, DCache perfectly supports tars service. In the previous article, we introduced how to create and use the kV module. This article will continue to introduce how to create and use the k-k-row cache module in DCache.

A series of articles

catalog

  • Introduction of k-k-row module
  • Create k-k-row cache module
  • Get DCache interface file
  • Create cache service proxy

  • Call cache module service

    • Read and write operation of k-k-row module
    • Running examples
  • summary

DCache is a distributed NoSQL storage system based on tars framework, which supports a variety of data structures, includingkey-value(key value pair),k-k-row(multiple key values),list(list),set(set),zset(orderly set) to meet a variety of business needs.

We are in the articleCreation and use of key value cache moduleIn this paper, we introducekey-valueIn addition, the disadvantages of structured data storage are also mentioned. andk-k-rowType is a structured storage solution.

Introduction of k-k-row module

k-k-row, andkey-valueSimilar, but here value is not a string, but is equivalent to a table, which can store structured data.k-k-rowNamelykey key rowThrough twokey, i.e. primary index / primary key（Main Key）Joint index（Union Key）To uniquely identify a recordrow, as follows

Creation and use of k-k-row cache module in DCache distributed storage system

It’s not hard to see,k-k-rowThe storage structure of is very similar to SQL database. The primary key is equivalent to the table name and mapped to value. It doesn’t need to store data repeatedly, and it doesn’t bring the problems of serialization and concurrent modification control.

Similar to the kV module, we only need to complete the following steps to use it in the servicek-k-rowCaching service

  1. Create k-k-row cache module
  2. Get DCache interface file
  3. Create cache service proxy
  4. Call cache module service

This article will continue to be based onTestDemoDescribes how to create K-K-Row cache modules and how to invoke the service in TARS services to cache data.

The examples used in this article can be found in the GitHub repositoryDCacheDemoView in.

Create k-k-row cache module

In the articleCreation and use of key value cache moduleIn this section, we have described how to create a key value cache module. The creation process of each type of cache module is similar. We will not repeat this part, but only introduce different parts.

Here we name the caching module serviceTestDemoKKRow，Cache typechoicek-k-row(MKVCache), as follows

Creation and use of k-k-row cache module in DCache distributed storage system

K-k-row is a multi key value type. When configuring a field, you can add multiple joint indexes or data fields. Clickadd to, as follows

Creation and use of k-k-row cache module in DCache distributed storage system

After confirming the configured information, clickInstallation releaseYou can finish publishing.

At this point, we can use this caching module to cache k-k-row data in other services.

Get DCache interface file

DCache is developed based on tars, so like tars service, it is also developed through.tarsInterface file to call the interface of the corresponding cache service.

We copyDCache/src/TarsCommNextCacheShare.tars, ProxyShare.tarsandDCache/src/ProxyNextProxy.tarsGo to your project directory.

In this paper, the project file structure after demo obtains DCache interface file is as follows

DCacheDemo
├── CacheShare.tars
├── ProxyShare.tars
├── Proxy.tars
├── config.conf
├── main.cpp
└── makefile

Create cache service proxy

In the previous article, we mentioned that after creating an application, a routing service and a proxy service will be automatically created, and the service will be sent throughTestDemoDescribes how to create a cache service proxy to call a service.

We continue to use itTestDemo, add a new module nameModuleTestDemoKKRow, the value is the module name we created earlierTestDemoKKRow, which is used to call the module through the proxy, as follows.

// main.cpp
#include <iostream>
#include <map>
#include "servant/Communicator.h"
#include "servant/ServantProxy.h"
#include "Proxy.h"

using namespace std;
using namespace tars;

//Testdemo proxy service object name
static string DCacheTestDemoObj = "DCache.TestDemoProxyServer.ProxyObj";

//Cache module name
static string ModuleTestDemoKV    = "TestDemoKV";
static string ModuleTestDemoKKRow = "TestDemoKKRow";

int main(int argc, char *argv[])
{
    CommunicatorPtr comm = new Communicator();
    try
    {
        TC_Config conf;
        //Parsing configuration files
        conf.parseFile("config.conf");
        //Load configuration
        comm->setProperty(conf);
        //Generation agent
        auto prx = comm->stringToProxy<DCache::ProxyPrx>(DCacheTestDemoObj);

        //Todo: call DCache cache service
    }
    catch (exception &e)
    {
        cerr << "error: " << e.what() << endl;
    }
    catch (...)
    {
        cerr << "Unknown Error" << endl;
    }
}

Call k-k-row cache module service

adoptTestDemoThe proxy object and module name of the proxy serviceTestDemoKKRowThen we can call the interface of the k-k-row cache module created earlier.

This section will introduce thek-k-rowThe use of some interfaces of type cache module. For information about other interfaces, seeProxy interface Guide。

The interface call process is consistent with the tars service interface call process. If you don’t know how and how to call tars service, you can read the articleTars RPC communication framework provides a variety of remote call methodsKnow how to call tars service.

In the following example, three utility functions are used, defined as follows

//Build updatevalue
DCache::UpdateValue genUpdateValue(DCache::Op op, const string &value)
{
    DCache::UpdateValue updateValue;
    updateValue.op = op;
    updateValue.value = value;
    return updateValue;
}

//Print map < string, string > type data
void printMapData(const map<string, string> &data)
{
    map<string, string>::const_iterator it = data.begin();
    while (it != data.end())
    {
        cout << "|" << it->first << ":" << it->second;
        ++it;
    }
    cout << endl;
}

//Print vector < Map > Data
void printVectorMapData(const vector<map<string, string>> &data)
{
    for (auto item : data)
    {
        printMapData(item);
    }
}

genUpdateValueFor buildingDCache::UpdateValueStructure, which is used to store the inserted or updated data values, is often used in the interface of other types of modules.printMapDataandprintVectorMapDataFor easy printing of returned data.

Next, let’s see how to use the k-k-row caching module.

Read and write operation of k-k-row module

K-k-row is a multi key module. A primary key can correspond to multiple records. Here we only introduce the write interfaceinsertMKVAnd read interfacegetMKVOther interfaces are similar.

insert data

InterfaceinsertMKVUsed to insert key value pair data, defined as follows

int insertMKV(const InsertMKVReq &req)

StructureInsertMKVReqAnd its nested structureInsertKeyValueThe definition is as follows

struct InsertMKVReq
{
  1 require string modulename; // module name
  2 require insertkeyvalue data; // data to be written
};

struct InsertKeyValue
{
  1 require string mainkey; // main key
  2 require map < string, updatevalue > mpvalue; // data of other fields except the main key
  3 require byte ver = 0; // version number
  4 require bool dirty = true; // set to dirty data
  5 require bool replace = false; // if the record already exists and replace is true, the old record will be overwritten
  6 require int expiretimesecond = 0; // data expiration time
};

Examples are as follows

void testInsertMKV(const string &mainKey, const map<string, string> &data, DCache::ProxyPrx prx)
{
    cout << "\t-- " << "insertMKV ";
    //Print data to be inserted
    printMapData(data);

    //Construct insert data
    DCache::InsertKeyValue insertData;
    insertData.mainKey = mainKey;

    map<string, string>::const_iterator it = data.begin();
    while (it != data.end())
    {
        //Construct updatevalue
        insertData.mpValue[it->first] = genUpdateValue(DCache::SET, it->second);
        ++it;
    }

    //Construct request
    DCache::InsertMKVReq insertReq;
    insertReq.moduleName = ModuleTestDemoKKRow;
    insertReq.data = insertData;

    prx->insertMKV(insertReq);
}

get data

InterfacegetMKVIt is used to obtain the key value pair corresponding to the primary key according to the primary key. The definition is as follows

int getMKV(const GetMKVReq &req, GetMKVRsp &rsp)

Request message structureGetMKVReqAnd return message structureGetMKVRspThe definition is as follows

struct GetMKVReq
{
  1 require string modulename; // module name
  2 require string mainkey; // main key
  3 require string field; // the set of fields to be queried. Multiple fields are separated by ',', such as "a, B" and "*" for all fields
  4 require vector < condition > cond; // query the condition set. For fields other than the main key, multiple conditions are directly and related
  5 require bool retentrycnt = false; // return the total number of records in the primary key
  6 require string idcspecified = "; // IDC region
};

struct GetMKVRsp
{
  1 require vector < map < string, string > > data; // query result
};

Examples are as follows

void testGetMKV(const string &key, DCache::ProxyPrx prx)
{
    cout << "\t-- " << "getMKV    " << '\n';

    //Construct request
    DCache::GetMKVReq req;
    req.moduleName = ModuleTestDemoKKRow;
    req.mainKey = key;
    req.field = "*";

    DCache::GetMKVRsp rsp;
    prx->getMKV(req, rsp);

    //Print return data
    printVectorMapData(rsp.data);
}

Running examples

Let’s actually run the above example. Complete examples can be found in GitHub repositoryDCacheDemoFrom.

We passedtestKKRowTo test the module read / write interface mentioned in the previous section, we insert two records into the same primary key,UIDThey aretest1, test2, as follows

void testKKRow(DCache::ProxyPrx prx)
{
    cout << START << " testKKRow" << endl;

    string mainKey = "Key";
    map<string, string> data;
    data["UID"] = "test1";
    data["VALUE"] = "hello";
    testInsertMKV(mainKey, data, prx);

    data["UID"] = "test2";
    data["VALUE"] = "hey";
    testInsertMKV(mainKey, data, prx);

    testGetMKV(mainKey, prx);

    cout << END << " testKKRow" << endl;
}

Then, in themainFunction

int main(int argc, char *argv[])
{
    ...

        auto prx = comm->stringToProxy<DCache::ProxyPrx>(DCacheTestDemoObj);

        //Call DCache cache service
        testKKRow(prx);
    ...
}

Compile, build and run the example, and the results are as follows

Creation and use of k-k-row cache module in DCache distributed storage system

As you can see,getMKVTwo records were returned. The above is the specific use process of DCache cache module. So far, we have successfully called DCache’s k-k-row caching service.

K-k-row cache module service interface

In addition to setting the key value interfaceinsertMKVAnd read key value interfacegetMKVDCache also provides rich k-k-row operation interfaces, including batch insert（insertMKVBatch）, delete（delMKV）, update（updateMKV）And so on, as follows

//Query by main key, support 'and' condition matching
int getMKV(GetMKVReq req, out GetMKVRsp rsp);
//Batch data query by primary key, given multiple primary keys, matching query with unified conditions
int getMKVBatch(MKVBatchReq req, out MKVBatchRsp rsp);
//Batch query by primary key
int getMUKBatch(MUKBatchReq req, out MUKBatchRsp rsp);
//Batch query by primary key, support 'and' or 'complex condition matching for each primary key
int getMKVBatchEx(MKVBatchExReq req, out MKVBatchExRsp rsp);
//Gets the number of records under the primary key. When the return value is positive, the number of records under the primary key is zero
int getMainKeyCount(MainKeyReq req);
//Get all the primary keys in the cache, excluding the DB keys
int getAllMainKey(GetAllKeysReq req, out GetAllKeysRsp rsp);
//Insert a record into the cache
int insertMKV(InsertMKVReq req);
//Insert bulk data into cache
int insertMKVBatch(InsertMKVBatchReq req, out MKVBatchWriteRsp rsp);
//Batch update interface. Only update of specified union key is supported
int updateMKVBatch(UpdateMKVBatchReq req, out MKVBatchWriteRsp rsp);
//Update cache record, update interface cannot update union key field.
int updateMKV(UpdateMKVReq req);
//Atomic update interface. It is suitable for data self increase and self decrease operation, and multi thread operation can ensure data atom update.
int updateMKVAtom(UpdateMKVAtomReq req);
//Delete cache record
int eraseMKV(MainKeyReq req);
//Delete cache and DB records
int delMKV(DelMKVReq req);
//Batch deletion, rsp.rspData The results of each delete request are stored in
int delMKVBatch(DelMKVBatchReq req, out MKVBatchWriteRsp rsp);

The interface is used in the same way as described aboveinsertMKVandgetMKVIt is similar. For the specific input and output parameter structures of the interface, please refer toProxy interface Guide。

summary

This paper introduces how to create and use the k-k-row cache module in DCache by using examples, so as to meet the needs of developers for structured cache data.

Tars can help developers and enterprises quickly build their own stable and reliable distributed applications in the way of micro services, so that developers only focus on business logic and improve operational efficiency. Multi language, agile R & D, high availability and efficient operation make tars an enterprise product.

Tars microservice helps you with digital transformation. Welcome to:

Tars website:https://TarsCloud.org

Tars source code:https://github.com/TarsCloud

Linux foundation official microservice free course:https://www.edx.org/course/bu…

Access to tars official training e-book:https://wj.qq.com/s2/7849909/…

Or scan the code to get:

Creation and use of k-k-row cache module in DCache distributed storage system

Tags: , , , ,

Recommended Today

Redis design and implementation 4: Dictionary Dict

In redis, the dictionary is the infrastructure. Redis database data, expiration time and hash type all take the dictionary as the underlying structure. Structure of dictionary Hashtable The implementation code of hash table is as follows:dict.h/dictht The dictionary of redis is implemented in the form of hash table. typedef struct dictht { //Hash table array, […]