The most important data in online games is the data generated by players in the process of games.
It can be simply divided into two categories:
- archived data
- Process record data
The first type of data mainly includes the basic information of similar characters, knapsack, skills, tasks, as well as the Kingdom, map, alliance and other information shared by all (or some) players.
The second type is mainly similar to the “log” information, such as the operation record of “a certain property is used by a certain role in a certain place”.
In this article, we mainly discuss the processing of the first kind of data. As for the data used for recording and analysis, we may write a special blog later.
First, we need to determine the shape of the data, and the way to describe it.
Here is a choice: take “item format” as the core or “document format” as the core.
Advantages of “entry format”: it looks flat, convenient for real-time landing, convenient for matching with various database storage modes, convenient for external tools to modify, and convenient for batch processing.
“Document format” is a bit: it is usually a tree structure, which is more suitable for most of the role models in the brain. When it is processed in memory, it is generally more efficient.
Adopting different data structures usually affects the technology used in the project, even directly affects the overall architecture and processing model. If entries are used, Memcache / redis may be used for storage and more stateless business logic processing methods may be used.
What our team is used to is
structAs the core, cooperate with
protobufImplementation of interface and serialization, the main operations are handled in memory.
When using, the data structure in memory is the structure of go; when landing, use
bsonDocument format; used when interacting with mobile and some internal RPC interface calls
protobufAs a transport format; in configuration, using
etcdSynchronized CSV entry data.
Whether or not
protobufOr CSV item data, you can use the extensibility of go
struct tagTo complete the automatic conversion.
Data compatibility and upgrade
In the long-term online business maintenance and development process, the data structure of players will always be modified and updated.
If you stop the system every time, it is impossible to upgrade tens of millions or even hundreds of millions of data stored in the database. Generally, data compatibility check and update will be carried out during data loading.
At this time, in
goUsed in the program
bsonAs the final storage method of data, it will bring us a lot of convenience.
bson unmarshalMost of the “add” and “delete” operations can be perfectly handled by the automatic compatibility processing of. For those fields or composite structures that need to be modified, we can completely use the add operation to replace them. After upgrading, we can delete the old data.
The pain of using binary data v0.01 – > v0.02 – > v0.0n upgrade process no longer needs to be seen. And we don’t need to wait for a huge library to perform the alter table operation just like a project that uses items to store data.
Efficiency of serialization
There are many people who have done tests and have different results. Our actual test results are:
Gob is the best, bson, protobuf and JSON are the worst.
As mentioned above, the data to be operated is stored in memory in the form of struct. Naturally, we can’t load all the data. Usually, we need to maintain a data structure and phase out the data that hasn’t been used for a long time. Here using an LRU table can be done properly and conveniently.
An LRU implementation of go
When we get the data that needs to be eliminated, it is usually persisted to the database. We have encountered such a scenario online:
- After the server is updated and started, a large number of operations completed by players during the service suspension are processed uniformly (architecture, combat, scientific research, etc.) after starting, resulting in a large number of players loading into memory. At the same time, it triggers the Federation operation, resulting in a large number of federation data loaded into the memory of the corresponding service going to the city.
- Because most of the data to be persisted is an interface, which defines the same save interval (or LRU threshold).
- Half an hour later, a huge amount of players and alliance data loaded at the time of startup exceed the storage threshold and are filtered out by LRU algorithm.
- Concurrent serialization and DB operations cause mongodb to get stuck
Therefore, it is better to cooperate with current limiting or hash operation to prevent elimination of large amount of data at the same time.
Loss of memory data
When data is put in memory, it always gives people a sense of “unreliability”. At least many people mentioned it to me.
But what is the most important thing in the game business? I personally think it’s response speed and usability.
In fact, every six months or so, there will always be some virtual machine (damn alicloud) who will be killed innocently. It has been proved by practice that the disaster caused by crash of software and hardware can be avoided at the upper level through some conventional meansThe newer data in the memory is not persistent. Lose it：
- Use the appropriate strategy of saving the minimum amount of data (every player saves the data at least once every half an hour), plus the immediate saving of logout or some special operations (such as recharging), to ensure the correctness of most of the data.
- All the sensitive operations are recorded in the track log (saved by Kafka, etc.), and the lost operations are recovered from the log when necessary.
- Use the backup function of ECs to backup all databases in one hour interval.
- Multiple referenced data can only be saved (or arbitrated) in one place to prevent consistency problems caused by data corruption of some nodes (even if there is an error, do not conflict).
Dynamic read / write memory
The dynamic feature of go (in fact, reflect) brings us some convenience in runtime debugging. There are at least two ways to modify data without stopping the server:
- I realize a set of get / set / del operations to read and write the structure object in memory.
- By introducing Lua, the user-defined Lua script is dynamically loaded to read and write data in memory.
Since go has no reliable dynamic update scheme so far (1.11), we began to consider that in some operation business oriented services, we tried to write business logic with Lua, so that we could fix bugs or even update programs without stopping.