[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

Time:2019-12-2

Welcome to Chapter 7. After the introduction of Chapter 6: how to deploy multi node cluster in a single machine and the design idea of mining pool in the previous chapter, we share the knowledge of how to deploy multi node cluster in a single machine and some ideas of mining pool design.

We will introduce it in this chapterReplication proof (porep) and spatiotemporal proof (post) in the implementation of filecoin project, and how to participate in the replication game initiated by protocol lab.

I. proof type designed by filecoin

Filecoin is a decentralized storage system realized by using blockchain technology. We all know that in a blockchain system, it is necessary to ensure that each node operates fairly and orderly according to the rules, can resist malicious attacks, and ensure the credibility and security of the whole system.

Therefore, under the filecoin system, a rigorous means of proof is also needed to ensure that the miner will not falsely claim that he has stored the user’s data without storing the data, and that the miner needs to compete for the block qualification by virtue of their storage capacity.

The proof algorithm in filecoin system originally originated from storage proof (POS), data holding certificate (PDP) and retrievable proof (Poret). Later, it gradually iterated and enhanced the constraint conditions to perfect the replication proof (porep) and spatiotemporal proof (post) used in today’s engineering. Here are their meanings in detail:

  • Proof of storage (POS): proof mechanism for storage space.
  • PDP: the user sends the data to the miner for storage. The miner proves that the data has been stored by himself. The user can repeatedly check whether the miner is still storing his data.
  • Proof of retrievability (Poret): similar to PDP process, prove that data stored by miners can be used for query.
  • Proof of replication (porp): a practical scheme for storing proof POS to prove that data is independently stored by miners, which can prevent witch attack, exogenous attack and generative attack.
  • Proof of space (pospace): proof of storage. Pospace is a kind of pow. The difference is that pow uses computing resources, while pospace uses storage resources.
  • Proof of space time (post): prove that in a period of time, miners actually store specific data on their own storage equipment.

As shown in the figure below,The definitions of these six kinds of proofs are not mutually exclusive and independent. POS includes PDP, Poret, porep, and pospace. And porep and post are two instances of pospace, and their definitions cross each other:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

The configuration of filecoin miner is determined by the implementation of replication proof and spatiotemporal proof. Indirectly determine the overall cost of the filecoin system. Filecoin provides storage and data download services, and the system cost ultimately determines the user’s use cost.

If too many resources are consumed by replication proof and spatiotemporal proof, the whole cost of filecoin will be systematically increased, which will greatly reduce the value of filecoin system.

The first version of the current open sourcego-filecoin0.1.xThe VDF mode of zigzagdrg and stackdrg is adopted in the series as the implementation of porp. The official thinks that there is still room for improvement. The protocol lab has set up RFPs fund for this purpose to study the subject,And the birth of the copy game is to better let community fans participate in the test in advance and assist the official optimization of this part.

II. Proof of reproduction

2.1 porp function

The function of porp algorithm is to prove that a storage system does store a copy of data, and each copy uses different physical storage, and to resist three common attacks in decentralized system:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

2.2 essence of porp

In essence, porp is an algorithm with long encryption time, short decryption time and efficient proof and verification process,In academic circles, this process is called verifiable time-delay encoding function:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

As shown in the figure above, we assume that the verification time of this encryption algorithm is twice, the decryption time is about 2-5 times, and the challenge effective time is 10 times, so this encryption time will take about 1000 times to achieve the relative security of 99.9% in probability.

2.3 challenge and proof model

The role and process of filecoin proof mechanism can be abstracted as follows: Challenger, certifier and verifier. They can be miners, users or any other role in the network. The definitions involved include the following:

  • Challenge: the system will ask the miner a question, which may be a question or a series of questions. If the miner answers correctly, the challenge will succeed, otherwise it will fail.
  • Prover: usually only miners. Providing the system with proof of completing the challenges initiated by the system.
  • Verifier: the party who challenges the miners to see if they have completed the data storage task.
  • Data: the data that the user submits to the miner for storage or that the miner has stored.
  • Proof: the miner’s answer to the challenge.

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

As shown in the figure above, the verification process can be expressed as follows: the verifier will challenge the miner according to certain rules, the challenge is randomly generated, and the miner cannot know in advance. As the certifier, the miner submits the certification to the inspector. The generation of the certification needs the original data and random challenge information. After the proof is generated, the certifier will give it to the verifier, and the verifier will determine whether the proof is valid. If it is, the challenge will be successful.

2.4 verifiable delay encryption function (VDF)

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

For VDF, filecoin was initially inspired by cipher block chaining, so as to improve and optimize its own VDF mode.

Let’s first look at the principle of CBC: large files are divided into D1-D4

Except for C1 initialization vector, XOR operation and AES encryption are required for all subsequent file block coding. For example, C2 is generated jointly by C1 and D2, and so on. In this way, the encoding process can not be parallel, so the speed becomes slower.

Looking at the process on the right, it is a further optimization of CBC algorithm. The depth robust chain is adopted, and the directed acyclic graph is used on the block to further compress the complexity of decoding verification and improve the randomness.

Filecoin’s current engineering implementation is based on the improved version of the previous two algorithms: zigzagdrg algorithm, as shown in the following figure:
[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

The original data data is divided into small data (d1-d5) one by one. Each small data will be calculated a hash value (32 bytes), and the hash value itself will be used as the encryption seed for encoding and decoding. The hash values of these small data are connected according to DRG (depth robot graph).

The hash relationship of data blocks will form the Merkle tree structure (the index of MySQL like database is implemented by B + tree).

This is for the sake of fast verification and better resistance to attack without decoding all data blocks when challenging and verifying.

For example, the root of the tree is the hash of the replica. The system or the user randomly initiates the challenge at D5. The miner only needs to calculate the path from D5 to the root of the root node and output a verification node to prove the challenge.

As for unit encryption of each data block in VDF, as shown in the figure below, bls12-381 is applied for unit coding and decoding:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

(source path: https://github.com/filecoin-p…)

Bls12-381 is a new encryption algorithm of zk-snark elliptic curve used in zcash. It belongs to Bellman library and is implemented by rust language. It is small and easy to use and can be verified quickly. Bellman’s goal is to make it easier for ordinary programmers to use ZK snarks.

Because hash functions need to be adapted to snarks, they currently use Pedersen in zcash (blake2 and sha256 are also implementing and selecting):

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

The calculation process of the whole porep is divided into several layers(currently, it is set to level 4 in filecoin and level 10 in copy game project), the arrow direction of DRG relationship of each layer is mutually exclusive, the upper layer is right, and the lower layer is left, so it is named zigzag (Z-type). In the process of data decoding, each layer is independent of each other and can be executed in parallel, which is faster than serial coding.

To sum up, it realizes the essence of porp: fast coding, fast decoding and verification, so as to prevent all kinds of attacks.

III. proof of time and space

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

As shown in the figure, post can be understood as the process of miners continuously generating replication proof and accepting challenge and verification within a certain period of time, and through this process, updating the whole network storage computing power.

The Challenger inputs a random challenge parameter C in the I-round repeated execution of the porep cycle, and then the challenge parameter C will be calculated by chain recursion, that is, the last output will be taken as the next input, and the last result will be taken as the proof of the post in the T-Time, and will accept the reverse verification.

Obviously, the improper miners can’t launch C back without the honest implementation of the post.

As shown in the figure below, at present, the threshold defined in go filecoin is that every 20000 blocks (average about 6 days), the storage miner must provide proof of proof of space time, indicating that there is still proof of user data. At the same time, the storage market (God) will verify the proof of post every 100 blocks (50 minutes on average) to determine whether the punishment needs to be issued.

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

(source path: https://github.com/filecoin-p…)

IV. filecoin living subsystem (FPS)

The file chain living subsystem (FPS) is the engineering implementation of all the proving algorithms in the file chain system. Because the underlying layer relies on the bellman library, it is completely written by rule. The source warehouse address: https://github.com/file chain-p

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

In the design of FPS, we pay great attention to decoupling: filecoin proofs implements the interface to provide storage proof for go filecoin, and relies on the other two modules: storage Proofs (storage proof generation and verification module) and sector base (sector control module).

These two modules rely on storage backend middleware to realize storage control and message forwarding.

For other details of FPS, I recommend an article by senior Li Xing:

  • Star idea: source code guide of filecoin – porep and post algorithm

After that, you can cooperate with the official trust fil proofs source code warehouse to read and obtain the latest changes:

  • github:rust-fil-proofs

As for the deduction and demonstration process of porep and post, the children’s boots that you want to know more about can be analyzed with reference to these two papers:

  • PoReps: Proofs of Space on Useful Data
  • Ben Fisch:Tight Proofs of Space and Replication

V. participate in the copy proof game

The copy game is a copy competition. Participants challenge the default copy proof algorithm (VDF mentioned above) provided by filecoin to see if they can provide better algorithm or execution results.

The way to participate in the game is to pass the copy algorithm (running or refactoring FPS) provided by the current filecoin, and send the execution result to the filecoin server.

5.1 compile game client

As follows, the preparation environment depends on the Linux system. The Mac OS environment is similar. Brew package management tool can be used instead of apt.

// install Rust
curl https://sh.rustup.rs -sSf | sh

Restart terminal

//Switch rust to nightly
rustup install nightly

//If you want to deploy the game server by yourself, you need to install the PostgreSQL database
brew install [email protected]

apt-get install postgresql

//Install clang and libclang 
apt-get install clang

//Install PQ Library
apt-get install libpq-dev

//Download the replication game source code
git clone https://github.com/filecoin-project/replication-game.git

cd replication-game

//Perform compilation
cargo +nightly build --release --bin replication-game

After compiling, you can see the play executable in the bin / Directory:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

5.2 start the game

#Start command:
# bin/play NAME SIZE TYPE

# E.g.

# Zigzag 10MiB
bin/play NAME 10240 zigzag

# Zigzag 1GiB
bin/play NAME 1048576 zigzag

# DrgPoRep 10MiB
bin/play NAME 10240 drgporep

# DrgPoRep 1GiB
bin/play NAME 1048576 drgporep
  • Name: your gamer name
  • Size: the size of the file you plan to copy, in kilobytes
  • Type: the name of the algorithm you want to run (currently, the optional values are zigzag and drgporep)

The play script will automatically download the seed from the game server, copy the data, generate the evidence, and then publish the evidence to the game server.

5.3 send game results to rank server

The play script will submit the game results through curl:

curl -X POST -H "Content-Type: application/json" -d @./proof.json https://replication-game.herokuapp.com/api/proof

We can view the ranking by visiting the rank page: http://replication-game.herok.

Model configuration of small Edition:

  • Processor Name: Intel Core i5
  • Processor Speed: 3.1 GHz
  • Number of Processors: 1
  • Total Number of Cores: 2
  • L2 Cache (per Core): 256 KB
  • L3 Cache: 4 MB
  • Memory: 8 GB

2.1s/mib for DRG replication proof:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

Zigzag replication proof took 11.2s/mib:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

Our game results will be recorded in proof.json. You can manually open proof.json and analyze the data structure of the following proof:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

  • Prover: verifier.
  • Seed: game seed, encryption key.
  • Proof params: proves the configuration item.
  • Proof: prove the data block relationship.
  • Tau: the root of one or more Merkle trees is called tau, the input of each layer is called D (data), and the result of VDE of each layer is called R (Replica).
  • Comm_d: the input of each layer builds a Merkel tree root of comm_d.
  • Comm_r: the output of each layer builds a Merkel tree root of comm_r.
  • Comm r Star: the result of hash calculation of comm R data and replica ID data of each layer.

By consulting the data structure of proof.json, we can also help you better understand the contents of 2.4.

5.4 how to raise the rank position?

  • Optimize from hardware and software: increase hardware configuration, faster CPU and larger ram, or use other replacement schemes such as FPGA, GPU, ASICs and other hardware that is better at deep computing. In addition, you can optimize some parameters of your operating system, such as IO parameters.
  • From the aspect of algorithm optimization: instead of using the default implementation algorithm provided by filecoin, you can design a new algorithm that can generate storage proof faster, such as breaking the order hypothesis, generating proof that stores less data, breaking Pedersen hash, etc.

Of course, the pursuit of rank data is meaningless. The delay of porep and post is only a factor affecting the storage efficiency under the filecoin system. For miners, what they need to pursue is comprehensive cost performance.

Reference

  • https://github.com/filecoin-project/replication-game
  • https://z.cash/blog/bellman-zksnarks-in-rust/

Review of previous series:

Chapter 1: build a filecoin test node
Chapter 2: how to create account wallet and obtain fil mock token
Chapter 3 (I): configuration operation of storage provider (miner)

Chapter 3 (2): configuration operation of storage provider (miner)

Chapter 4: configuration operation of storage demander (user)

Chapter 5: searching market and miners

Chapter 6: how to deploy a multi node cluster on a single machine

Sponsored brands in this chapter:

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

Guangdong Xinglan blockchain Technology Co., Ltd. has gathered a group of “pioneers” who aspire to IPFs ecological construction, and is also the earliest and most professional IPFs ecological preaching and communication community in China. The company has the most mature and complete industrial supply chain, including all-round three-dimensional mining machines, mines, mining pools, cooperative custody, data support, professional operation and maintenance, application development, knowledge service, etc Service enterprises provide the most comprehensive and professional support for IPFs ecology. As the first batch of preachers of domestic IPFs ecology, the team of “star blue” is very optimistic about the future value of IPFs, and will make every effort to promote the development and application of IPFs. Our team is in the front line of the blockchain revolution and is committed to IPFs ecological construction. We firmly believe that IPFs will bring better experience to the world and create greater value for business. Core values of star blue: co construction, win-win and sharing, block chain with frame, star blue boundless, star blue product “timebook” looks forward to your attention!

Thanks to Guangdong Xinglan blockchain Technology Co., Ltd. (www.xlipfs. Com)Jiale SOHOSupport for original content.

Contact author:

I have limited experience in this field, but I have some shortcomings. Please correct me and discuss more. I can do it in privateWeChat public address: jialesoho, or add meWechat: daijiale6239, if you find it helpful, you canHelp Click to promoteandReward and supportOh, thank you so much!

[full analysis of filecoin source warehouse] Chapter 7: learn about porep and post and participate in the copy proof game

(identify 2D code in the picture, pay attention to Jiale SOHO WeChat public number)