Building a blockchain with go — Part 7: Network

Time:2020-11-28

A series of translated articles I have put on GitHub:blockchain-tutorialAny subsequent updates will be on GitHub and may not be synchronized here. If you want to run the code directly, you can also run it in the SRC directory from the tutorial repository on clone GitHubmakeThat’s fine.


introduction

So far, our prototype has all the key features of blockchain: anonymous, secure, randomly generated address; blockchain data storage; workload proof system; reliable storage of transactions. Although these features are indispensable, there are still shortcomings. It is possible to make these features truly glowing and make cryptocurrency possibleNetwork。 What is the use of such a blockchain if it only runs on a single node? If there is only one user, what is the use of these cryptographic features? It is the network that makes the whole mechanism work and glow.

You can think of these blockchain features as rules, similar to the rules established by human beings living together and reproducing, a kind of social arrangement. Blockchain network is a program community, in which every program follows the same rules. It is because of the same rules that the network can survive forever. Similarly, when people have the same idea, they can clench their fists together and build a better life. If someone follows different rules, they will live in a divided community (state, commune, etc.). Similarly, if a blockchain node follows different rules, it will form a split network.

The point is: if there is no network, or most nodes do not follow the same rules, then the rules will be useless!

Announcement: Unfortunately, I don’t have enough time to implement a real P2P network prototype. In this article, I’ll show you one of the most common scenarios that involve different types of nodes. It is a good challenge and Practice for you to continue to improve this scenario and realize it as a P2P network! Except for the scenario in this article, I can’t guarantee that it will work properly in other scenarios. i ‘m sorry!

The code implementation of this article has changed a lot, please clickhereView all code changes.

Blockchain network

Blockchain networks are decentralized, which means that there are no servers and clients do not need to rely on servers to obtain or process data. In the blockchain network, there are some nodes, and each node is a full fledged member of the network. A node is everything: it’s both a client and a server. This needs to be kept in mind because it is very different from traditional web applications.

Blockchain network is a P2P (peer-to-peer) network, that is, nodes directly connect to other nodes. Its topology is flat because there is no hierarchy in the world of nodes. Here is a diagram of it

Building a blockchain with go -- Part 7: Network

Business vector created by Dooder – Freepik.com

It is more difficult to implement such a network node because they have to perform a lot of operations. Each node must interact with many other nodes. It must request the state of other nodes, compare it with its own state, and update when the state is over.

Node role

Although nodes have mature properties, they can also play different roles in the network. For example:

  1. miner
    Such nodes run on powerful or specialized hardware, such as ASIC, and their only goal is to dig out new blocks as quickly as possible. Miners are the only role in the blockchain that might use proof of work, because mining actually means solving the pow problem. In the blockchain of proof of interest POS, there is no mining.
  2. All nodes
    These nodes verify the validity of the blocks dug out by miners and confirm the transaction. To do this, they must have a full copy of the blockchain. At the same time, the whole node performs routing operations to help other nodes discover each other. For the network, it is very important to have enough full nodes. Because it is these nodes that perform the decision function: they determine the validity of a block or a transaction.
  3. SPV
    SPV stands for simplified payment verification. These nodes do not store copies of the entire blockchain, but can still validate transactions (though not all transactions are verified, but a subset of transactions, such as transactions sent to a specified address). An SPV node relies on a full node to obtain data, and there may be multiple SPV nodes connected to a full node. SPV makes wallet applications possible: a person doesn’t need to download the entire blockchain, but can still verify his transaction.

Network simplification

In order to implement the network in the current blockchain prototype, we have to simplify some things. Because we don’t have so many computers to simulate a multi node network. Of course, we can use virtual machines or dockers to solve this problem, but it will make everything more complicated: you will have to solve the virtual machine or docker problems that may arise first, and my goal is to focus on the blockchain implementation. So, we want to run multiple blockchain nodes on a single machine, and we want them to have different addresses. To achieve this, we will use thePort number as node identifierInstead of using an IP address, for example, a node that will have such an address:127.0.0.1:3000127.0.0.1:3001127.0.0.1:3002wait. We call it the port node ID and use the environment variableNODE_IDSet them up. Therefore, you can open multiple terminal windows and set differentNODE_IDRun different nodes.

This approach also requires different blockchain and wallet files. They must now rely on the node ID for naming, such as blockchain_ 3000.db, blockchain_ 30001.db and wallet_ 3000.db, wallet_ 30001.db, etc.

realization

So what happens when you download bitcoin core and run it for the first time? It must be connected to a node to download the latest state of the blockchain. Considering that your computer is not aware of all or part of the bitcoin nodes, what is the “one node” connected to?

Hard coding an address in bitcoin core has been proved to be an error: the node may be attacked or shut down, which will result in the new node unable to join the network. In bitcoin core, hard codedDNS seeds。 Although these are not nodes, the DNS server knows the addresses of some nodes. When you start a new bitcoin core, it will connect to a seed node, get the full node list, and then download the blockchain from these nodes.

However, in our current implementation, we can not achieve complete decentralization, because there will be centralization. We will have three nodes:

  1. A central node. All other nodes are connected to this node, which sends data between the other nodes.
  2. A miner node. This node will store new transactions in the memory pool, and when there are enough transactions, it will pack up and dig out a new block.
  3. A wallet node. This node will be used to send money between wallets. But unlike the SPV node, it stores a full copy of the blockchain.

scene

The goal of this paper is to achieve the following scenarios:

  1. The central node creates a blockchain.
  2. One other (wallet) node connects to the central node and downloads the blockchain.
  3. The other (miner) node connects to the central node and downloads the blockchain.
  4. The wallet node creates a transaction.
  5. The miner node receives the transaction and saves the transaction to the memory pool.
  6. When there are enough transactions in the memory pool, the miners start digging a new block.
  7. When a new block is dug out, it is sent to the central node.
  8. The wallet node is synchronized with the central node.
  9. The users of the wallet node check whether their payment is successful.

This is the general process in bitcoin. Although we will not implement a real P2P network, we will implement a real and most important user scenario for bitcoin.

edition

Nodes communicate through messages. When a new node starts running, it takes several nodes from a DNS seed and sends themversionMessage, in our implementation, looks like this:

type version struct {
    Version    int
    BestHeight int
    AddrFrom   string
}

Since we only have one version of blockchain, soVersionFields don’t actually store any important information.BestHeightStores the height of the node in the blockchain.AddFromStores the sender’s address.

ReceivedversionWhat should the node of the message do? It will respond to its ownversionNews. Is it a handshake?: there can be no other communication without greeting each other in advance. However, it is not polite:versionUsed to find a longer blockchain. When a node receivesversionMessage, which will check whether the blockchain of this node is better thanBestHeightThe value of is higher. If not, the node requests and downloads the missing block.

In order to receive messages, we need a server:

var nodeAddress string
var knownNodes = []string{"localhost:3000"}

func StartServer(nodeID, minerAddress string) {
    nodeAddress = fmt.Sprintf("localhost:%s", nodeID)
    miningAddress = minerAddress
    ln, err := net.Listen(protocol, nodeAddress)
    defer ln.Close()

    bc := NewBlockchain(nodeID)

    if nodeAddress != knownNodes[0] {
        sendVersion(knownNodes[0], bc)
    }

    for {
        conn, err := ln.Accept()
        go handleConnection(conn, bc)
    }
}

First, we hard code the address of the central node: because each node must know where to start initialization.minerAddressParameter specifies the address to receive the mining award. Code fragment:

if nodeAddress != knownNodes[0] {
    sendVersion(knownNodes[0], bc)
}

This means that if the current node is not a central node, it must send to the central nodeversionMessage to query if your own blockchain is out of date.

func sendVersion(addr string, bc *Blockchain) {
    bestHeight := bc.GetBestHeight()
    payload := gobEncode(version{nodeVersion, bestHeight, nodeAddress})

    request := append(commandToBytes("version"), payload...)

    sendData(addr, request)
}

Our messages, at the bottom, are byte sequences. The first 12 bytes specify the command name (such as hereversion)The following bytes containgobCoded message structure,commandToBytesIt looks like this:

func commandToBytes(command string) []byte {
    var bytes [commandLength]byte

    for i, c := range command {
        bytes[i] = byte(c)
    }

    return bytes[:]
}

It creates a buffer of 12 bytes and fills it with a null name. The following is the opposite function:

func bytesToCommand(bytes []byte) string {
    var command []byte

    for _, b := range bytes {
        if b != 0x0 {
            command = append(command, b)
        }
    }

    return fmt.Sprintf("%s", command)
}

When a node receives a command, it runsbytesToCommandTo extract the command name and select the correct processor to process the command body:

func handleConnection(conn net.Conn, bc *Blockchain) {
    request, err := ioutil.ReadAll(conn)
    command := bytesToCommand(request[:commandLength])
    fmt.Printf("Received %s command\n", command)

    switch command {
    ...
    case "version":
        handleVersion(request, bc)
    default:
        fmt.Println("Unknown command!")
    }

    conn.Close()
}

Here isversionCommand processor:

func handleVersion(request []byte, bc *Blockchain) {
    var buff bytes.Buffer
    var payload verzion

    buff.Write(request[commandLength:])
    dec := gob.NewDecoder(&buff)
    err := dec.Decode(&payload)

    myBestHeight := bc.GetBestHeight()
    foreignerBestHeight := payload.BestHeight

    if myBestHeight < foreignerBestHeight {
        sendGetBlocks(payload.AddrFrom)
    } else if myBestHeight > foreignerBestHeight {
        sendVersion(payload.AddrFrom, bc)
    }

    if !nodeIsKnown(payload.AddrFrom) {
        knownNodes = append(knownNodes, payload.AddrFrom)
    }
}

First, we need to decode the request and extract the valid information. All processors are similar in this section, so we will omit this part in the following code snippet.

The node then extracts theBestHeightCompare with yourself. If the blockchain of its own node is longer, it will replyversionMessage; otherwise, it will be sentgetblocksNews.

getblocks

type getblocks struct {
    AddrFrom string
}

getblocksShow me what blocks you have. Notice that instead of saying “give me all your blocks,” it requests a list of block hashes. This is to reduce the network load, because blocks can be downloaded from different nodes, and we don’t want to download tens of gigabytes of data from a single node.

The processing command is very simple:

func handleGetBlocks(request []byte, bc *Blockchain) {
    ...
    blocks := bc.GetBlockHashes()
    sendInv(payload.AddrFrom, "block", blocks)
}

In our simplified implementation, it returnsAll block hashes

inv

type inv struct {
    AddrFrom string
    Type     string
    Items    [][]byte
}

Bitcoin usageinvTo show other nodes what blocks and transactions the current node has. Again, it doesn’t contain a complete blockchain and transaction, just a hash.TypeField indicates whether this is a block or a transaction.

handleinvSlightly complicated:

func handleInv(request []byte, bc *Blockchain) {
    ...
    fmt.Printf("Recevied inventory with %d %s\n", len(payload.Items), payload.Type)

    if payload.Type == "block" {
        blocksInTransit = payload.Items

        blockHash := payload.Items[0]
        sendGetData(payload.AddrFrom, "block", blockHash)

        newInTransit := [][]byte{}
        for _, b := range blocksInTransit {
            if bytes.Compare(b, blockHash) != 0 {
                newInTransit = append(newInTransit, b)
            }
        }
        blocksInTransit = newInTransit
    }

    if payload.Type == "tx" {
        txID := payload.Items[0]

        if mempool[hex.EncodeToString(txID)].ID == nil {
            sendGetData(payload.AddrFrom, "tx", txID)
        }
    }
}

If we receive block hashes, we want to save them in theblocksInTransitVariable to track downloaded blocks. This allows us to download blocks from different nodes. When the block is placed in the transfer state, we give theinvSender sending of messagegetdataCommand and updateblocksInTransit。 In a real P2P network, we want to transfer blocks from different nodes.

In our implementation, we will never send multiple hashesinv。 That’s whypayload.Type == "tx"Only the first hash will be obtained. Then we check whether the hash already exists in the memory pool, and if not, send itgetdataNews.

getdata

type getdata struct {
    AddrFrom string
    Type     string
    ID       []byte
}

getdataA request for a block or transaction that can contain only the ID of a block or transaction.

func handleGetData(request []byte, bc *Blockchain) {
    ...
    if payload.Type == "block" {
        block, err := bc.GetBlock([]byte(payload.ID))

        sendBlock(payload.AddrFrom, &block)
    }

    if payload.Type == "tx" {
        txID := hex.EncodeToString(payload.ID)
        tx := mempool[txID]

        sendTx(payload.AddrFrom, &tx)
    }
}

This processor is relatively intuitive: if they request a block, they return the block; if they request a transaction, they return the transaction. Note that we don’t check to see if the block or transaction actually exists. This is a flaw:)

Block and TX

type block struct {
    AddrFrom string
    Block    []byte
}

type tx struct {
    AddFrom     string
    Transaction []byte
}

It is these messages that actually complete the data transfer.

handleblockThe message is simple:

func handleBlock(request []byte, bc *Blockchain) {
    ...

    blockData := payload.Block
    block := DeserializeBlock(blockData)

    fmt.Println("Recevied a new block!")
    bc.AddBlock(block)

    fmt.Printf("Added block %x\n", block.Hash)

    if len(blocksInTransit) > 0 {
        blockHash := blocksInTransit[0]
        sendGetData(payload.AddrFrom, "block", blockHash)

        blocksInTransit = blocksInTransit[1:]
    } else {
        UTXOSet := UTXOSet{bc}
        UTXOSet.Reindex()
    }
}

When a new block is received, we put it into the blockchain. If there are more blocks to download, we continue to request from the node of the last downloaded block. When all the blocks are finally downloaded, the utxo set is re indexed.

Todo: not unconditional trust, we should validate each block before adding it to the blockchain.

Todo: not running UTXOSet.Reindex Instead, you should use UTXOSet.Update (block), because if the blockchain is large, it will take a lot of time to re index the entire utxo set.

handletxNews is the most difficult part:

func handleTx(request []byte, bc *Blockchain) {
    ...
    txData := payload.Transaction
    tx := DeserializeTransaction(txData)
    mempool[hex.EncodeToString(tx.ID)] = tx

    if nodeAddress == knownNodes[0] {
        for _, node := range knownNodes {
            if node != nodeAddress && node != payload.AddFrom {
                sendInv(node, "tx", [][]byte{tx.ID})
            }
        }
    } else {
        if len(mempool) >= 2 && len(miningAddress) > 0 {
        MineTransactions:
            var txs []*Transaction

            for id := range mempool {
                tx := mempool[id]
                if bc.VerifyTransaction(&tx) {
                    txs = append(txs, &tx)
                }
            }

            if len(txs) == 0 {
                fmt.Println("All transactions are invalid! Waiting for new ones...")
                return
            }

            cbTx := NewCoinbaseTX(miningAddress, "")
            txs = append(txs, cbTx)

            newBlock := bc.MineBlock(txs)
            UTXOSet := UTXOSet{bc}
            UTXOSet.Reindex()

            fmt.Println("New block is mined!")

            for _, tx := range txs {
                txID := hex.EncodeToString(tx.ID)
                delete(mempool, txID)
            }

            for _, node := range knownNodes {
                if node != nodeAddress {
                    sendInv(node, "block", [][]byte{newBlock.Hash})
                }
            }

            if len(mempool) > 0 {
                goto MineTransactions
            }
        }
    }
}

The first thing to do is put the new transaction into the memory pool (again, it’s necessary to verify the transaction before putting it in the memory pool). Next clip:

if nodeAddress == knownNodes[0] {
    for _, node := range knownNodes {
        if node != nodeAddress && node != payload.AddFrom {
            sendInv(node, "tx", [][]byte{tx.ID})
        }
    }
}

Check whether the current node is the central node. In our implementation, the central node does not mine. It will only push new transactions to other nodes in the network.

The next big piece of code is the miner node “exclusive.”. Let’s decompose it as follows:

if len(mempool) >= 2 && len(miningAddress) > 0 {

miningAddressIt will only be set on the miner node. If there are two or more transactions in the memory pool of the current node (miner), start mining:

for id := range mempool {
    tx := mempool[id]
    if bc.VerifyTransaction(&tx) {
        txs = append(txs, &tx)
    }
}

if len(txs) == 0 {
    fmt.Println("All transactions are invalid! Waiting for new ones...")
    return
}

First, all transactions in the memory pool are validated. Invalid transactions are ignored and if there are no valid transactions, the mining is interrupted.

cbTx := NewCoinbaseTX(miningAddress, "")
txs = append(txs, cbTx)

newBlock := bc.MineBlock(txs)
UTXOSet := UTXOSet{bc}
UTXOSet.Reindex()

fmt.Println("New block is mined!")

The validated transaction is put into a block, along with a coinbase transaction with a bonus. When the blocks are dug out, the utxo set is re indexed.

Todo: a reminder that you should use UTXOSet.Update instead of UTXOSet.Reindex .

for _, tx := range txs {
    txID := hex.EncodeToString(tx.ID)
    delete(mempool, txID)
}

for _, node := range knownNodes {
    if node != nodeAddress {
        sendInv(node, "block", [][]byte{newBlock.Hash})
    }
}

if len(mempool) > 0 {
    goto MineTransactions
}

When a transaction is dug out, it is removed from the memory pool. All other nodes to which the current node is connected receive theinvNews. After processing the message, they can request the block.

result

Let’s review the scenarios defined above.

First, in the first terminal window, theNODE_IDSet to 3000(export NODE_ID=3000)。 In order to let you know what node performs what operation, I will useNODE 3000orNODE 3001Identify.

NODE 3000

Create a wallet and a new blockchain:

$ blockchain_go createblockchain -address CENTREAL_NODE

(I’ll use fake addresses for brevity.)

Then, a blockchain containing only the creation blocks is generated. We need to save the block and use it on other nodes. The genesis block acts as a chain identifier (in bitcoin core, the creation block is hard coded)

$ cp blockchain_3000.db blockchain_genesis.db 

NODE 3001

Next, open a new terminal window and set the node ID to 3001. This will act as a wallet node. adoptblockchain_go createwalletGenerate addresses that we call wallet_ 1, WALLET_ 2, WALLET_ Three

NODE 3000

Send some coins to the wallet address:

$ blockchain_go send -from CENTREAL_NODE -to WALLET_1 -amount 10 -mine
$ blockchain_go send -from CENTREAL_NODE -to WALLET_2 -amount 10 -mine

-mineThe flag is that the block will be immediately dug out by the same node. We have to have this sign, because in the initial state, there is no miner node in the network.

Start node:

$ blockchain_go startnode

This node will continue to run until the end of the scenario defined in this article.

NODE 3001

Start the blockchain on which the creation block node is saved

$ cp blockchain_genesis.db blockchain_3001.db

Running node:

$ blockchain_go startnode

It will download all blocks from the central node. To check that everything is OK, pause the node and check the balance:

$ blockchain_go getbalance -address WALLET_1
Balance of 'WALLET_1': 10

$ blockchain_go getbalance -address WALLET_2
Balance of 'WALLET_2': 10

You can also checkCENTRAL_NODEAddress balance, because node 3001 now has its own blockchain:

$ blockchain_go getbalance -address CENTRAL_NODE
Balance of 'CENTRAL_NODE': 10

NODE 3002

Open a new terminal window, set its ID to 3002, and then generate a wallet. This will be a miner node. Initialize blockchain:

$ cp blockchain_genesis.db blockchain_3002.db

Start node:

$ blockchain_go startnode -miner MINER_WALLET

NODE 3001

Send some currency:

$ blockchain_go send -from WALLET_1 -to WALLET_3 -amount 1
$ blockchain_go send -from WALLET_2 -to WALLET_4 -amount 1

NODE 3002

Quickly switch to the miner node and you’ll see a new block dug out! At the same time, check the output of the central node.

NODE 3001

Switch to the wallet node and start:

$ blockchain_go startnode

It will download the recently dug blocks!

Pause node and check balance:

$ blockchain_go getbalance -address WALLET_1
Balance of 'WALLET_1': 9

$ blockchain_go getbalance -address WALLET_2
Balance of 'WALLET_2': 9

$ blockchain_go getbalance -address WALLET_3
Balance of 'WALLET_3': 1

$ blockchain_go getbalance -address WALLET_4
Balance of 'WALLET_4': 1

$ blockchain_go getbalance -address MINER_WALLET
Balance of 'MINER_WALLET': 10

That’s all!

summary

This is the last article in this series. I could have continued with a real P2P prototype, but I really didn’t have that much time. I hope this article has answered some of the questions about bitcoin technology, as well as some questions for readers, which you can find out for yourself. There are many interesting things hidden in bitcoin technology! good luck!

Postscript: you can realizeaddrMessages to start improving the network, as described in the bitcoin network protocol (links can be found below). This is a very important message because it allows nodes to discover each other. I’ve started to implement it, but I haven’t finished yet!

Link:

  1. Source codes
  2. Bitcoin protocol documentation
  3. Bitcoin network

Original text:Building Blockchain in Go. Part 7: Network