Using golang to implement a JSON command line tool

Time:2020-7-26

Let’s start with a question,"abc"123perhaps[1, 2, 3]Is it a legitimate JSON?

JQ, a JSON command-line tool, has been used before. This tool is based on flex and bison (to understand these are based on the experience of learning PHP in those years). Later, I found a good lexical and grammatical analysis tool ANTLR, which supports the generation of multiple languages and provides basic grammar files for multiple languages. So I want to be able to implement a go language version of the JSON command-line tool based on it.

Now let’s go step by step (if you want to see the code directly, you can pull it to the bottom). I named this project asjtlr

Features provided

According to the scenarios I often use, I want to implement the following functions:

Basic usage:

jtlr ‘{“a”: 1}’

Interactive mode, which can be input multiple times, and it is better to support switching up and down:

jtlr -a

Read the content from standard input, and format the real-time output log:

tail -f xxx.log | jtlr -s

Read from file:

jtlr -f xxx.log

What is JSON

Before you start, you should have a comprehensive understanding of JSON. Let’s take a look at the beginning of the BNF paradigm of JSON provided on the official website

json
    element

value
    object
    array
    string
    number
    "true"
    "false"
    "null"

...

element
    ws value ws

wsWhitespace is the abbreviation of whitespace, that is, white space character. If you ignore this, you can see the valid value in JSON simply and clearly. Although our common JSON content is based on object, it doesn’t have to start with object, so do you have an answer to the question at the beginning of the article?

In the implementation, I did not copy the BNF provided by the official website, but adopted the syntax provided by antlr4. For its implementation, here is an article that explains: https://andreabergia.com/a-grammar-for-json-with-antlr-v4/ 。

In short, JSON has seven kinds of data, among whicharrayandobjectYes, it can be included againvalueThe remaining five are basic numerical data.

In addition, there is another kind of special situation, that is to saystringHow to use:

member
    ws string ws ':' element

stringIt can be a basic typevalue, or the key value of an object member. This will lead us tostringThere are two situations to consider when doing coloring and other treatments.

Interface provided by antlr4

Use the following command to generate lexer and parser based on go language:

antlr -Dlanguage=Go -o parser/ JSON.g4

The next step is to realize the function.

The interface generated by antlr4 is relatively complete, including the interfaces for each branch logic entry, exit and error node access. And it has a good error correction and prompt mechanism.

But for the case of JSON itself, we should pay attention tovalueandstring。 As mentioned above, all seven types of values arevalue, so it will triggerEnterValueandExitValueevent,stringIn the same way.

aboutobjectandarrayFor example, nested data is the most difficult problem

{“a”: [134, {“a”: 1}, true, [1, 2, 3], false]}

When using the interface provided by antlr4, it is necessary to mark the order of entry and exit.

Problems in interactive mode

At the beginning, I made a very simple implementation of interaction mode

reader := bufio.NewReader(os.Stdin)
for {
    fmt.Print(">>> ")
    text, err := reader.ReadString('\n')
    if err == io.EOF {
        break
    }
    if text == "\n" || text == "\r\n" {
        continue
    }
    fn(text)
}

However, in this logic, the buttons such as up, down, left and right will be printed directly on the screen and cannot be handled correctly because the terminal is in thecooked modeNext. Go itself does not provide TTY encapsulation. So you have to enterraw modeOne is to call the command line directly

func raw(start bool) error {
    r := "raw"
    if !start {
        r = "-raw"
    }

    rawMode := exec.Command("stty", r)
    rawMode.Stdin = os.Stdin
    err := rawMode.Run()
    if err != nil {
        return err
    }

    return rawMode.Wait()
}

The other is to operate stdin file handle, which is quite complicated to implement.

For the sake of compatibility and maintainability, I use the terminal package provided by golang / crypto, which is the only third-party package (if it is a third-party package) introduced in addition to ANTLR in the project.

But one problem with this package is that it has to be used\r\nEnter (the official issue explains some historical reasons), otherwise the cursor will not return to the beginning of the line, but it is used in the FMT package of go standard\nLine feed, and ANTLR uses FMT for error output, so the error output needs to be overloaded.

hang in the air

It takes about two weeks from the beginning of conception to the realization to the current stage.

Due to the laziness in the early stage, FMT is used for all formatted output, which needs to be optimized in the future.

The current implementation for ANTLR is a bit like killing a chicken with a knife. JQ supports node selection, which is a direction of subsequent implementation.

In addition, although go provides official JSON serialization and deserialization tools, there are also some third-party implementations on the market that are used. I also want to discuss the implementation methods.

In addition, there is no complete compatibility test under windows.

Finally, attach the project address: https://github.com/XiaoLer/jtlr-go

Recommended Today

Deeply analyze the principle and practice of RSA key

1、 Preface After experiencing many dark moments in life, when you read this article, you will regret and even be angry: why didn’t you write this article earlier?! Your darkest moments include: 1. Your project needs to be connected with the bank, and the other party needs you to provide an encryption certificate. You have […]