Engineering Practice of Go Module (2): Principle of go get

Time:2019-3-11

Next: Go Module Engineering Practice (1): Basic Concept Chapter.

2. go getPacking Principle Chapter

Whether open or notGo ModuleFunction,go getSlave version control systemVCSThe underlying process of fetching packages in the new implementation is similar, except that the fetching is not recycled in the new implementation.submoduleOutside of the sub-module.

2.1 go getBasic Packing Procedure

Assuming dependency packagesgithub.com/liujianping/fooNot local, need to passgo getObtain. Launch the following order:

$: go get github.com/liujianping/foo

After the order is issued:

2.1.1 First step, the query path of dependent packages is matched regularly

go getSpecific packages can be specifiedimportPath or through its own analysis of code inimportGet the path to get the package. howeverimportPath is not directly the query path of the package. staygo getIn the implementation of source code, the query path of the package is matched by a set of regularities. In other words,importPaths must match this set of regular expressions, and if they do not, the code will certainly not compile. I will post the GitHub rule and the private warehouse rule in this set of regular expressions.

// Github
    {
        prefix: "github.com/",
        re:     `^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[\p{L}0-9_.\-]+)*$`,
        vcs:    "git",
        repo:   "https://{root}",
        check:  noVCSSuffix,
    },
    
    // Eliminate other VCS...
    
    // General syntax for any server. 
    // Must be last. Private warehouses will use this rule
    {
        re:   `^(?P<root>(?P<repo>([a-z0-9.\-]+\.)+[a-z0-9.\-]+(:[0-9]+)?(/~?[A-Za-z0-9_.\-]+)+?)\.(?P<vcs>bzr|fossil|git|hg|svn))(/~?[A-Za-z0-9_.\-]+)*$`,
        ping: true,
    },

Packet pathgithub.com/liujianping/fooFor example, after regular matching, the query path is as follows:

https://github.com/liujianping/foo

Recombinationgo-getParameter, initiated to remote VCS systemhttps://github.com/liujianping/foo?go-get=1Request.

Step 2.1.2: Find out the remote warehouse address of the package.

The remote warehouse address of the package can be accessed throughgo getIn response to a requestgo-importObtained from content in the meta tag of.

$: curl https://github.com/liujianping/foo?go-get=1 | grep go-import
<meta name="go-import" content="github.com/liujianping/foo git https://github.com/liujianping/foo.git">

The remote warehouse address corresponding to the package in the example is:https://github.com/liujianping/foo.git.

2.1.3 Step 3, Depending on the warehouse addresscloneTo local

Although version control systemVCSThere are various differences in itself, but some basic operations are mostly similar. staygo getMedium specificcloneThe process will be specific.VCSThe corresponding operation is adopted.

2.2 go getAgent Packing Procedure

Got it.go getAfter the basic process of picking up the package, let’s talk about it.Go ModuleComplete process after function opening.

openGo ModuleLater,go getA new environmental variable was added.GOPROXY。 Once the environment variable is opened,go getSwitch completely to the new package fetching process, i.e.GOPROXYTechnological processLet’s call it that for the time being.

stayGOPROXYTechnological processIn this article, a set of proxy interfaces are officially defined. Please refer to the official interface definition.

GET $GOPROXY/<module>/@v/list returns a list of all known versions of the given module, one per line.

GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata about that version of the given module.

GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file for that version of the given module.

GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive for that version of the given module.

In fact, the definition of this set of interfaces is$GOPATH/pkg/mod/cache/downloadThe file system in. That is to say, we can use the file system in this directory as a proxy directly, as follows:export GOPROXY=file:///$GOPATH/pkg/mod/cache/download/

aboutGOPROXYAgent services, there are many online implementations, the official also recommended a few. Each has its own problems, so we can only say so. Because, for some customized calls, such as:

  • Private warehouse rights
  • Mirror images of individual libraries are not accessible domestically, etc.

There is no perfect solution yet. But even so, we can build a set of standards within the enterprise according to the specific engineering needs.GO ModuleProcess. Specific plan, in the next articleEngineering Practice ChapterExplain in Chinese.

2.3 Common problems in the process of picking up packages from private warehouses

Most of the problems in the process of picking up packages in private warehouses are concentrated in the process of basic picking up packages. Specific anomalies may occur at any stage from 2.1.1 to 2.1.3. List the common problems and solutions.

2.3.1 Private WarehousecloneThe issue of permissions for phases

Usually, access to private warehouses is based on account privileges. For example,private.vcs.com/group/fooPacket paths, in the ___________go getIn the process, a regular match is made.https://private.vcs.com/group/foo.gitThe warehouse Path of VCS is assumed to be built by gitlab.

So ingit clone https://private.vcs.com/group/foo.gitIn the process, the system will remind the user to provide the username and login password. Each input is cumbersome.

SolutionThere are two:

  • Method 1:

increase$HOME/.gitconfigTo configure:

[url “ssh://[email protected]/MYORGANIZATION/”]
insteadOf = https://github.com/MYORGANIZA…

Replace the original HTTPS access mode with SSH mode.

  • Method two:

increase$HOME/.netrc:

machine github.com login YOU password APIKEY
Replace APIKEY with your own login KEY.

Although GitHub is used as an example, it is applicable to gitlab services. In fact, there is another solution, this solution can also solve the problem of 2.3.2, so it will be explained in the next section.

2.3.2 Private VCS Non-standard Path Problem

For historical reasons, the gitlab service address of the author’s company is a non-standard path. The standard path should be:https://private.vcs.comThe gitlab path of my company is:https://private.vcs.com:888.

If pressedgo getProcess, import package path should adopt d:private.vcs.com:888/group/fooThen the reasonable address of the warehouse can be matched correctly. But unfortunately, in practice, the failure ended. Readers can test for specific reasons.

The only way to do this is to build an intermediate service:https://private.vcs.comBe able to passgo getPacket path matching queries for the correct warehouse address.

In fact, the implementation of the intermediate server is very simple. Specific implementation, the author left to the next side:Engineering Practice ChapterExplain.