Code Engineered

Go Exposed: How Remote Import Paths Work

One of the confusing and not well documented aspects of Go is the way remote import paths work. That is, how do the naming conventions tell Go how to get from the right remote locations? In Glide we have chosen to keep compatibility with go get based names. This has turned out to be one of the areas we get the most questions and see the most confusion. If you’ve run into a problem you’re not alone.

Remote package names have a few steps to resolve the location and version control system. Go does support Git, SVN, Bzr, and Hg meaning the type of VCS matters.

First, Go (as of 1.6) looks at a list of hosts and patterns to detect the location and VCS. This list allows it to detect locations on:

  • Google Code (this should hopefully redirect somewhere else now that it’s deprecated but not always)
  • GitHub (e.g., github.com/foo/bar)
  • Bitbucket (e.g., bitbucket.org/foo/bar)
  • IBM DevOps Services (JazzHub) (e.g., hub.jazz.net/git/foo/bar)
  • Git on Apache
  • Imports ending in .git, .svn, .bzr, and .hg. That means example.com/foo.git or example.com/foo.git/bar will be detected.

During some of these phases, such as checking Bitbucket which can host repos of varying types, Go may reach out to a known API to find out the type.

If none of these work Go moves to a second detection phase. If the path is a URL it will append ?go-get=1 to the end and perform a GET request to it. It’s looking for an HTML (XML really) document containing a meta tag with a name of go-import. The value of this has the root package, VCS type, and VCS location to use.

For example, the import gopkg.in/yaml.v2 causes Go to do an HTTP get on https://gopkg.in/yaml.v2?go-get=1 which returns:

<html>
<head>
<meta name="go-import" content="gopkg.in/yaml.v2 git https://gopkg.in/yaml.v2">
<meta name="go-source" content="gopkg.in/yaml.v2 _ https://github.com/go-yaml/yaml/tree/v2{/dir} https://github.com/go-yaml/yaml/blob/v2{/dir}/{file}#L{line}">
</head>
<body>
go get gopkg.in/yaml.v2
</body>
</html>

Note, Go does some security checks on this. If you want to dig through the code you’ll see them.

From this any tooling can see it should fetch the package gopkg.in/yaml.v2 using git from https://gopkg.in/yaml.v2.

If you want a more interesting example, you can try golang.org/x/net/context. The import for that reads:

<meta name="go-import" content="golang.org/x/net git https://go.googlesource.com/net">

Here you can see that the root package golang.org/x/net (the repo) is available via git at https://go.googlesource.com/net.

Finally, if this fails there is one more phase to try and detect via other means. Right now it only tries to detect lanuchpad.net (e.g., imports like launchpad.net/foo/bar). This 3rd phase (instead of keeping this as part of the 1st phase) is to allow some to transition to meta tag support (step 2).

If you want to dig further into this detection you can start reading the repoRootForImportPath function inside Go.

Glide respects these same naming conventions to allow for clean transitions with go get.