“Those who don’t know history are doomed to repeat it.” ― Edmund Burke

There are many variations on this saying but the essential element is that it’s important, and I would argue useful, to know the history of something. It can provide depth, understanding, and insight.

At the moment there are many debates going on about package management in Go. There are questions being asked, such as how should it work or whose ideas should we follow?

To give context to these ideas it’s worth looking at the history of dependency management in Go. It’s a story of people, discovery, differences, disconnects, and even a little drama.

In The Beginning…

In the beginning there was go get and the GOPATH.

The GOPATH is one workspace where all your source lives. It comes from the way Google does source control where most of their source is in a monorepo. The GOPATH can essentially point to a monorepo location.

go get would pull, or update with the -u flag, from the tip of source control for a remote location. It was like cp but knew how to talk to remote source control systems.

There were no versions. It was built around how Google handles source and dependencies. Go is a Google owned and run project so a system built around their style is where this all started.

Godep

The first major community tool was godep. Early on it provided a way to snapshot the VCS revisions you were using in your GOPATH and then restore those to the GOPATH. This provided a way for different applications to use different revisions of the same dependencies.

Godep did have some manual steps you had to do when switching between applications. For example, you needed to restore the revisions of dependencies for that application to the GOPATH. But, it would work alongside the Google workflow so it worked.

There were some other tools lurking around about the same time but none of them gained the traction of godep.

The Vendor Directory

After awhile there were several tools trying to solve dependency management. A common theme for these projects was the desire for different applications to use different versions of the same dependency.

With that we got the vendor directory. Prior to this the Go toolchain would look in the GOPATH and GOROOT, where the standard library lives, for a package. With the vendor directory there was a new discovery point relative to an imports current directory.

This next step was safely layered in. It worked for those at Google because they could use it or skip it and safely keep their existing workflows. It worked for the community because any dependency manager could put the dependencies in the vendor directories and things just worked with the toolchain.

Dependency Management Like Everyone Else

While godep was popular, some other tools came along that received a significant amount of usage. For example, there was a project I worked on named glide. The idea with Glide was to provide package management similar to how it worked in JavaScript, Python, Rust, PHP, and pretty much all of the languages with modern package managers.

Glide didn’t stand alone. There were numerous package managers that had market share. Things were getting splintered.

From Many To One

How can we solve a problem of too many package managers? Form a committee, right?

A committee was formed to figure out package dependency management for Go. Andrew Gerrand, from the Go team at Google, was a member of this committee. There was a second advisory group, which I was a member of, that provided them with insight.

To support the committee a couple of other activities kicked off.

1. A survey of the Go community about package management. This asked questions about Go, package management, and package managers. It even queried people on their experience with package management in other languages.
2. Interviews with people at companies who used Go. The idea was to understand their needs if they were to build applications around Go.

The committee came to some conclusions on what we needed based on the information from the community and some debate. They wrote up what was needed in a specification.

dep was built to have one new tool that met the needs outlined in the spec. Some of us who worked on tools like godep and glide put our support behind this effort.

The community had come up with a solution that was a single point to collaborate on continuity solving the problem. That solution worked with the existing vendor setup and could be layered in alongside the workflows of people at Google.

The expectation was that dep, a community initiative with the support of the Go team, would become the standard manager.

But, this is not the end of the story.

Enter vgo

At the GopherCon in Denver in 2017 there was a contributor summit. It was a place where some people could get together to discuss Go the day prior to GopherCon.

At this summit there was time set aside for people to discuss dependency management and we did for some time. Around the conclusion of that time Russ Cox, the current lead of the Go team, came to the table. He made a comment that he could do better if he went off on his own and built something. That something was later announced as vgo. It is the thing he went off on his own, apart from the community, and created.

Despite community objections, vgo was recently accepted as the path forward.

My 2 Cents

This is where we stand now. It’s interesting because of the social dynamics.

dep and the spec that started it was based on the community learning its own needs and collaborating on a solution. vgo was built when the Go lead at Google went off on his own to build a solution.

Practical issues with vgo were discussed, at length, and most of them were dismissed. A common reason for that was two assumptions of the Go team:

1. The community of package developers will need to change how they do things from today
2. Packages will always maintain SemVer compatibility

For the first point to come to reality in large projects like Kubernetes or Docker there will be a significant amount of work. The dependencies they use will need to make changes or be replaced and the way they currently do dependency management will need to change. There’s potential for a lot of work.

When there’s a lot of work at stake around a refactor it raises questions about cost, loss of velocity, and priority.

Is the second point, about SemVer, possible? Many language package management systems have been using SemVer for years and breaking from SemVer, sometimes by accident, happens. People aren’t perfect and disagree on how to do things.

The biggest take away may be the gap between the Go team at Google and the community. The community’s solution was rejected. The Go team created a solution in isolation from the community and mostly rejected the community feedback on it. This feedback came from people who weren’t just members of the community but experts in complex projects and, in some cases, dependency management.

At the heart of this may be two things. Where the community sits relative to the language and toolchain and the way Google does things different from the majority, for example a monorepo vs multi-repo.

I want to give a special thanks to Matt Butcher who reviewed and helped me work through this post.