Debating and discussing Go package management has become a popular topic. Most people agree that the out of the box package management is insufficient. But, what should be used to complement the Go toolchain and what changes, if any, should the Go toolchain make?
Package management for programming languages is nothing new. It’s worth learning and discussing how package management is handled by other languages. This can fuel our discussions and decision making. I’ve personally been studying this as part of my work on Glide.
The following is a look at some features in other package managers. For this analysis I looked at Cargo (Rust), Composer (PHP), npm (Node.js), Bundler (Ruby), Nuget (.NET), CocoaPods (Swift & Objective-C Cocoa Apps), Pip (Python), and Maven (Java). These are 8 different package managers for some of the most popular languages and platforms.
Note, you may be saying there are apples and oranges in here with things like Rust being compared to .NET. For practical purposes and the concepts of package managers it turns out not to matter. Read on to understand why.
It’s important to look at how others do things. If there are common patterns we should ask why. When people move from these languages to Go they already have expectations. From a developer experience standpoint, if the Go community does something differently it should be able to explain how it works (so users don’t stumble) and why this way is better than the common pattern done by others. And, the use cases solved by the other patterns need to have a pattern in Go that works well.
Manifest File
Each of these package managers has a manifest file to list dependencies. They come in a variety of formats, including json, xml, and others. Even systems like NuGet where you may typically use a UI are backed by a configuration file.
This is one of those cases where I’m curious if there is a modern package manager that doesn’t use a manifest file.
Release Versions
Version numbers have long been used for releases. That includes packages. Each of these package managers leverages versions including several cases that require a version to be set (that could be a branch for dev versions). The idea is simple, you usually want to retrieve a known release instead of something in development.
Since they all support versions it’s more interesting to look at how they do that.
Semantic Version
Semantic Versions (SemVer) has risen in popularity in recent years. With it numerous newer package managers leverage that as their version numbering scheme. That includes Cargo and npm.
Some languages recommend SemVer explicitly like Bundler (and RubyGems) and CocoaPods but don’t enforce it, other languages recommend you use SemVer or SemVer like patterns without calling it out by name like Composer, and some other languages use patterns similar to SemVer in their docs like pip and Maven.
NuGet is the odd one out that uses a slightly different numbering system. While a little different it has the same concepts incorporated in different ways.
Required Versions
Some of the specs require a version to be specified, even if it’s simply a branch name to track. It turns out the package managers are pretty split on this. Cargo and Bundler are examples where the version is optional. Composer and npm are examples where one is generally required.
Version Ranges
When specifying a version you may want to use a version within a range. For example, in the range >= 1.2.3, < 2.0.0. If you follow SemVer it means the API is compatible within that range, the feature you need that was added during the major version release cycle is there, and a bug you need fixed is fixed.
All of these package managers support specifying version ranges.
Central Registry
A number of these ecosystems have a central repository (in fact all that I looked at here) where you can see packages and install them from. The directory isn’t directly tied to the package manager but in many cases (but not all) they are developed by the same people.
The interesting difference to look at is how they operate. There are two types of central registry. Ones that contain artifacts to be downloaded and ones that are purely metadata. The metadata ones point to other locations, such as GitHub, to get the projects. It’s done in a manner than can be completely automated.
Nuget and Maven are examples of systems that have central artifact repositories. Cargo and Composer are examples of systems providing metadata and pointers to their home location.
It’s also worth noting that these tools don’t force you to use open publicly available libraries. The tooling allows you to easily work with alternative locations including protected source control systems.
Locking To Version
When you can have versions specified as ranges there are opportunities to get into some trouble. For example, in your development environment you used one version of a package. But, a newer version came out between pushing your code to a CI/CD system, the application is built with the new version (because CI/CD manages external dependencies), and now your application deploys with the new version of the library. But wait, there’s an issue and now production reflects that.
This isn’t an uncommon case. To overcome this and others is the idea of locking to a specific version as opposed to a range. These often live in files such as Cargo.lock or composer.lock and are managed by the tooling.
Each of these toolchains offer the ability to lock or pin to a specific version. It happens in different ways depending on the tool. But, each tool provides a way to do this.
Recommends Storing Outside Packages In Your VCS
Should other people’s libraries (packages) be stored alongside your application or library code in your repo? This topic doesn’t normally come up but due to the way the Go ecosystem (inspired by practices at Google) it’s worth looking at.
I’ll start with the summary from Composer which reads,
The general recommendation is no. The vendor directory (or wherever your dependencies are installed) should be added to .gitignore/svn:ignore/etc.
The best practice is to then have all the developers use Composer to install the dependencies. Similarly, the build server, CI, deployment tools etc should be adapted to run Composer as part of their project bootstrapping.
Do others follow this? Cargo was another interesting example I looked at. When it creates a project the target directory containing things like dependencies is ignored. The dependencies in there aren’t the source but instead a compiled format. So, the library source isn’t even easily available to add to your projects repo.
Composer and Cargo aren’t the only examples like this. npm and pip are also examples. Most of these recommend not checking the code into your repo but don’t stop you from doing so.
CocoaPods is an example of one that recommends storing external packages in version control.
Not every package manager has a documented standard practice for this.
Recursion
An application getting its dependencies isn’t enough. Instead, the entire dependency graph needs to be handled. The dependencies of your dependencies and the dependencies of your dependencies dependencies need to be installed. Recursion goes all the way down.
Each of these systems handles fetching recursive dependencies. This generally happens in two ways and I’ll highlight two examples.
- Where the dependencies of each library are installed. For example, this is how npm operates. Each of an applications or libraries dependencies are stored in their own location. That means it usually handles multiple versions well.
- When dependencies are flattened to the top level. This works in conjunction with version ranges. If multiple versions are specified hopefully they are ranges. Then the tooling can find the version that best meets the multiple requirements and install that.
In any case, recursion just works.
What To Do With This?
My hope is that this can spark some discussion on what should be in the Go community and why it should be there. Along with that, knowing how package management worked in the languages people came from can help us to understand their needs, devise ways to meet them, and help them ease into the right Go tools for them.
Note, this was a quick and dirty analysis. If something here should be updated please let me know so I can more accurately reflect the state of these.