Perseus

With the introduction of Go modules, our projects' dependency trees have exploded in complexity. Perseus is the hero we need to battle that Kraken.

Overview

At CrowdStrike, the move to Go Modules from our existing GOPATH-mode monorepo has brought with it some pain points, especially around tracing descendant dependencies. In the GOPATH days, when our monorepo lived under $GOPATH/src and all engineers always had a full copy of the entire codebase, it was a straightforward grep command to find all imports of a given package to see what other packages depend on it.

Now that we've moved the majority of our development effort to modules, neither of those two conditions are true. Engineers are no longer required to have the code under $GOPATH/src and almost no one has the entire codebase locally anymore. We have dozens of functional teams working on hundreds of microservices, so most developers have pared down their local workspace to only the things they're directly working on on a day-to-day basis.

An unfortunate side effect of this paradigm shift has been that there is no longer a direct way to see which other modules depend on your work. Existing tools like go mod graph and go list -m all will show you what modules you depend on - with some rough edges - and the pkg.go.dev site has an Imported By view that shows what other packages depend on your code. The go tool won't show which things depend on you, though. The pkg.go.dev site can only show things that it knows about, so it won't help for private modules, and it doesn't show you which versions of those other packages depend on your code.

See CHANGELOG.md for a detailed history of changes.

Existing Tooling

Unfortunately, the go CLI commands, the pkg.go.dev site, and OSS tools like goda and godepgraph don't quite cover the ground we need, specifically querying for downstream dependencies. The go CLI, goda and godepgraph all do an excellent job of surfacing up which modules your code depends on in multiple ways. The pkg.go.dev site also provides a nice Imported By view, but it shows which packages depend on your code, not which modules, and doesn't include the version(s) of those dependents.

Perseus To The Rescue

The Database

For simplicity, Perseus uses a PostgreSQL database rather than an actual graph database like Neo4J, Cayley, and the like. After some initial investigation, we found that the relatively small number of entries (compared to other graph datasets) didn't warrant a specialized graph database. Additionally, our IT organization already has all of the necessary infrastructure in place to support PostgreSQL.

One potentially signficant caveat, however, is that Amazon RDS currently does not support the pg-semver extension that we use for storing a module's semantic version.

The Service

Service Architecture

The perseus service is built on Connect to provide an API that supports binary gRPC and HTTP JSON/REST requests. Both RPC bindings are provided on a single port using Vanguard, with the JSON/REST endpoints at a nested path of /api/v1/*. Additionally, a basic web-based user interface is available at /ui.

In addition to the interactive endpoints, the service also exports an HTTP health check at /healthz and basic Prometheus metrics at /metrics. For debugging and troubleshooting, the service supports retrieving Go pprof data via HTTP at /debug/pprof*.

Running the Service

For simplicity, we publish a pre-built Docker image (based on a scratch base) to the GitHub Container Registry when each release is tagged. The image runs perseus server and exposes port 31138, which you can map to whatever is appropriate for your environment. You will need to provide the URL location of the Perseus database, along with a valid username and password to connect, via environment variables. The default database name is "perseus", but you can override that by also providing a DB_NAME environment variable.

> docker run --rm -e DB_ADDR=... -e DB_USER=... -e DBPASS=... -p 31138:31138 ghcr.io/crowdstrike/perseus:latest

This example uses the latest tag but you should always reference a specific version for stability.

We also generate pre-built binaries for Windows, Linux, and Mac that can be downloaded from the releases page.

The perseus CLI

The perseus binary is both the server and a CLI tool for interacting with it. The update and query commands are the workhorses as a client. The Perseus server address can be specified by either setting the PERSEUS_SERVER_ADDR environment variable or passing it directly to the CLI using the --server-addr flag.

perseus update analyzes a Go module, on disk or available via public Go module proxies, and adds it to the Perseus graph. For a module on disk that is a Git repository, the CLI will try to infer the version by looking at the Git tags on the current commit. If there is exactly 1 module version tag, that version will be used. If there is no semver tag or there are multiple, you will need to specify a version.

# process the current version of the perseus module on disk
> perseus update --path ~/code/github.com/CrowdStrike/perseus

# process a specific version of example/foo
> perseus update --path ~/code/github.com/example/foo --version v1.2.3

For public modules a version must always be specified.

> perseus update --module github.com/example/foo --version v1.2.3

Once you have data in your graph, perseus query is the way to retrieve it. There are 4 available sub-commands: list-modules, list-module-versions, ancestors, and descendants.

The first two commands return modules and versions based on glob pattern matches:

# list all modules under the github.com/example organization along with the highest version
> perseus query list-modules 'github.com/example/*' --list
Module                  Version
github.com/example/foo  v1.2.0
github.com/example/bar  v1.1.0
github.com/example/baz  v1.17.23

# list all versions of github.com/example/foo using an explicit format
> perseus query list-module-versions github.com/example/foo --format 'module {{.Path}} has version {{.Version}}.'
module github.com/example/foo has version v1.2.0
module github.com/example/foo has version v1.1.0
...

The ancestors and descendants commands walk the graph to return dependency trees, either what a specified version of a module depends on or what modules depend on it.

# show the modules that v1.2.0 of github.com/example/foo depends on as a tabular list
> perseus query ancestors github.com/example/foo@v1.2.0 --list
Dependency                                            Direct
github.com/pkg/errors@v0.9.1                          true
golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c  true
...

# show the modules that depend on v0.9.1 of github.com/pkg/errors as nested JSON, using jq to format it nicely
> perseus query descendants github.com/pkg/errors@v0.9.1 --json | jq .
{
    "module": {
        "Path":    "github.com/pkg/errors",
        "Version": "v0.9.1"
    },
    "deps": [
        {
            "module": {
                "Path":    "github.com/example/foo",
                "Version": "v1.2.0"
            },
            ...
        }
    ]
}

In addition to text-based results using --json, --list or --format, the ancestor and descendant commands also supports outputting DOT directed graphs using the --dot flag.

# generate an SVG image of the dependency graph for the highest version of github.com/example/foo
> perseus query ancestors github.com/example/foo --dot | dot -Tsvg -o ~/foo_deps.svg

The depth of the tree can be controlled by the --max-depth flag, with a default of 4 hops.

Disclaimer: perseus is an open source project, not a CrowdStrike product. As such, it carries no formal support, expressed or implied. The project is licensed under the MIT open source license.