The one-and-only, must-have, eternal Go project layout

Twitter

Spoiler Alert: There is no such thing. The optimal Go project layout depends on your exact use case.

Design the architecture, name the components, document the details.

This quote is one of the Go proverbs.

Rob Pike presented the Go proverbs at a talk at Gopherfest 2015. They are available as a list since then, or recently also as (AI-generated) Limericks if you prefer.

While Rob Pike focuses on the naming aspect of this proverb, I want to talk about architecture, or rather, one particular part of it:

How to lay out a Go project.

This topic is frequently discussed among Gophers, mainly for one reason:

There is no such thing as a standard Go project layout.

If you think about it, there cannot be a single way of laying out a Go project because different project types have different needs.

Whenever I come across an article or a repo that promotes a particular Go project layout as a “standard,” I think: “framework.”

Frameworks are unpopular in Go because they impose a rigid structure on a project that might or might not match the project's specific needs.

The same applies to the files-and-directories layout of a project. No single layout fits all purposes.

So how to lay out a Go project then?

By following best practices.

Go project layout best practices: Ensure to get these aspects right

Best practices are a strong guidance without being dogmatic. Apply common sense when using them.

Start small

If you start a new project and don't know how large it will become over time, use the simplest layout possible, and add structure only if required.

Name your packages by their functionality

Avoid grab-bag package names like util, models,controllers, helpers, or similar. (This is the “name your components” part of the proverb.) Rather, group your code by functionality or responsibility.

Follow the (few) conventions

Known directory names

While you are free to choose arbitrary directory names for your project layout, you'll want to know and memorize these special directory names:

  • internal/: Packages inside the internal directory are not accessible outside the Go module they belong to. Even the Go toolchain follows this convention. If you expose a package to the public, it becomes a public API, so you have to ensure it follows semantic versioning. And you will receive issues and pull requests from users of this package, even if the package was only meant for use inside your project. Use the internal directory to avoid this situation.
  • testdata/: Put ancillary test data inside this directory. The Go toolchain ignores anything inside. (Side node: in addition to testdata, the Go toolchain also ignores any directory starting with . or _.)
  • vendor/: While the Go proxy server does already a good job of helping to ensure consistent builds, you can go one step further and download all 3rd-party dependencies into a vendor directory. go mod vendor expects this directory name.

Do not use "src"

Do not use a src directory. It simply makes no sense because a Go project is plain source code.

Moreover, every extra directory makes your import paths longer. Strive to keep the import paths short by omitting directories that do not add value.

Layouts by example

Best practices and conventions alone are not sufficient to define how useful project layouts look like. So let me list a few examples.

CLI tools

Command-line tools do not need much structure. The simplest ones require only a single directory. This directory is the root of the repository.

Let's assume you want to write a data compression tool. You name the root directory compress because Go picks the main package's directory name as the name of the binary by default. All your .go files reside in the compress directory, no subdirectories required.

compress/
+-- main.go
    go.mod
    go.sum

Assuming the project repository has been published to github.com/your/compress, users can install your tool directly through

go install github.com/your/compress@v1.2.3

(although I always recommend pre-packaging public binaries for package managers. This is easier than you might think.)

Chances are that your tool includes one or more library packages for its own use. Add them to the internal directory, for the reasons stated above.

compress/
+-- main.go
    go.mod
    go.sum
    internal/
    +-- deflate/
        +-- deflate.go

I'll come to public library packages later.

Pure library projects

A library project with no auxiliary CLI tools can start with a single directory, just like the simple CLI tool project.

A library for compressing data, for example, would look like so:

compress.go
go.mod
go.sum

Additional (public) packages, such as encoding and decoding packages, get separate folders at the root level.

compress/
+-- compress.go
    go.mod
    go.sum
    encode/
    +-- encode.go
    decode/
    +-- decode.go

The import paths are straightforward:

import "github.com/your/compress"
import "github.com/your/compress/encode"
import "github.com/your/compress/decode"

CLI tools with public packages, and libraries with CLI tools

The next two project types share the same resulting layout.

  1. A CLI tool that also exposes packages to its users
  2. A library that comes with one or more auxiliary CLI tools

The convention for both cases is to have the library code at the project's root and put code for command-line tools in a cmd directory.

So if we combine our compress tool and compress library into one, that's what we get:

compress/
+-- compress.go
    go.mod
    go.sum
    cmd/
    +-- compress/
        +-- main.go

For library users, the import path remains as short as in the pure library example.

import "github.com/your/compress"

Tool users would install the tool as

go install github.com/your/compress/cmd/compress@v1.2.3

...which is a bit longer than before, but that's not a deal-breaker.

Interim result: Everything together

If we throw all of the above together—a library with sub-packages, internal packages, and command-line tools, we do not need to change any of the previous approaches. They fit nicely together.

compress/
+-- compress.go
    go.mod
    go.sum
    encode/
    +-- encode.go
    decode/
    +-- decode.go
    internal/
    +-- deflate/
        +-- deflate.go
    cmd/
    +-- compress/
        +-- main.go

Note that even though the internal deflate package belongs to the command, placing the internal directory at the root is still better because the resulting import path is shorter.

Large and mixed-language projects

Move forward to large application projects that add non-code artifacts, other languages, documentation, DB migration files, build and deploy scripts, or Website assets. To avoid re-inventing the wheel every time you start a new project, you'll want to have consistent places for these items.

This is where project templates like the self-proclaimed “Standard Go Project Layout” come into play. “Self-proclaimed” because this template is anything but an official standard layout. Still, it is a good showcase of how a full-featured project layout can look like.

Imagine that our compress project has grown into a full-blown Web app. If we follow the Standard Go Project Layout, our project might look like this:

compress/
+-- go.mod
    go.sum
    README
    Makefile
    api/
    assets/
    build/
    cmd/
    ...
    pkg/
    vendor/
    web/
    website/

Despite the size of such a project, the layout remains clear at the top level because everything is tucked away under a specific directory.

Even the Go library packages.

They live inside a directory named pkg. This layout change is nontrivial because it is incompatible with the above project layouts.

  • All root-level library code shifts into the pkg directory.
  • All import paths of the packages inside pkg will contain a /pkg/ element.

The Standard Go Project Layout's README says about the pkg directory:

This is a common layout pattern, but it's not universally accepted and some in the Go community don't recommend it.

It's ok not to use it if your app project is really small and where an extra level of nesting doesn't add much value (unless you really want to :-)). Think about it when it's getting big enough and your root directory gets pretty busy (especially if you have a lot of non-Go app components).

To pkg or not to pkg?

“But Christoph, doesn't the Go community frown upon the use of a pkg directory?” Yes and no. What I see is that one part of the Go community is happy with using pkg, and another part is happy without.

Both sides have good reasons.

Pro pkg

Among the various arguments in favor of using a pkg directory, three major benefits stick out.

  1. A cleaner repository. When everything else is stored in a subdirectory, library packages should be no exception. The repository layout remains consistent and navigable.
  2. pkg is a clear sign that these packages are meant for public use. A dedicated directory for public packages is a clear sign for the project users: “Here you can find our public packages.”
  3. No naming collision with non-Go assets. Imagine your project contains package web inside a web directory at the root level. Then you want to add a directory for Website assets. The name web would be the well-known standard name for this, but alas, it's already taken! A pkg directory prevents this from happening.

Proponents of using pkg:

Peter Bourgon: Go best practices, six years in. The Industrial Programmer suggests using cmd and pkg as a minimal layout. "All of your artifacts remain go gettable. The paths may be slightly longer, but the nomenclature is familiar to other Go developers. And you have space and isolation for non-Go assets." However, Peter clarifies that while this layout may be a good fit for many types of projects, there is no single best repo structure.

Travis Jeffery: I'll take pkg over internal. Travis criticizes the internal directory because it diminishes the use of public projects. I disagree. Any package outside internal must meet the elevated standards of a public package API. Hence, a project owner is entitled to declare a package for internal use only. But he has a couple of very reasonable arguments towards using pkg. For example: "...the directory is useful boilerplate that clarifies the project’s layout for people. Useful boilerplate for clarity sounds like a Go tagline."

Kyle C. Quest: Go Project Layout. Kyle observes that the pkg pattern is "pretty popular" but admits that it can confuse Go newcomers because of the pkg directory inside GOPATH that has nothing but the name in common with the pkg directory in repository layouts.

Con pkg

Not using a pkg directory has also several benefits.

  1. Import paths are shorter. The pkg directory is only a “pass-through” step to package subdirectories. It contains no packages directly, hence a /pkg/ element inside an import path is a pointless no-op that provides no useful grouping.
  2. Any package outside the internal directory is public anyway. A pkg directory does not prevent the users of a project from looking for public libraries elsewhere. If you want to avoid maintaining a package for public use (which is always more effort than maintaining an internal-only package), then you should better put it inside internal anyway.
  3. Do not use what you do not need. Promoting a pkg directory as a standard might make Go newcomers believe that even the smallest projects require a pkg directory, where it would, in fact, add no value at all.
  4. Large application projects should not double as library projects. If an application contains one or more packages that are valuable to others independently of the application project (and hence should not be inside internal), these packages should be moved to a separate library project.

Proponents of a life without pkg:

Eli Bendersky: Simple Go project layout with modules. Eli argues that pkg is useful only in large application projects with lots of non-Go stuff around, but such application projects most certainly contain packages for their own use only, and hence the packages should better live in internal. (Similar to #2 above.) And pure library projects do not have the problem of isolating Go code from a truckload of non-Go assets.

Xe Iaso: The Within Go Repo Layout. Xe points out possible confusion with $GOPATH/pkg. Since Go Modules, however, the GOPATH directory has become much less visible, and I think the likeliness of confusing newcomers has diminished drastically. Xe also points out that not imposing a pkg directory leaves the development team more degrees of freedom in naming things. And if an application project also exposes packages, then those packages might as well go into a separate, pure library project. (#4 above)

My advice? If you use pkg, use it with caution

My conclusion from all the aspects I laid out above is this:

Pure library projects need no pkg directory.

Projects that contain only library packages do not require any isolation from non-Go assets. Do not use pkg and let your users enjoy shorter, more readable import paths. Even if the library happens to require a few non-Go assets, like a Web UI library that uses HTML and CSS files, there is usually no point in adding a pkg directory. After all, the Go packages are first-class citizens in a Go library project.

Small projects that contain library packages and CLI tools require no pkg directory.

Even when tools and libraries live side-by-side, adding pkg has no benefit. The cmd directory is enough. Keep the import paths clean.

Large applications, might use a pkg directory, but there are consequences

In a large application, possibly with lots of non-Go assets, isolating packages from the rest of the project through a pkg project might make sense.

But if you do so, be aware of the consequences.

  1. These packages will be open for everyone to use. You will receive issues and pull requests for those packages that are not related to the parent project. The packages might start to live their own life.
  2. Because of #1, you have way fewer options for modifying those packages. They have a public API now, and so you implicitly gave stability guarantees to the public. If you modify those packages to meet the changing needs of your application, you always risk breaking some client code that's completely unrelated to your application.
  3. Given that #1 and #2 are still fine for you, consider that all packages belong to the same module (unless you are adventurous enough to build and maintain a multimodule repository). They all stick to the same versioning. If you add a feature to one of the public packages, you have to increase the minor version of the whole project, or clients would not be able to upgrade the package to include that feature.
Consider splitting dual application/library projects

To make your life easier, you might want to keep public library packages separate from your application project.

  • Packages that can exist in their own right and that are worth exposing to the public can be collected in one or more separate library projects.
  • Packages that serve the application but do not work well as stand-alone packages go into internal.
  • Packages that serve the application, and for which you want to reserve the right to apply breaking changes to their API, should live in the internal directory.

More brain food

Package-level layout

Dividing a project into library packages and commands is only the first step. At the next level, you'll want to think about how to organize your code into packages. Should you use a monolithic package, or a Rails-style scheme with handler, controller, and model packages?

Ben Johnson might have an answer for you: Standard Package Layout

The Go team also shared some insights about Organizing Go code, but be aware that the article is from 2012, where GOPATH as the single Go workspace was still a thing. Apart from that, their advice still holds.

Applied Go Courses helps you get up to speed with Go without friction. Our flagship product, Master Go, is an online course with concise and intuitive lectures and many practices and exercises. Rich graphic visualizations make complex topics easy to understand and remember. Lectures are short and numerous to save your precious time and serve as quick reference material after the course. Learn more at https://appliedgo.com.

Categories: : Best Practices, Ecosystem

© 2024 Applied Go Courses