Go modules and private GIT repositories

Some time ago i stumbled upon an issue that was giving me a headache. I had to download and import private Go module, and as it turns out, that can be little tricky. I'm using SSH keys for authenticating with my repositories, so this solution is applied only for that type of authentication.
Of course, i scouted internet looking for solutions, and there are some examples you can find, but none was working for me. Until i realized that there is small but crucial difference depending on which Git repository hosting service you are using. I will show you how to do it for GitHub and Bitbucket. Of course, all examples here are using URLs to my private repositories, and you will have to change them to match yours.

I prepared two simple private repositories for this little demo. Both are essentially the same. They have one package named hello with function World() returning string Hello World!. They are located on:

• GitHub - github.com/matijakrajnik/goprivatemodule
• Bitbucket - bitbucket.org/matijakrajnik/goprivatebucket

Now, if i tried simply running go get like below, that will not work.

go get github.com/matijakrajnik/goprivatemodule
go get bitbucket.org/matijakrajnik/goprivatebucket

First thing i had to do is set GOPRIVATE environment variable. This variable contains a list of module path prefixes that are considered to be private, so go will not use proxy nor trying to validate checksum from public database.

To set that variable, run:

go env -w GOPRIVATE=github.com/matijakrajnik/goprivatemodule

It's also possible to use glob pattern to match all of your repositories:

go env -w GOPRIVATE=github.com/matijakrajnik/*

And even make comma separated list to include multiple values:

go env -w GOPRIVATE=github.com/matijakrajnik/*,bitbucket.org/matijakrajnik/*

Setting GOPRIVATE variable is only first step. When you run go get command, Go uses git for downloading module source files. And by default, git uses HTTPS, which will not work because terminal prompts are disabled and git don't have any credentials for authentication. So you will end up will error like this:

$ go get -v github.com/matijakrajnik/goprivatemodule
go get: module github.com/matijakrajnik/goprivatemodule: git ls-remote -q origin in /home/matija/go/pkg/mod/cache/vcs/f53d9a18ffd3065378124236da32538eaf1adb1addf29c1ac217b2540997b252: exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

Fortunately, if you are using SSH keys for authentication, solution is very simple. You just need to configure git to use git instead of https.

For GitHub repository use:

git config --global url."[email protected]:matijakrajnik/goprivatemodule.git".insteadOf "https://github.com/matijakrajnik/goprivatemodule"

For Bitbucket repository use:

git config --global url."[email protected]:matijakrajnik/goprivatebucket".insteadOf "https://bitbucket.org/matijakrajnik/goprivatebucket"

If you look carefully at these two commands above, you can see that subtle little difference i mentioned at the beginning. Notice that in GitHub URL you have to include .git suffix, while for Bitbucket you can't use it. This can also vary from one git service implementation to another, so if you have problem, try to add or remove that .git suffix.

You can check git configuration file located in ~/.gitconfig, to check that commands above took effect. If everything is ok, you will see something like this:

[url "[email protected]:matijakrajnik/goprivatemodule.git"]
  insteadOf = https://github.com/matijakrajnik/goprivatemodule
[url "[email protected]:matijakrajnik/goprivatebucket"]
  insteadOf = https://bitbucket.org/matijakrajnik/goprivatebucket

With that setup i was able to download my private modules with:

go get github.com/matijakrajnik/goprivatemodule
go get bitbucket.org/matijakrajnik/goprivatebucket

And use them in my project:

package main

import (
  "fmt"

  "github.com/matijakrajnik/goprivatemodule/hello"
)

func main() {
  fmt.Println(hello.World())
}

package main

import (
  "fmt"

  "bitbucket.org/matijakrajnik/goprivatebucket/hello"
)

func main() {
  fmt.Println(hello.World())
}