Go Frequently Asked Questions

Versions

What's the difference between github.com/golang/protobuf and google.golang.org/protobuf?

The github.com/golang/protobuf module is the original Go protocol buffer API.

The google.golang.org/protobuf module is an updated version of this API designed for simplicity, ease of use, and safety. The flagship features of the updated API are support for reflection and a separation of the user-facing API from the underlying implementation.

We recommend that you use google.golang.org/protobuf in new code.

Version v1.4.0 and higher of github.com/golang/protobuf wrap the new implementation and permit programs to adopt the new API incrementally. For example, the well-known types defined in github.com/golang/protobuf/ptypes are simply aliases of those defined in the newer module. Thus, google.golang.org/protobuf/types/known/emptypb and github.com/golang/protobuf/ptypes/empty may be used interchangeably.

What are proto1, proto2, and proto3?

These are revisions of the protocol buffer language. It is different from the Go implementation of protobufs.

  • proto3 is the current version of the language. This is the most commonly used version of the language. We encourage new code to use proto3.

  • proto2 is an older version of the language. Despite being superseded by proto3, proto2 is still fully supported.

  • proto1 is an obsolete version of the language. It was never released as open source. See the Protocol Buffers FAQ for more information on proto1.

There are several different Message types. Which should I use?

Common problems

"go install": working directory is not part of a module

You have set the environment variable GO111MODULE=on and are running the go install command outside of a module directory.

Set GO111MODULE=auto, or unset the environment variable.

constant -1 overflows protoimpl.EnforceVersion

You are using a generated .pb.go file which requires a newer version of the "google.golang.org/protobuf" module.

Update to a newer version with:

go get -u google.golang.org/protobuf/proto

undefined: "github.com/golang/protobuf/proto".ProtoPackageIsVersion4

You are using a generated .pb.go file which requires a newer version of the "github.com/golang/protobuf" module.

Update to a newer version with:

go get -u github.com/golang/protobuf/proto

What is a protocol buffer namespace conflict?

All protocol buffers declarations linked into a Go binary are inserted into a global registry.

Every protobuf declaration (e.g., enums, enum values, or messages) has an absolute name, which is the concatenation of the package name with the relative name of the declaration in the .proto source file (e.g., my.proto.package.MyMessage.NestedMessage). The protobuf language assumes that all declarations are universally unique.

If two protobuf declarations linked into a Go binary have the same name, then this leads to a namespace conflict, and it is impossible for the registry to properly resolve that declaration by name. Depending on which version of Go protobufs is being used, this will either panic at init-time or silently drop the conflict and lead to a potential bug later during runtime.

How do I fix a protocol buffer namespace conflict?

The way to best fix a namespace conflict depends on the reason why a conflict is occurring.

Common ways that namespace conflicts occur:

  • Vendored .proto files. When a single .proto file is generated into two or more Go packages and linked into the same Go binary, it conflicts on every protobuf declaration in the generated Go packages. This typically occurs when a .proto file is vendored and a Go package is generated from it, or the generated Go package itself is vendored. Users should avoid vendoring and instead depend on a centralized Go package for that .proto file.

    • If a .proto file is owned by an external party and is lacking a go_package option, then you should coordinate with the owner of that .proto file to specify a centralized Go package that a plurality of users can all depend on.
  • Missing or generic proto package names. If a .proto file does not specify a package name or uses an overly generic package name (e.g., "my_service"), then there is a high probability that declarations within that file will conflict with other declarations elsewhere in the universe. We recommend that every .proto file have a package name that is deliberately chosen to be universally unique (e.g., prefixed with the name of a company).

    • Warning: Retroactively changing the package name on a .proto file can potentially cause the use of extension fields or messages stored in google.protobuf.Any to stop working properly.

Hyrum's Law

What is Hyrum's Law, and why is it in this FAQ?

Hyrum's Law states:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

A design goal of the latest version of the Go protocol buffer API is to avoid, where possible, providing observable behaviors that we cannot promise to keep stable in the future. It is our philosophy that deliberate instability in areas where we make no promises is better than giving the illusion of stability, only for that to change in the future after a project has potentially been long depending on that false assumption.

Why does the text of errors keep changing?

Tests depending on the exact text of errors are brittle and break often when that text changes. To discourage unsafe use of error text in tests, the text of errors produced by this module is deliberately unstable.

If you need to identify whether an error is produced by the protobuf module, we guarantee that all errors will match proto.Error according to errors.Is.

Why does the output of protojson keep changing?

We make no promises about the long-term stability of Go's implementation of the JSON format for protocol buffers. The specification only specifies what is valid JSON, but provides no specification for a canonical format for how a marshaler ought to exactly format a given message. To avoid giving the illusion that the output is stable, we deliberately introduce minor differences so that byte-for-byte comparisons are likely to fail.

To gain some degree of output stability, we recommend running the output through a JSON formatter.

Why does the output of prototext keep changing?

We make no promises about the long-term stability of Go's implementation of the text format. There is no canonical specification of the protobuf text format, and we would like to preserve the ability to make improvements in the prototext package output in the future. Since we don't promise stability in the package's output, we've deliberately introduced instability to discourage users from depending on it.

To obtain some degree of stability, we recommend passing the output of prototext through the txtpbfmt program. The formatter can be directly invoked in Go using parser.Format.

Miscellaneous

How do I use a protocol buffer message as a hash key?

You need canonical serialization, where the marshaled output of a protocol buffer message is guaranteed to be stable over time. Unfortunately, no specification for canonical serialization exists at this time. You'll need to write your own or find a way to avoid needing one.

Can I add a new feature to the Go protocol buffer implementation?

Maybe. We always like suggestions, but we're very cautious about adding new things. See the Protocol Buffers FAQ.

The Go implementation of protocol buffers strives to be consistent with the other language implementations. As such, we tend to shy away from feature that are overly specialized to just Go. Go-specific features hinder the goal of protocol buffers being a language-neutral data interchange format.

Unless your idea is specific to the Go implementation, you should join the protobuf discussion group and suggest it there.

If you have an idea for the Go implementation, file an issue on our issue tracker: https://github.com/golang/protobuf/issues

Can I customize the code generated by protoc-gen-go?

In general, no. Protocol buffers are intended to be a language-agnostic data interchange format, and implementation-specific customizations run counter to that intent.