Protocol Buffers for the Rest of Us

Motivation

Are you annoyed by having to fix the absolute imports generated by protoc?

If so, then protoletariat is the tool for you.

protoletariat has one goal: fixing the broken imports for the Python code generated by protoc.

See protocolbuffers/protobuf#1491 for the discussion that inspired this tool.

Installation

Artifact	Status	Installation Command
PyPI Package		pip install protoletariat
Conda Package		conda install protoletariat -c conda-forge
Docker Image	∅	docker pull ghcr.io/cpcloud/protoletariat:latest

Note: the conda-forge package version may lag behind the other artifacts by a few hours.

Usage

protoletariat is designed to be run as a post-processing step after running protoc. It operates directly on the generated code.

Example

Here's an example of how to use the tool, called protol:

1. Create a few protobuf files

// thing1.proto
syntax = "proto3";

import "thing2.proto";

package things;

message Thing1 {
  Thing2 thing2 = 1;
}

// thing2.proto
syntax = "proto3";

package things;

message Thing2 {
  string data = 1;
}

2. Run protoc on those files

$ mkdir out
$ protoc \
  --python_out=out \
  --proto_path=directory/containing/protos thing1.proto thing2.proto

3. Run protol on the generated code

$ protol \
  --create-package \
  --in-place \
  --python-out out \
  protoc --proto-path=directory/containing/protos thing1.proto thing2.proto

The out/thing1_pb2.py module should show a diff containing at least these lines:

-import thing2_pb2 as thing2__pb2
-
+from . import thing2_pb2 as thing2__pb2

How it works

At a high level protoletariat converts absolute imports to relative imports.

However, it doesn't convert just any absolute import to a relative import.

The protol tool will only convert imports that were generated from .proto files. It does this by inspecting FileDescriptorProtos generated from the proto files.

The core rewrite mechanism is implemented using a simplified form of pattern matching that looks at the Python AST, and invokes rewrite rules for matched import patterns.

Subcommands

protoletariat has a subcommand for each tool that you might like to use to generate FileDescriptorSet bytes:

Subcommand	Description
protoc	Uses protoc to generate FileDescriptorSet bytes
buf	Uses buf to generate FileDescriptorSet bytes
raw	You provide the FileDescriptorSet bytes as a file or directly from stdin

Help

$ protol
Usage: protol [OPTIONS] COMMAND [ARGS]...

  Rewrite protoc or buf-generated imports for use by the protoletariat.

Options:
  -o, --python-out DIRECTORY      Directory containing protoc or buf-generated Python code  [required]
  --in-place / --not-in-place     Overwrite all relevant files under `--python-out` with adjusted imports  [default: not-in-place]
  --create-package / --dont-create-package
                                  Recursively create __init__.py files under `--python-out`  [default: dont-create-package]
  -s, --module-suffixes TEXT      Suffixes of Python/mypy modules to process  [default: _pb2.py, _pb2.pyi, _pb2_grpc.py, _pb2_grpc.pyi]
  --exclude-google-imports / --dont-exclude-google-imports
                                  Exclude rewriting imports prefixed with google/protobuf
  -e, --exclude-imports-glob TEXT
                                  Exclude imports matching a glob pattern from being rewritten. Multiple values are allowed
  --help                          Show this message and exit.

Commands:
  buf     Use buf to generate the FileDescriptorSet blob
  protoc  Use protoc to generate the FileDescriptorSet blob
  raw     Rewrite imports using FileDescriptorSet bytes from a file or stdin