Software, technology, sysadmin war stories, and more. Feed
Sunday, April 22, 2012

Using protocol buffers with no drama

After building more than a few systems using Protocol Buffers in the past couple of years, I've become rather fond of them. They let me do useful things without too much trouble and basically stay out of my way. Sure, it's from You Know Who, but it's some of the "old school" stuff which had the right people behind it. It's also "escaped" by way of an open source release, so it belongs to the world now.

It should come as no surprise that I still use protobuf to get things done. This made for interesting times when I started switching from make to my own build system, since the usual tricks do not apply. With protobuf, you start with your definitions in a .proto file. The "protoc" tool then transforms that into a pair of files ending with.pb.h and .pb.cc. Then you have to compile and link it in the usual way.

My project was built to look for #include "foo/bar.h". Given that, it opens that file and analyzes it. It also looks for a matching foo/bar.cc, and analyzes that as well. This works just fine for ordinary code written by me, but it completely fails for generated stuff like this. The reason is simple: I don't store the output from protoc in my tree.

So, I've extended my build system. Now, an #include "foo/bar.pb.h" will be recognized as a protobuf, and it will run protoc for you. It then directs protoc to write these files into a "genfiles" path, since it is neither source code nor per-build-target output. There is no difference in how you call protoc for debug, default, coverage, or optimized builds, in other words.

In practical terms, it means you can add protobufs to C++ code really quickly and without much fuss. Recall from last week that inputs and outputs had been split, and I created a really simple file called main.cc as a test. Let's take that and extend it to use protobufs as a demonstration.

So here we are in the tree...

~/depot$ find

I'll build "main" right up front to give a reference point.

~/depot$ bb core/main
I0422 045131 977 build/deptracker.cc:131] Analysis of all deps done
I0422 045131 977 build/dep.cc:297] cd src && g++ -Wall -Werror -I. -c core/main.cc -o ../bin/core/main.o
I0422 045132 977 build/deptracker.cc:199] g++ -Wall -Werror bin/core/main.o -o bin/core/main
-rwxr-xr-x 1 bb users 8121 Apr 22 04:51 bin/core/main

Okay, that's nothing surprising. Now let's create our protobuf definition.

~/depot$ cat > src/core/demo.proto
message Song {
  optional string title = 1;
  optional string artist = 2;

Right, so now I amend main.cc to #include the header which will be created and then actually use this new "Song" class. Now it looks like this:

#include <stdio.h>
#include "core/demo.pb.h"
static void do_stuff(int i) {
  if (i == 0) {
    printf("hello from main\n");
  } else {
int main() {
  Song rio;
  rio.set_artist("Duran Duran");
  printf("song debug info: %s\n",
  return 0;

Easy enough. Let's build it.

~/depot$ bb core/main
I0422 045806 1126 build/dep.cc:411] Creating protobuf: core/demo
I0422 045806 1126 build/dep.cc:412] protoc --cpp_out=genfiles/bin/core -Isrc/core src/core/demo.proto
I0422 045806 1126 build/deptracker.cc:131] Analysis of all deps done
I0422 045806 1126 build/dep.cc:250] cd genfiles/bin && g++ -Wall -Werror -I. -c core/demo.pb.cc -o core/demo.pb.o
I0422 045806 1126 build/dep.cc:297] cd src && g++ -I../genfiles/bin -Wall -Werror -I. -c core/main.cc -o ../bin/core/main.o
I0422 045806 1126 build/deptracker.cc:199] g++ -Wall -Werror -lprotobuf -lpthread bin/core/main.o genfiles/bin/core/demo.pb.o -o bin/core/main
-rwxr-xr-x 1 bb users 40979 Apr 22 04:58 bin/core/main

Our protobuf has been detected and turned into source code, and then that was compiled into an object. That object was then linked into the binary, so let's run it and see what happens...

~/depot$ bin/core/main
song debug info: title: "Rio"
artist: "Duran Duran"
hello from main

That's it. If I want to use this same definition in another program, I need only #include it in the same way and start working. Instead of having to worry about tracking all of this stuff in a Makefile or similar construct, my build tool will just find it and figure it out for me.

This sort of thing is useful to me. If I'm cruising along and working on something which calls for a protobuf, I can just write a quick definition, add the #include line, then start using it. I don't have to push all of that state onto my mental stack to go off and deal with build scripts or wrangle Makefile rules.

When it comes down to it, you're either writing code which furthers the cause of solving whatever problem you currently have, or you're off shaving yaks. Which one really deserves your valuable time?

Sorry, yaks. You're going to have to stay furry this summer.