Using Protobuf a.k.a Google Protocol buffers


Consider this. You are measuring values using a IoT-device and decide you want to send the data to a server. But how to break down the data into sendable packages of bytes? This process is called encoding. An encoding needs to be understood by both the device and the server. You could come up with your own encoding, but as you will learn in this article, working solutions exists. This article covers one such solution called Protobuf.

Let’s start by giving an example. A client measures two things: light intensity and its own battery voltage. The values need to be sent to a server that will store them to a database(DB) for later presentation.

Client server set up. Send a message a from client to server

The client will need to construct a message of bytes for the server to read. But how to encode the measured values into bytes? One approach would be to invent an encoding yourself:

Let’s say that the client sends a light intensity of 10 and a voltage of 3.7 volts. With the above encoding some pseudo c code would look like this:

The above message would result in 9 bytes being sent to the server. Naturally, this means that the server will need to know what encoding that was used. This is fine, but will have implications for you as a developer, more about this later. For the keen eyed, you might have noticed that I represent light intensity value as the decimal value of two bytes, but battery voltage as the char ‘integer’ of two bytes, one byte for the number part and a byte for the fractional part. This has the upside of simplicity when decoding, but the downside of using more bytes to represent the value.

So this works, right? So why bother using something to aid in this process?

My guess is that you will be interested of:

  1. Keeping message encodings updated on all devices.

    If you have many devices communicating with each other, making sure that all of them interpret messages correctly is paramount! This would require that there is no mismatch between encoding protocols between devices. Trust me, forgetting to update one device or wrongly ordering values in a message is just a question of when, not if.

  2. Value encoding

    For speed and efficiency you are interested in encoding values as tightly packed into a byte array as possible. Figuring this out by yourself could be interesting, but probably not what you want to devote your building-a-monster-robot-time to.

  3. Platforms and programming languages

    Maybe you have devices running different programming languages and platforms. You will quickly realize that managing message encodings under several programming languages and platforms takes its fair share of time and effort.

With that said! Let me introduce Protocol buffers, a.k.a Protobuf

Protobuf – one definition to rule them all

At its core Protobuf is a language and platform agnostic markup language, similar to JSON. You define your messages once, and preferably in one file, then let Protobuf’s compiler (protoc) generate hard typed message structures for you to include in your favorite programming language.

The proto files is the common definition of all devices, regardless of programming language or platform

Example of a proto message definition

The message in the introduction example would look like this in proto3 syntax:

Yes, it is this simple. Nothing more needs to be defined in order for this message to be interpreted by your devices. If you want more messages simple add more, just make sure to begin with the message prefix. There are many different variable types supported; visit Google protobuf page for more information.

Hey, what gives?

Your not using required and optional prefixes!?

Answer: Quite right! I’m using the latest proto3 syntax, where the required and optional no longer are needed. All attributes are now optional.

Protoc – compilation

We now have our message.proto file containing a message definition. The next step is to run it through protoc, a compiler that will generate programming-language specific files for inclusion in your device code.

The easiest way to get protoc is to grab a a pre-compiled binary of it from github. Go to the bottom of that page for a list of zip files. Select a protoc-zip for the system you are on.

If you are using a linux system download it by doing the following:

I’m on Ubuntu, can I apt-get?

You can! But make sure that you get a protoc binary that runs at least version 3.0.0. Otherwise you won’t have support for the latest proto3 syntax.

When writing this article it would give me the following version: 2.6.1

Example: compiling for python

Example: Encoding a message in python

Example: Decoding a message in python

That’s it!

I use protobuf extensively in the open source IoT platform IOAnt. Check it out if you want to see protobuf in action. 😉

Feel free to leave a comment if you have any questions about protobuf!