Protocol Buffers with SwiftNIO

Previously, we looked at sending a simple string across the wire through TCP. Chances are you want to send more complex objects over the network. So how do we do that? One way is with Google Protocol Buffers, or protobufs for short. Taken from their developer page:

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data — think XML, but smaller, faster, and simpler.

In this post, we are going to look at how to use protocol buffers with Swift and SwiftNIO!

Overview

You define your message schema in a .proto file and then use the protobuf compiler to generate the data structures for your language, thereby reducing writing boilerplate parsing and data access code. You can then share this .proto file with other parts of your system and generate data access classes for those languages. For example, you could have a Java backend and an iOS, Android and web frontend that all share this .proto file defining the shared schema.

Advantages

Protobufs were designed to be fast and compact. According to Google, protocol buffers are 3 to 10 times smaller and 20 to 100 times faster than XML.

Protobufs are language agnostic and provide backwards compatibility. You could replace an old sluggish system component with a new high-performance system component in an entirely different language and have confidence that the system will continue to work as long you use the same .proto schema. In fact, you can even add new fields to your message formats in the .proto file and your system will continue to function. Systems with the older .proto will simply ignore the new fields when parsing.

Downsides

Protobufs are not for everyone, and although they support many languages, there might not be support for your language.

XML and JSON are more human readable than protobufs. Because they are designed to be compact, protobufs strip away field names when sent across the network. A protobuf is only useful if you have the .proto file.

Google designed protocol buffers for inter-application communication. If you are building a public API for outside consumers to consume, you really wouldn’t expect the clients of your server to use your defined .proto file, generate data access classes for their service, and communicate to your service that way.

Getting started

You will need the protobuf compiler to generate the language specific files. There are install instructions located here: https://github.com/apple/swift-protobuf. Since I already have Homebrew installed, I went with the Homebrew option.

$ brew install swift-protobuf

Note: This install might take some time…

To check that it installed correctly, you can type:

$ protoc --version 
// my output is libprotoc 3.5.1

.proto

Protobufs use the extension .proto for their file format. You create .proto files to define your messaging schema and then run the protobuf compiler to generate your Swift code.

You can use any IDE or text editor to create a .proto file, including Xcode. However, what I did not like about using Xcode was the lack of proper indentation and syntax highlighting. I ended up using VS Code along with the free vscode-proto3 protobuf extension from their marketplace.

Now that we have the protobuf compiler installed, let’s create a Movie type that we will send across the network. Create a new file called movie.proto.

The first line of the .proto file is what protocol buffer version to use. We are going to use the latest version at the time of this writing, proto3.

syntax = "proto3"

Next we declare our message type with the message keyword. For us, this will be a message of type Movie.

message Movie {
}

Next let’s declare an enum for the movie genre. You can declare an enum outside of message in the global scope, but for our example, since genre is describing the movie, we will declare it inside Movie.

message Movie {
    enum Genre {
        COMEDY = 0;
        ACTION = 1;
        HORROR = 2;
        ROMANCE = 3;
        DRAMA = 4;
    }
}

Let’s add some actual fields to our Movie.

message Movie {
    enum Genre {
        COMEDY = 0;
        ACTION = 1;
        HORROR = 2;
        ROMANCE = 3;
        DRAMA = 4;
    }
    
    string title = 1;
    Genre genre = 2;
    int32 year = 3;
}  

Here we defined a string to describe the title, an enum for the genre, and an int32 for the year the movie was released. You can find out more about the different fields in the Google protocol buffer documentation.

Field names should be lowercased letters. If you have multiple words, the preferred method is to use an underscore (“_”) to separate the words. Pascal casing and Camel casing are not recommended. Don’t worry, when we run the protobuf compiler, it will generate Swift code that follows the standard coding style for Swift.

The last element in the fields is the field tag. This is what gets sent across the wire in order to reduce the size of the payload. With JSON, we send the field description along with the value which makes it more readable, but also a bigger payload.

{ "title" : "Avengers" }

But with protobufs, it’s more like:

1Avengers

These field tags are used as identifiers and must be unique integers. They are going to be converted to bytes and sent across the wire therefore smaller values are more efficient. If you have lots of fields that could be optional, the ones that are most likely to be set should have the lowest numbers. Optional fields that are not set do not get sent across the wire. The default values for fields are their zero values (e.g. 0 for int, empty string for strings). Note that for enumerations you should have a default value defined for the value 0 because the default value for an enumeration is 0.

Here’s what our movie.proto looks like when we are done.

Compile

Now that we have the message format, we can use the protobuf compiler to generate our data access classes. Change directory to where you created your .proto file and run this to generate a Swift struct.

$ protoc --swift_out=. movie.proto

Breaking down this command, we use the protobuf Swift compiler with the --swift_out argument. This tells the compiler that we want to generate a Swift source file. The argument expects the location of the directory you want the source file generated. In our case, we use the . to denote we want the source file generated to the current directory. The next argument we pass is movie.proto which is our input file.

The compiler generates movies.pb.swift. The pb prefix denotes that it’s a protocol buffer generated class. Now we are ready to integrate this code into our project. You don’t want to edit this file, we want to just use this file. If you do edit it by accident, no sweat. Just run the compiler again to regenerate the data.

Setup

We are going to set up a client called MovieClient and a server called MovieServer and send our Movie struct from the client to the server with SwiftNIO. If you need details on setting up a client and server with Swift Package Manager (SPM) and SwiftNIO, check out this article: Getting Started with SwiftNIO.

In addition to using SwiftNIO, we are also going to use Swift protobufs so we need to declare that as a dependency in our Package.swift file.

My Package.swift file for MovieClient looks like:

// swift-tools-version:4.0
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
    name: "MovieClient",
    dependencies: [
        .package(url: "https://github.com/apple/swift-nio.git", .exact("1.6.0")),
        .package(url: "https://github.com/apple/swift-protobuf.git", .exact("1.0.3"))
],
targets: [
    .target(
        name: "MovieClient",
        dependencies: ["NIO", "SwiftProtobuf"]),
    ]
)

Similarly, my Package.swift file for MovieServer looks like:

// swift-tools-version:4.0
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
    name: "MovieServer",
    dependencies: [
        .package(url: "https://github.com/apple/swift-nio.git", .exact("1.6.0")),
        .package(url: "https://github.com/apple/swift-protobuf.git", .exact("1.0.3"))
],
targets: [
    .target(
        name: "MovieServer",
        dependencies: ["NIO", "SwiftProtobuf"]),
    ]
)

Note: We are using an exact dependency for this article, but you alternatively can take advantage of SPM’s semantic versioning and use:

.package(url: "https://github.com/apple/swift-nio.git", from: "1.6.0"),
.package(url: "https://github.com/apple/swift-protobuf.git", from: "1.0.3")

This way we depend on the minor version that we require and still get all the latest and greatest fixes and performance enhancements from the great team at Apple and other contributors to SwiftNIO!

Once your Package.swift is updated, don’t forget to run:

$ swift package update
$ swift package generate-xcodeproj

Now that the projects are setup, drag the movie.pb.swift file into both Xcode projects. Make sure that the Copy Items if Needed box is checked.

MovieClient

The main change we are going to do from sending a simple String across the network is now send a Movie instance. We do this in our ChannelInboundHandler subclass, MovieClientHandler. Because we included movie.pb.swift into our project, we are now able to create Movie objects.

var movie = Movie()
movie.genre = .action
movie.title = "Avengers: Infinity War"
movie.year = 2018

We can then serialize our data with the serializedData() function from the protobuf generated file.

let binaryData: Data = try movie.serializedData()

From there, we follow what we did previously, create a ByteBuffer and write to the buffer.

var buffer = ctx.channel.allocator.buffer(capacity: binaryData.count)
buffer.write(bytes: binaryData)

At this point, we are ready to send the data. But let’s introduce another concept of SwiftNIO, the EventLoopPromise<T>.

Promises and futures

Taken from SwiftNIO’s Github page:

An EventLoopFuture<T> is essentially a container for the return value of a function that will be populated at some time in the future. Each EventLoopFuture<T> has a corresponding EventLoopPromise<T>, which is the object that the result will be put into. When the promise is succeeded, the future will be fulfilled.

If you are familiar with promises from another programming language, you should feel right at home. By using promises and futures, we are able to write code that will get executed when an event happens. For example, you don’t want to block the main thread and have an unresponsive UI when downloading a large image from the internet. Instead your goal should be to start a long task, and then do an action when that long task is completed. Promises provide this capability.

We create a promise that prints a message to the console and then closes the channel from the client after we send the data successfully.

let promise: EventLoopPromise<Void> = ctx.eventLoop.newPromise()
promise.futureResult.whenComplete {
    print("Sent data, closing the channel")
    ctx.close(promise: nil)
}

Now we pass this promise when we write the data to the socket.

ctx.writeAndFlush(wrapOutboundOut(buffer), promise: promise)

The channelActive method inMovieClientHandler.swift should look like:

MovieServer

On the server, we create a new MovieServerHandler that will handle the incoming data. The code is similar to what we had done previously when reading a String from the network. We unwrap the NIOAny to a ByteBuffer and get the number of bytes that are readable.

var buffer = unwrapInboundIn(data)
let readableBytes = buffer.readableBytes

We use a guard statement to read and unwrap the bytes from the buffer.

guard let received = buffer.readBytes(length: readableBytes) else {
    return
}

Then we create a Data object with the received bytes.

let receivedData = Data(bytes: received, count: received.count)

With one of the generated methods from movie.pb.swift, we deserialize our data.

let movie = try Movie(serializedData: receivedData)

Now you are able to work your Movie type within the application instead of raw bytes!

The channelRead method in MovieServerHandler.swift should look like:

Run it

Server:

• choose the MovieServer scheme to run on My Mac and hit Run
• once you see a message in the Xcode console that you are connected on port 3010, run the client

Client:

• choose the MovieClient scheme to run on My Mac and hit Run

In the console of MovieClient, you should see Sent data, closing the channel followed by Program ended with exit code: 0. The client has connected to the server, sent the Movie instance, and then closed the channel in the promise. In the console of MovieServer, you should see the received Movie object displayed!

For the full projects:

jonathanwong/MovieClient

jonathanwong/MovieServer

Conclusion

You explored protocol buffers, defined a .proto file to describe the Movie schema, and generated Swift data access code with the protobuf compiler. You looked at how easy it is to serialize and deserialize your Movie instance with protobufs. From there, you were able to send a Movie instance over TCP with SwiftNIO and took a brief look at promises.

Edit: In a real-world TCP server with multiple different message types of various lengths going through the network, you would need a way to determine where your message boundaries are. For example, you could prepend each message with a length field. This example does not cover that scenario.