Analyzing TV audience data with Rust at TVbeat

TVbeat crunches a lot of data. Whether it's ingesting 1st and 3rd party data provided by our customers, or providing instant responses to queries made on top of that data, our expected turnover times are short. This allows our users to quickly make decisions based on their data, but also poses some exciting engineering challenges.

The market of high-performance programming languages with decent abstraction mechanisms isn't too big. For the longest time, TVbeat was a C++ shop, a traditional choice for many companies trying to shave every possible millisecond off of the running time of their software.

Our system has a fair amount of core, infrastructural code. However, we also help our customers get their data into our system through a process we call "integration". Separating the infrastructural code from the code dealing with integrations was one of the key features that has allowed our team to stay agile and able to respond to changes quickly.

In C++, we've used LuaJIT to expose simple-to-use APIs for integrations. This allowed the developers in charge of the integration to focus on the logic of transforming the data to TVbeat's format in a safe, performant language, instead of having to deal with the intricacies of C++.

In early 2018, we have started exploring Rust as a possible companion — if not an outright alternative — to C++. As is usual with exploratory, experimental deployments of new technologies, we have started with smaller, more isolated components of the system.

The inaugural project was a new generation of a parsing system for incoming EPG data. It turns out Rust has had a lot to offer in this
particular case!

Along with the fair amount of lower-level control of memory management, making it a great choice for systems software, Rust is also an expressive language suitable for writing higher-level software like HTTP handlers and, in our case, customer-dependent integration code. This allowed us to avoid an additional dependency (LuaJIT), and keep all logic in a single language.

Rust's support for generic programming naturally gives way to writing pluggable and composable libraries, making the whole process of separating the "core" and "integration" code even more obvious. Reasonably-typed boundaries of these systems are also proving
beneficial, a thing that was hard-to-impossible to emulate across the FFI boundary.

In non-greenfield projects, it's hard to overstate the importance of interoperability with existing C code, and Rust was once again very cooperative. Since we've already had C wrappers for a lot of our code in order to make it usable from LuaJIT, migrating these to be used in Rust was a matter of tweaking the FFI definitions.

Cargo and the tooling ecosystem made life easier for our devops and infrastructure teams, making it easier to handle dependencies. Having a standardized build system means best practices are shared easier, and it also plugs nicely into nix, an innovative package manager and build system we use extensively.

The impressive bit was that all of this was achieved without the loss of performance compared to similar components written in C++, and a significant gain in safety and confidence in our code. The test run was highly successful, and soon thereafter Rust has started finding its way into more and more of our software.

At the moment, we have successfully used the language to write everything from number crunching software to HTTP APIs, and we're successfully running them in production environments. The prospects of the language, most notably the WebAssembly backend, keep us excited about extending this reach even further.

If you're interested in building the next-generation TV audience measurement platform and solving interesting problems using Rust, nix, and more — contact us! TVbeat is always hiring.

We are hiring!