RumbleDB: Data independence for large, messy datasets

We introduce Rumble, a query execution engine for large, heterogeneous, and nested collections of JSON objects built on top of Apache Spark. While data sets of this type are more and more wide-spread, most existing tools are built around a tabular data model, creating an impedance mismatch for both the engine and the query interface. In contrast, Rumble uses JSONiq, a standardized language specifically designed for querying JSON documents. The key challenge in the design and implementation of Rumble is mapping the recursive structure of JSON documents and JSONiq queries onto Spark's execution primitives based on tabular data frames. Our solution is to translate a JSONiq expression into a tree of iterators that dynamically switch between local and distributed execution modes depending on the nesting level. By overcoming the impedance mismatch in the engine, Rumble frees the user from solving the same problem for every single query, thus increasing their productivity considerably. As we show in extensive experiments, Rumble is able to scale to large and complex data sets in the terabyte range with a similar or better performance than other engines. The results also illustrate that Codd's concept of data independence makes as much sense for heterogeneous, nested data sets as it does on highly structured tables.

Abbreviations used in this presentation:

CLI: Command-Line Interface
CSV: Comma-Separated Values
DAG: Directed Acyclic Graph (a graph with no directed cycles, also known as dependency graph)
ETL: Extract-Transform-Load, to import data in a database
FLWOR: for-let-where-orderby-return (pronounced as 'flower')
HDFS: Hadoop Distributed File System, it is an open-source framework for storing very large datasets on a cluster.
HTTP: Hypertext Transfer Protocol, the basis of the Web
JSON: JavaScript Object Notation
RDD: Resilient Distributed Dataset, Spark's data primitive
ROOT: CERN's native format for high-energy physics data.
S3: Simple Storage Service, Amazon's cloud storage service
SQL: Structured english Query Language
UDF: User-Defined Function