Abstract |
Continuous data stream processing systems have offered limited support for data persistence in the past, for three main raisons: First, online, real-time queries examine current streaming data and under the assumption of no server failures, do not require access to past data. Second, most fault-tolerance solutions for streaming systems are currently memory-based, keeping state replicas in the memory of stream processing nodes. Finally, stable storage devices are commonly thought to be constraining system throughput and response times when compared to main memory, and are thus kept off the common path.
In this thesis we provide data streaming systems with a scalable path to persistent storage. Our persistance path allows two fundamental enhancements to data streaming systems' capabilities: First, it allows stream persistence for reference/archival purposes. In other words, queries can now be applied an past data on-demand. Second, fault tolerance is achievable by checkpointing and stream replay schemes that are not constrained by the size of main memory. For stateful operators we present continuous eventual checkpoints (CEC), a novel mechanism to provide fault-tolerant guarantees by taking continuous state checkpoints without pausing the operator. We achieve this by separating state into parts and by treating these parts independently.
Our results show that our system can achieve scalable stream persistence over a high-performance tunable paralles file system and low overhead fault-tolerance with an adjustable checkpoint inetrval, trading off recovery time with performance.
|