LogScale's High-Availability and Disaster Recovery implementation ensures data resilience through a robust combination of Kafka integration and file system storage mechanisms, allowing for reliable data replication and recovery across different deployment configurations. The system provides flexible options for data redundancy through both bucket storage and non-bucket storage implementations, while maintaining data integrity during node failures through careful management of event processing, ingest operations, and search functionality.

This document outlines LogScale's approach to ensuring data resilience and high availability through its integration with Kafka and file system storage mechanisms. It explains how LogScale manages data replication, event processing, and storage across different deployment configurations, including implementations with and without bucket storage. The documentation covers the system's ability to maintain data integrity during node failures, the role of Kafka in temporary storage, and the replication processes for segment files. Key topics include ingest operations, digest and search functionality, and how the system handles data availability in various scenarios. The content specifically addresses how LogScale can be configured for different levels of data redundancy and disaster recovery, depending on whether bucket storage is utilized or not, and explains the operational requirements for maintaining system availability during both normal operations and failure scenarios.

LogScale uses a combination of replicated files and data in Kafka topics to make sure to not lose any data. LogScale relies on Kafka as the commit-log for recent and in-progress work, and local file systems (or Bucket Storage) as the trusted persistent long term storage.

When events or requests arrive on the HTTP API, the 200 OK message is issued only after Kafka has acknowledged all writes that stem from them. This delegates the responsibility of not losing incoming changes to Kafka which is a proven solution for this task.

When the digest nodes construct segment files from incoming events, they include the offsets from the Kafka partition the events went into as part of the meta data in Global. Segment files then get replicated to other nodes in the cluster. The cluster calculates the offset for each partition where all events are in files that are properly replicated and then tells Kafka that it is okay for Kafka to delete events older than that.

LogScale can be configured to be resilient against data loss in more than one way depending on the hosting environment.

In all cases the hosting environment must ensure that Kafka has sufficient replicas and fail-over nodes to support the desired resilience.