Future Data Systems: Unbundling & Asynchronous Integrity

Future data systems will assemble specialized components via data flow, leveraging asynchronous processing and immutable logs for integrity, rather than relying on monolithic databases and distributed transactions.

Core Principles

• No single tool fits all use cases; assemble specialized components.
• Data flow is key, not monolithic databases.
• Route all writes through one system that defines order.
• Derive everything else asynchronously.
• Unbundle databases into best-of-breed tools.
• Use event logs for unified writes and auditing.
• Embrace asynchronous constraint checking over distributed transactions.
• Prioritize ethical data handling and user privacy.

Action Steps

• Route all writes through a single, ordered system.
• Capture writes using Change Data Capture (CDC) or event logs.
• Derive specialized views (indexes, caches, ML models) asynchronously.
• Implement end-to-end request IDs for idempotence.
• Utilize deterministic derivation for consistent outputs.
• Employ asynchronous constraint checking for integrity.
• Build auditing into systems by recomputing derived data.
• Delete data when no longer needed.
• Respect user dignity and enable control over data.

Key Terms

• Derived data: Indexes, caches, views, ML models created from primary data.
• Primary system: The source of truth, typically an OLTP database.
• CDC (Change Data Capture): Captures writes in order to update derived systems.
• Event log: Determines the total order of writes, ensuring consistency.
• 2PC (Two-Phase Commit): A distributed transaction protocol, considered fragile and prone to poor fault tolerance.
• Unbundled Approach: Using separate, best-of-breed tools for different database functions (e.g., separate search index, stream processor).
• Federated DB (Polystore): A unification strategy providing a unified read interface across multiple databases.
• Unbundled DB: A unification strategy using event logs for unified writes.
• Lambda Architecture: An older pattern running batch and stream processing separately, leading to complex, duplicate logic.
• Unified Future: A modern approach where a single system (e.g., Kafka + Flink/Beam) handles both batch and stream processing.
• Write Path (eager): Pre-computes data and updates indexes during write operations.
• Read Path (lazy): Queries data on demand, potentially performing computations or searches during read operations.
• Linearizability: Strict recency, ensuring operations appear to take effect instantaneously in a global order.
• Idempotence: Ensuring that an operation can be applied multiple times without changing the result beyond the initial application.
• Deterministic derivation: A process where the same input always produces the same output.
• Event sourcing: Storing all changes to application state as a sequence of immutable events.
• Hash trees (Merkle trees): Cryptographic data structures used for efficient and secure verification of data integrity.

Pro Tips

• Use event logs (Unix-like pipes) for unified writes.
• Consider a unified read interface (polystore) for federated databases.
• The unified future uses systems like Kafka + Flink/Beam for both batch and stream.
• Shift work from the read path to the write path using indexes.
• Cache frequent queries for performance.
• Temporary constraint violations ('apologies') are often cheaper than strict linearizability.
• Client subscriptions to change streams enable real-time, offline-capable UIs.
• Hash trees (Merkle trees) provide cryptographic verification for auditing.

Pitfalls to Avoid

• Relying on monolithic databases.
• Using distributed transactions (2PC) due to fragility and poor fault tolerance.
• Complex, duplicate logic in Lambda Architecture.
• Ignoring the ethical implications of data systems.
• Treating privacy as secrecy rather than control over disclosure.

Myth vs Reality

• Strict linearizability is always necessary for system integrity.: Temporary constraint violations ('apologies') are often acceptable and cheaper than strict linearizability, especially when dealing with overbooking or out-of-stock scenarios.
• Eventual consistency is the same as eternal inconsistency.: Eventual consistency is annoying but temporary, while eternal inconsistency implies catastrophic, permanent data loss or corruption.

Real World Examples

• Building a real-time analytics dashboard.: Use CDC to capture writes from the primary database, stream them through a processor (like Flink), and update materialized views or caches for low-latency reads.
• Implementing a user profile system.: Use an event log to record all user actions. Derive user profile data asynchronously based on these events, ensuring consistency via end-to-end IDs.
• Handling e-commerce order processing.: Route all order writes through a single system. Accept temporary overbooking ('apologies') for inventory checks, rather than using fragile distributed transactions.
• Creating a search engine.: Unbundle the database by using a separate search index (like Elasticsearch) instead of relying on built-in database indexes.

People

• Bruce Schneier: Security technologist and author, quoted on data privacy.