Future Data Systems: Unbundling & Asynchronous Derivation

Future data systems will assemble specialized components via data flow, deriving data asynchronously from an immutable event log, rather than relying on monolithic databases and brittle distributed transactions.

Core Principles

• No single tool fits all use cases; assemble specialized components.
• Future systems rely on data flow, not monolithic databases.
• Derive data asynchronously from a primary source of truth.
• Event logs provide order and consistency for writes.
• Avoid distributed transactions (2PC) due to fragility.
• Route all writes through one system that defines order.
• Unbundle databases into best-of-breed tools.
• Unified read interfaces (polystore) and unified writes via event logs are key strategies.
• Batch and stream processing can be unified.
• Write Path is eager (pre-compute), Read Path is lazy (query on demand).
• Integrity can be achieved without distributed transactions using end-to-end IDs, deterministic derivation, and event sourcing.
• Ethical considerations are paramount in data system design.

Action Steps

• Route all writes through a single system that defines order.
• Derive all other data asynchronously.
• Unbundle traditional database functionalities into specialized tools.
• Implement unified read interfaces (polystore) or unified writes via event logs.
• Consider unified batch and stream processing (e.g., Kafka + Flink/Beam).
• Shift work from the read path to the write path using indexes.
• Pre-compute frequent queries using caches.
• Use end-to-end request IDs for idempotence.
• Ensure deterministic derivation (same input -> same output).
• Implement asynchronous constraint checking.
• Utilize event sourcing for an immutable, replayable log.
• Delete data when no longer needed.
• Build auditing into systems.
• Respect user dignity over optimization.
• Enable user control over their data, do not steal it.

Key Terms

• Derived data: Data created from primary data, such as indexes, caches, views, and ML models.
• Primary system: The main source of truth for data, often an OLTP database.
• CDC (Change Data Capture): Process that captures database writes in order to update derived systems.
• Event log: A record that determines the total order of writes, ensuring consistency.
• 2PC (Two-Phase Commit): A type of distributed transaction protocol known for being fragile and having poor fault tolerance.
• Polystore: A system providing a unified read interface across multiple underlying databases.
• Lambda Architecture: An older pattern using separate batch and stream processing layers, leading to complex, duplicated logic.
• Event sourcing: A pattern where all changes to application state are stored as a sequence of events in an immutable log.
• Idempotence: The property that an operation can be applied multiple times without changing the result beyond the initial application.
• Hash trees (Merkle trees): Cryptographic data structures used for efficient and secure verification of data integrity, commonly used in blockchains.

Pro Tips

• Embrace 'best-of-breed' tools for each data task.
• Leverage event logs for robust, ordered writes.
• Consider 'apologies over coordination' for temporary constraint violations.
• Unified systems (Kafka + Flink/Beam) simplify complex architectures.
• Shift work to the write path (indexing, caching) for faster reads.
• Event sourcing is powerful for auditing and replayability.
• Hash trees (Merkle trees) provide cryptographic verification.

Pitfalls to Avoid

• Relying on monolithic databases for all needs.
• Using brittle distributed transactions (2PC).
• Duplicate logic in separate batch and stream systems (Lambda Architecture).
• Expensive reads with no indexing (like grep).
• Ignoring the ethical implications of data collection and usage.
• Treating privacy as secrecy rather than control over disclosure.

Myth vs Reality

• Distributed transactions (2PC) are necessary for data integrity.: Integrity can be achieved through end-to-end IDs, deterministic derivation, asynchronous constraint checking, and event sourcing, avoiding the fragility of 2PC.
• Batch and stream processing must be run as separate, complex systems.: Unified systems like Kafka + Flink/Beam can handle both, enabling features like event replay and exactly-once semantics.
• Privacy means data should be secret.: Privacy is about control over disclosure; users should have agency over how their data is used.

Real World Examples

• Building a real-time analytics dashboard.: Use CDC to capture writes from the primary database, process them through a stream processor (like Flink), and update materialized views or caches for low-latency querying.
• Implementing a user profile service.: Store user data in a primary database. Use an event log to capture updates. A stream processor can then derive search indexes (for quick lookups) and update caches for frequently accessed profiles.
• Ensuring data consistency across microservices.: Instead of distributed transactions, use an immutable event log. Each service can consume events relevant to it and update its local state deterministically.

People

• Bruce Schneier: Security expert and author quoted on data privacy as an environmental policy challenge.