Data Engineering Pipelines: Reliability, Quality, and Evolution

Data pipelines fail when treated as one-off ETL scripts. Production failures are almost never caused by wrong SQL — they are caused by late data that arrives after a window closes, schema drift from an upstream team that added a nullable column without announcing it, silent data quality failures (row counts look fine but 10% of records have NULL user_ids), and unrecoverable state when a streaming job checkpointed corrupted state and cannot replay.

Strong engineers treat pipelines as products with explicit contracts, SLAs, and operational runbooks. The key questions to answer before writing code: Who are the downstream consumers? What is the freshness SLA (15 minutes? 24 hours?)? What is the completeness SLA (99.9% of events, or all-or-nothing)? How do we detect a silent failure? How do we backfill 30 days of historical data if we change the schema?

The candidate gap: A junior engineer writes a Spark job that reads from S3 and writes to a warehouse. A senior engineer defines a data contract, adds quality gates at ingestion, designs for idempotent replay, and instruments freshness and row-count metrics. A staff engineer discusses schema registry integration, exactly-once semantics in Flink, late-data handling with event-time watermarks, and the architectural trade-offs between Lambda and Kappa.

The choice of execution model is the first and most consequential pipeline design decision. Most teams default to batch out of familiarity, then retrofit streaming when freshness requirements tighten — which is the wrong order.

Batch (Spark, Hive, dbt): Processes a bounded dataset (yesterday's events, the last hour of logs) on a schedule. Strengths: simple to reason about, cheap to operate, easy to backfill (re-run the job with a different date partition), and Spark's optimizer produces efficient execution plans for large joins. Weaknesses: data is stale by definition. A nightly batch job produces metrics 24 hours late. For ML feature pipelines serving real-time inference, this is unacceptable.

Streaming (Flink, Kafka Streams, Spark Structured Streaming): Processes an unbounded event stream with millisecond-to-second latency. Strengths: low end-to-end latency, natural fit for event-driven architectures. Weaknesses: operationally complex (checkpointing, exactly-once semantics, state management), harder to backfill (requires replaying from Kafka's retention window or a separate historical source), and joins across streams require windowing with associated late-data risk.

Micro-batch (Spark Structured Streaming in trigger interval mode): Processes small batches every 1–5 minutes. The sweet spot for teams that need near-real-time freshness but want Spark's familiar APIs and SQL support. Latency is higher than true streaming (1–5 min vs. milliseconds) but dramatically simpler to operate. Uber's analytics platform uses micro-batch for most ML feature pipelines where 5-minute freshness is sufficient.

Decision rule: Ask "what is the freshness SLA?" If > 1 hour: batch. If 1–60 minutes: micro-batch. If < 1 minute: streaming. The majority of "we need real-time" requirements are actually micro-batch once you push on the actual business need.

Many production data platforms are built on one of two architectural patterns for serving both real-time and historical queries.

Lambda architecture maintains two separate processing paths: a batch layer (Spark jobs over the full historical dataset, producing accurate but stale metrics) and a speed layer (Flink or Kafka Streams over the live event stream, producing fast but potentially approximate metrics). A serving layer merges results from both. The batch layer periodically overwrites the speed layer's output, correcting any approximations.

Lambda's strength: the batch layer produces accurate numbers from complete data; the speed layer provides freshness. Its weakness: two codebases for the same business logic. When the definition of "daily active users" changes, both the Spark job and the Flink job must be updated, tested, and deployed in sync. LinkedIn's early data platform suffered from this — business logic drifted between the two layers, producing different numbers from the "same" metric depending on which layer you queried.

Kappa architecture eliminates the batch layer. All processing goes through the stream processor (Flink). Historical reprocessing is done by replaying from Kafka (requires long retention — 30–90 days — or a separate event store like S3). The same pipeline code processes both live events and replayed historical events.

Kappa's strength: one codebase, one definition of truth. Its weakness: Kafka retention for 30+ days is expensive, and reprocessing 30 days of data through Flink takes significant cluster time. For exploratory analytics (ad-hoc SQL on historical data), Kappa requires materializing results to a query-friendly store (Iceberg, Delta Lake).

In practice: Most large organizations use a hybrid. Flink for real-time signals, Spark for historical backfill and complex joins, unified business logic in a shared library, and Iceberg/Delta as the unifying table format that both can write and read. This is what Airbnb's Minerva metric platform does.

Silent data quality failures are the most dangerous pipeline failure mode because they don't page anyone. The row count looks fine; the latency is normal; but 15% of user_id fields are NULL because a mobile SDK update sent malformed events. Downstream ML models silently degrade.

Four mandatory quality dimensions at ingestion:

1. Completeness: Are all expected records present? For event pipelines: compare the event count in Kafka vs. the count written to the warehouse. A >2% gap triggers an alert. For batch pipelines: compare today's partition row count against the 7-day moving average. A >5% deviation is anomalous.

2. Validity: Do values conform to the schema and business rules? Null rate per critical column (null user_id should be <0.1%), enum value coverage (all event_type values must be in the known set), referential integrity (every restaurant_id in order events must exist in the restaurant table).

3. Uniqueness: Are there duplicate records? Event deduplication requires a unique event ID. COUNT(*) vs COUNT(DISTINCT event_id) should differ by <0.01%. Duplicates cause double-counting in metrics and can corrupt ML training labels.

4. Timeliness: Did the data arrive within the SLA? For a batch job that should produce output by 6am, a late-arrival monitor alerts at 6:15am if the Iceberg partition is not yet written.

Implementation — Great Expectations or custom: LinkedIn's DataHub and Airbnb's Minerva both use a quality framework layered over Spark jobs. Each quality check is a function that returns (passed: bool, metric_value: float, threshold: float). Failing checks surface in a quality dashboard and, for critical pipelines, fail the Airflow DAG before writing to the downstream table — catching bad data before it contaminates downstream consumers.

Schema drift is the second most common cause of pipeline failures after late data. An upstream service adds a new required field to an Avro schema. The Flink consumer that expects the old schema cannot deserialize the new message. The job throws SchemaDeserializationException at 3am. The on-call engineer wakes up to a 4-hour backlog.

Schema registry solves this by versioning schemas and enforcing compatibility at publish time. Confluent Schema Registry stores schemas by subject (typically <topic>-value). Every Kafka producer registers its schema before publishing; every consumer fetches the schema by ID embedded in the message header.

Three compatibility modes:

• BACKWARD (recommended): New schema can read data written with the old schema. Consumers can upgrade first, then producers. Safe for adding optional fields with defaults. Does NOT allow removing fields or making optional fields required.
• FORWARD: Old schema can read data written with the new schema. Producers can upgrade first. Allows removing fields (consumers ignore unknown fields).
• FULL: Both BACKWARD and FORWARD. Most restrictive. Required for Avro schemas shared across teams with independent release cycles.

Schema evolution rules for Avro:

• ✅ Add a field with a default value ("default": null)
• ✅ Remove a field that had a default value
• ❌ Remove a required field (breaks backward compatibility)
• ❌ Rename a field (treated as remove + add)
• ❌ Change a field's type (e.g., int → long without union)

CI enforcement: Add a schema compatibility check to the CI pipeline. Before merging a schema change, the CI job registers the proposed schema against the registry in dry-run mode. If it fails the compatibility check, the merge is blocked. This prevents 3am incidents from schema drift — a pattern used by LinkedIn for their internal Avro schemas.

Streaming pipelines have three delivery guarantees: at-most-once (fast, may lose events), at-least-once (reliable, may duplicate events), and exactly-once (no loss, no duplication — the hardest). Most teams think they need exactly-once; in practice, idempotent at-least-once is sufficient for most use cases and much simpler to operate.

Why at-least-once is the common default: When a Flink job fails and restarts from a checkpoint, it replays events from the Kafka offset recorded in the checkpoint. Any events processed between the last checkpoint and the failure are reprocessed — causing duplicates. If the sink (e.g., Iceberg, Kafka output topic) is idempotent (writing the same event twice produces the same result), at-least-once is functionally exactly-once.

Flink's exactly-once with two-phase commit: Flink achieves true exactly-once by coordinating checkpointing with the sink. The Kafka sink uses a transactional producer. When Flink initiates a checkpoint:

1. Flink pauses processing and sends a checkpoint barrier through the event stream
2. Each operator snapshots its state to durable storage (S3 or HDFS)
3. The Kafka sink pre-commits its transactional producer (writes are buffered, not yet visible to consumers)
4. Once all operators confirm their checkpoint is durable, the Kafka sink commits the transaction — records become visible
5. Flink records the Kafka output offset and the input offset together in the checkpoint

On failure, Flink restores state from the last checkpoint. The Kafka output transaction for any incomplete checkpoint is aborted — its records were never visible. The input is replayed from the checkpointed offset. Result: exactly once, end to end.

Checkpointing configuration in Flink:

env.enable_checkpointing(60_000)  # checkpoint every 60 seconds
env.get_checkpoint_config().set_checkpoint_storage("s3://my-bucket/checkpoints")
env.get_checkpoint_config().set_min_pause_between_checkpoints(30_000)
env.get_checkpoint_config().set_max_concurrent_checkpoints(1)

The checkpoint interval is a latency/durability trade-off: shorter intervals mean less data to replay on failure but more checkpoint overhead.

When NOT to use exactly-once: The two-phase commit adds latency (records are not visible until the next checkpoint commit) and reduces throughput by ~10–20%. For use cases like clickstream analytics where a 0.1% duplication rate is acceptable, exactly-once is overkill. Use at-least-once with an idempotent sink instead.

Backfills are the most dangerous operation in a data engineering pipeline because they write to partitions that downstream consumers are already reading. A poorly designed backfill corrupts production data.

Partition-based backfill (batch): For Spark pipelines writing to Iceberg or Hive, backfill by reprocessing one partition (typically one day) at a time. The Airflow DAG accepts a start_date and end_date parameter and loops over partitions sequentially. Write to a staging location first (s3://data/staging/table/event_date=2025-01-01), validate quality, then atomically swap into the production location with ALTER TABLE ... ADD PARTITION.

Shadow write pattern: For critical tables, run the new pipeline in parallel with the old pipeline for 7–14 days. Compare row counts, key metrics, and sample records between the two outputs. Only cut over to the new pipeline when you have high confidence in its correctness. Netflix uses shadow writes for all major pipeline migrations.

Idempotency is the prerequisite: A backfill job must be safely re-runnable. If re-running the job writes duplicate rows, the backfill causes more damage than it fixes. Design for idempotency: use INSERT OVERWRITE PARTITION (not INSERT INTO) so re-running the job replaces the partition rather than appending to it.

Streaming backfill from Kafka: If Kafka retention is sufficient (7–30 days), replay the consumer group from the earliest offset. For historical data older than Kafka retention, replay from S3 archive using the same Flink job with a FileSource connector — the pipeline code is identical, only the source changes. This is the Kappa architecture advantage.

Coordinating with downstream consumers: Notify downstream consumers before starting a backfill on a partition they depend on. If a downstream Airflow DAG reads the same Iceberg table partition, it may read a partially-written partition during backfill. Use Iceberg's snapshot isolation: the downstream consumer reads the snapshot that existed before the backfill started, while the backfill writes a new snapshot. The downstream consumer upgrades to the new snapshot only after the backfill is complete and validated.

Freshness lag is the most important pipeline health metric. Define it as: freshness_lag = current_time - max(event_time) in latest partition. For a pipeline with a 30-minute SLA, alert if freshness_lag > 45 minutes. This catches: pipeline failures, source slowdowns, and Kafka consumer lag accumulation.

Row count anomaly detection: Day-over-day row count should be within a defined band (typically ±10% for stable data, ±30% for event-driven data). Compute the 7-day median row count for each partition and alert if the current partition is outside [median × 0.7, median × 1.3]. This catches silent failures where a pipeline ran but produced 10% of expected output (e.g., a filter bug).

Per-column null rate monitoring: Track null_rate for every non-nullable business key (user_id, order_id, event_type) in every partition. Alert if null_rate rises above 0.1%. A sudden jump from 0.01% to 5% null user_ids indicates a source schema change or a bug in the enrichment step.

Airflow SLA miss callbacks: Airflow's sla parameter on a task triggers a callback if the task doesn't complete within the defined window. Use this to page the on-call engineer before downstream consumers start their morning jobs:

t = SparkSubmitOperator(
    task_id="daily_orders_agg",
    sla=timedelta(hours=2),
    sla_miss_callback=pagerduty_alert,
)

Kafka consumer group lag: For streaming pipelines, monitor consumer group lag (messages in Kafka not yet consumed). A growing lag means the Flink job is falling behind. Alert if lag exceeds 5 × average_throughput_per_minute (i.e., more than 5 minutes of unprocessed events).