Idempotent Pipeline

Definitions

1. A data pipeline that produces the same output regardless of how many times it's run with the same input. Achieved through techniques like UPSERT/MERGE, partition overwriting, and deduplication keys. Essential for safe retries, backfills, and recovery from partial failures.

   In plain English: A data pipeline you can safely re-run without worrying about duplicate or incorrect results.

   Example: "The pipeline uses MERGE with a dedup key — running it twice on the same day's data won't create duplicate rows."

Related Terms

• Idempotent
• Webhook vs. Polling
• Event Sourcing
• Idempotency Key
• Dead Letter Queue
• Slowly Changing Dimension
• Medallion Architecture
• Outbox Pattern