Data Pipeline

Definitions

A series of automated data processing steps that moves data from sources through transformations to destinations — extract, transform, load (ETL). The plumbing of the data world: essential, unglamorous, and incredibly fragile when it breaks.
In plain English: An automated system that collects data from various sources, processes it, and delivers it where it needs to go — like an assembly line for data.
Modern data pipelines have shifted from batch ETL (run nightly) to streaming architectures (process events in real-time) using tools like Apache Kafka, Apache Flink, and Spark Streaming. The trend toward real-time processing means pipelines are becoming always-on infrastructure.
Example: 'We replaced our nightly batch pipeline with a streaming pipeline on Kafka. Dashboards now update in seconds instead of showing yesterday's data.'
Source: batch vs. streaming