Spark Streaming

Apache Spark's stream processing engine providing unified batch and streaming analytics with exactly-once semantics.

Visit Website → GitHub

Key Features

✓ Same API: DataFrame/Dataset for batch and streaming
✓ Code reuse: Write once, run batch or streaming
✓ Consistent semantics: Same results either way
✓ Easy migration: Convert batch to streaming easily
✓ Declarative API: SQL-like operations
✓ Event time: Watermarks and late data handling
✓ Exactly-once: End-to-end guarantees
✓ Continuous processing: Sub-millisecond latency mode
✓ Micro-batch: Default, ~100ms latency
✓ Continuous: Experimental ~1ms latency
✓ Triggers: Once, processing time, available-now
✓ Kafka: Native connector
✓ Kinesis: AWS streaming
✓ Event Hubs: Azure streaming
✓ Files: S3, HDFS, cloud storage