Luigi

Python-based workflow orchestration framework from Spotify for building complex data pipelines with dependency resolution.

Visit Website → GitHub

Key Features

✓ Python tasks: Define work units as classes
✓ Dependencies: Automatic resolution and ordering
✓ Targets: Output tracking (files, databases)
✓ Parameters: Configurable task inputs
✓ Hadoop: HDFS and MapReduce support
✓ Spark: PySpark task support
✓ Databases: PostgreSQL, MySQL, Redshift
✓ Cloud: S3, GCS file targets
✓ Central scheduler: Coordinates task execution
✓ Worker processes: Parallel execution
✓ Retries: Configurable failure handling
✓ Idempotency: Target-based completion tracking
✓ Dependency graph: Visual task relationships
✓ Task history: Execution timeline
✓ Web interface: Basic monitoring UI