Flyte

Kubernetes-native workflow orchestration platform built at Lyft for data and ML pipelines, emphasizing reproducibility and type safety.

Visit Website → GitHub

Key Features

✓ Strong typing: Python type hints enforced at runtime
✓ Data classes: Structured inputs and outputs
✓ Schema validation: Automatic type checking
✓ Serialization: Efficient data passing between tasks
✓ Versioning: Every workflow version is immutable
✓ Caching: Memoization of task outputs
✓ Lineage: Complete execution history
✓ Deterministic: Same inputs → same outputs
✓ Container-based: Tasks run in isolated containers
✓ Resource management: CPU, memory, GPU specifications
✓ Spot instances: Cost-optimized execution
✓ Multi-cluster: Federated execution across clusters
✓ Dynamic workflows: Runtime DAG construction
✓ Map tasks: Parallel processing of collections
✓ Plugins: Spark, Ray, Dask, SageMaker, Kubeflow
✓ Model registry: Track and version models