Flyte
Kubernetes-native workflow orchestration platform built at Lyft for data and ML pipelines, emphasizing reproducibility and type safety.
✨ Key Features
- ✓ Strong typing: Python type hints enforced at runtime
- ✓ Data classes: Structured inputs and outputs
- ✓ Schema validation: Automatic type checking
- ✓ Serialization: Efficient data passing between tasks
- ✓ Versioning: Every workflow version is immutable
- ✓ Caching: Memoization of task outputs
- ✓ Lineage: Complete execution history
- ✓ Deterministic: Same inputs → same outputs
- ✓ Container-based: Tasks run in isolated containers
- ✓ Resource management: CPU, memory, GPU specifications
- ✓ Spot instances: Cost-optimized execution
- ✓ Multi-cluster: Federated execution across clusters
- ✓ Dynamic workflows: Runtime DAG construction
- ✓ Map tasks: Parallel processing of collections
- ✓ Plugins: Spark, Ray, Dask, SageMaker, Kubeflow
- ✓ Model registry: Track and version models
📁 More Orchestration Tools
Apache Airflow
The industry-standard workflow orchestration platform
Argo Workflows
Dagster
Cloud-native orchestration platform for data pipelines
dbt Cloud