Dataform
SQL-based data transformation framework for BigQuery with dependency management, testing, and documentation. Now part of Google Cloud.
✨ Key Features
- ✓ SQL + JavaScript: Extend SQL with JS functions
- ✓ ref() function: Automatic dependency management
- ✓ config blocks: Define table properties
- ✓ Jinja-like templating: Dynamic SQL generation
- ✓ Web IDE: Browser-based development
- ✓ Git integration: Native version control
- ✓ Compilation: Validate before running
- ✓ Preview: See generated SQL
- ✓ Assertions: Row-level data tests
- ✓ Schema tests: Not null, unique, relationships
- ✓ Custom tests: SQL-based assertions
- ✓ Freshness checks: Data recency validation
- ✓ Auto-generated docs: Table and column descriptions
- ✓ Lineage graphs: Visual dependency tracking
- ✓ Data dictionary: Centralized metadata
📁 More Transformation Tools
Coalesce
Visual data transformation for Snowflake
dbt
Transform data in your warehouse using SQL and software engineering best practices
Pandas
Polars
Lightning-fast DataFrame library for Rust and Python