Dataform

SQL-based data transformation framework for BigQuery with dependency management, testing, and documentation. Now part of Google Cloud.

Visit Website → GitHub

Key Features

✓ SQL + JavaScript: Extend SQL with JS functions
✓ ref() function: Automatic dependency management
✓ config blocks: Define table properties
✓ Jinja-like templating: Dynamic SQL generation
✓ Web IDE: Browser-based development
✓ Git integration: Native version control
✓ Compilation: Validate before running
✓ Preview: See generated SQL
✓ Assertions: Row-level data tests
✓ Schema tests: Not null, unique, relationships
✓ Custom tests: SQL-based assertions
✓ Freshness checks: Data recency validation
✓ Auto-generated docs: Table and column descriptions
✓ Lineage graphs: Visual dependency tracking
✓ Data dictionary: Centralized metadata