Skip to content

03 Metadata‐driven Orchestration

Jump to bottom
Benny Austin edited this page Oct 26, 2024 · 5 revisions

ELT Framework

The Extract Load Transform (ELT) framework is a metadata-driven orchestration tool designed for modern cloud data platforms. It simplifies ingestion and transformation pipelines, ensuring a consistent development experience and ease of maintenance. The framework supports batch ingestion and is extensively tested with Microsoft Fabric and Azure managed services like Azure Databricks and Azure Synapse. It uses an ANSI-compatible control database as the metadata repository.

Key Features

  • Configurable and Extendable: Easily adapt the framework to meet specific needs.

  • Data Source Agnostic: Ingest data from various sources such as databases, Delta Lake, REST API, flat files, JSON, XML, without storing connection strings as metadata.

  • Delta and Full Loads: Support for both incremental and full data loads.

  • Re-run and Retry Capability: Automatically handle failures without manual intervention.

  • In-built Audit Tracking: Track data processing activities with built-in audit capabilities.

  • Extended Audit Capability: Enhance audit tracking with Azure PaaS services like Diagnostic Logging.

  • Eliminates Manual Data Patching: Streamline data processing by removing the need for manual interventions.

  • Data Lineage Support: Maintain data lineage throughout the data lifecycle.

  • Level 1 and Level 2 Transformations: Support for one-to-many and many-to-many transformations.

  • On-demand Pipeline and Transformation Management: Enable or disable pipelines and transformations as needed.

For more details, visit ELT Framework Wiki

Clone this wiki locally