Focused on:
- Extracting raw data through Azure Databricks notebooks from Azure Data Lake Gen2 Storage.
- Transforming data through Azure Databricks notebooks.
- Loading transformed data into Databricks SQL DB.
- Create production ready pipelines by integrating Databricks notebook into Azure Data Factory.
- Monitor pipelines through Azure Monitor and Log Analytics.