Skip to content

Development Workflow

Development and Deployment Flow Using Fabric Spark Notebook Adapter

Advantages

  • Available today
  • Native Fabric Notebooks Generated and Deployed
    1. Non dbt users able to view notebooks and business logic
    2. Monitoring and debugging of loads directly in Fabric without the need for a separate tool
  • Re-occurring loads achieved using native Fabric scheduling
  • Simplified code promotion process using native Fabric Git integration
  • No need for dbt hosted in a virtual machine
    1. No need for service account
    2. No need for Azure Landing Zone
    3. No need for secure network connectivity between Azure VM and Fabric
  • Allows for disconnected development environment providing
    1. Faster DBT build times
    2. Greater developer flexibility
  • Simplified code promotion Process using native Fabric Git integration
    1. Single, native promotion process for all Fabric artifacts including non-dbt ones

Disadvantages

  • Requires additional steps to extract metadata and generate notebooks but this is mitigated by our wrapper script that automates these.

Development and Deployment Flow Using Original Fabric Spark Adapter

Detailed Workflow

Inital Setup 1. Provision Workspace - Development Environment: Fabric Portal - Re-occurence: Do once per development environment set-up - Instructions: Create a new workspace in the Power BI Portal, or use an existing workspace.

  1. Get Workspace Connection Details
  2. Development Environment: Fabric Portal
  3. Re-occurence: Do once per development environment set-up

  4. Instructions: Get the workspace connection details from the Power BI Portal.

  5. Create or Update profiles.yml

  6. Development Environment: VS Code on local, developemnt machine

  8. Create or Update dbt_project.yml

  9. Build Project
  10. Manually Upload Notebooks
  11. Run Meta Data Extract

Ongoing Development Cycle

  1. Download Metadata:

  2. Update Dbt Project

  3. Build Dbt Project
  4. Verify Outputs
  5. Update Notebooks
    1. Upload to Onelake
    2. Update to GIT repo

  6. Promote to Workspace
    1. Run Import Notebook
    2. Promote GIT branch

  7. Run Master Notebook
  8. Validate Results
  9. Run Metadata Extract