Environment Setup¶

This section outlines the steps required to setup the development environment to use this dbt-adapter as part of a dbt data transformation project.

To provide a common, cross-platform set of instructions we will first install Powershell. To facilitate the installation process we will use package managers such as winget for Windows, brew for MacOS and apt for Linux.

Core Tools Installation¶

Tip

Following core tools can be installed using the standard powershell or in VS Code terminal (powershell)

WindowsMacOSLinux

# Winget Installs 
winget install Microsoft.PowerShell

brew install powershell/tap/powershell

# TBA

Next we will install Python and development tools such as vscode.

WindowsMacOSLinux

# Winget Installs 
winget install Python.Python.3.12
winget install -e --id Microsoft.VisualStudioCode
winget install --id Git.Git -e --source winget

# Python Environment Manager
Python -m pip install --user virtualenv

# Brew Installs
brew install python@3.12
brew install --cask visual-studio-code
brew install git

# Python Environment Manager
Python -m pip install --user virtualenv

# TBA

Other tools¶

Now that we have pwsh installed, Make sure that you have install the following additional required tools.

Install Azure PowerShell on Windows - Refer Azure Powershell Doc for windows (You might need to run this in admin mode)

Important

Optional packages you may need to install (Only run if you face issues)

pip config set global.trusted-host "pypi.org files.pythonhosted.org pypi.python.org"

Source Directory & Python Env¶

Now lets create and activate our Python environment and install the required packages.

Tip

When executing the following, it can take a few minutes to complete on some machines. Occasionally pip may get stuck and in such cases break the execution using ctrl-c and run the same pip again.

WindowsMacOSLinux

# Ensure that you are in the pwsh shell
pwsh

# Create a new source code directory
mkdir dbt-fabricsparknb-test #Note that the name of the directory is arbitrary... call it whatever you like
# Navigate to the new directory
cd dbt-fabricsparknb-test

# Create the Python environment
python -m venv .env

#Optional step to run if activate.ps1 failes due to security policy
Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser

# Activate the Python environment
./.env/Scripts/Activate.ps1

# Install the dbt-fabricsparknb package from the repository
pip install --upgrade --force-reinstall git+https://github.com/Insight-Services-APAC/APAC-Capability-DAI-DbtFabricSparkNb

# Ensure that you are in the pwsh shell
pwsh

# Create a new source code directory
mkdir dbt-fabricsparknb-test #Note that the name of the directory is arbitrary... call it whatever you like
# Navigate to the new directory
cd dbt-fabricsparknb-test

# Create the Python environment
python -m venv .env

#Optional step to run if activate.ps1 failes due to security policy
Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser

# Activate the Python environment
./.env/Scripts/Activate.ps1  

# Install the dbt-fabricsparknb package from the repository
pip install --upgrade --force-reinstall git+https://github.com/Insight-Services-APAC/APAC-Capability-DAI-DbtFabricSparkNb

# TBA

Tip

To obtain a specific or the latest version of the framework packages in dbt, you need to specify the framework tags as following (example: for version 0.4.0)

pip install --upgrade --force-reinstall git+https://github.com/Insight-Services-APAC/APAC-Capability-DAI-DbtFabricSparkNb@V0.4.0

Info

You are now ready to move to the next step in which you will set up your dbt project. Follow the Dbt Project Setup guide.