Skip to content

adb-overwatch

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

adb-overwatch

README.md

Deploying Overwatch on Azure Databricks

This example contains Terraform code used to deploy Overwatch using the following modules :

Example content

This code uses the multi-workspace deployment of Overwatch. Overwatch runs in a dedicated, or existing, Azure Databricks workspace, and monitors the specified workspaces in the config file overwatch_deployment_config.csv. This configuration file is generated automatically by the module adb-overwatch-ws-to-monitor.

Overwatch_Arch_Azure

The deployment is structured as followed :

  • Use an existing Resource group
  • Deploy Eventhubs topic per workspace, that could be in the same Eventhubs namespace
  • Deploy Storage Accounts, one for the cluster logs and one for Overwatch database output
  • Deploy the dedicated Azure Databricks workspace, or use an existing one for Overwatch, with some Databricks quick-start notebooks to analyse the results
  • Deploy Azure Key Vault to store the secrets
  • Configure Role Assignments and mounts to attribute the necessary permissions
  • Configure Diagnostic Logs on the Databricks workspaces to monitor

Note
As Terraform requires providers and modules to be declared statically before deploying the resources, we are using in this example a bash script that generates the provider configurations for N workspaces along with the modules references.

How to use

  1. Configure the workspaces that will be observed by Overwatch in workspaces_to_monitor.json
  2. Make the script dynamic_providers_modules_generation.sh executable : chmod +x dynamic_providers_modules_generation.sh
  3. Update the terraform.tfvars file with your environment values
  4. Run the script dynamic_providers_modules_generation.sh : ./dynamic_providers_modules_generation.sh. This will dynamically generate providers_ws_to_monitor.tf and main_ws_to_monitor.tf files with the right terraform setup for all the workspaces defined in workspaces_to_monitor.json
  5. Run terraform init to initialize terraform and get provider ready
  6. Run terraform plan to check the resources that are affected
  7. Run terraform apply to create the resources

Requirements

No requirements.

Providers

Name Version
random n/a

Modules

Name Source Version
adb-overwatch-analysis ../../modules/adb-overwatch-analysis n/a
adb-overwatch-main-ws ../../modules/adb-overwatch-main-ws n/a
adb-overwatch-mws-config ../../modules/adb-overwatch-mws-config n/a
adb-overwatch-regional-config ../../modules/adb-overwatch-regional-config n/a

Resources

Name Type
random_string.strapp resource

Inputs

Name Description Type Default Required
ehn_name Eventhubs namespace name string n/a yes
key_vault_prefix AKV prefix string n/a yes
logs_sa_name Logs storage account name string n/a yes
overwatch_spn_app_id Azure SPN application ID string n/a yes
overwatch_spn_secret Azure SPN secret string n/a yes
overwatch_ws_name Overwatch Databricks workspace name string n/a yes
ow_sa_name Overwatch ETL storage account name string n/a yes
rg_name Resource group name string n/a yes
subscription_id Azure subscription ID string n/a yes
tenant_id Azure tenant ID string n/a yes
active Whether or not the workspace should be validated / deployed bool true no
api_waiting_time API Tunable - Overwatch makes async api calls in parallel, api_waiting_time signifies the max wait time in case of no response received from the api call string "" no
auditlog_prefix_source_path Location of auditlog (AWS/GCP Only) string "" no
automated_dbu_price Contract price for automated DBUs number 0.3 no
databricks_secret_scope_name Databricks secret scope name (backed by Azure Key-Vault) string "overwatch-akv" no
enable_unsafe_SSL API Tunable - Enables unsafe SSL string "" no
error_batch_size API Tunable - Indicates the size of the error writer buffer containing API call errors string "" no
excluded_scopes Scopes that should not be excluded from the pipelines string "" no
interactive_dbu_price Contract price for interactive DBUs number 0.55 no
jobs_light_dbu_price Contract price for interactive DBUs number 0.1 no
max_days This is the max incremental days that will be loaded. Usually only relevant for historical loading and rebuilds number 30 no
proxy_host Proxy url for the workspace string "" no
proxy_password_key Key which contains proxy password string "" no
proxy_password_scope Scope which contains the proxy password key string "" no
proxy_port Proxy port for the workspace string "" no
proxy_user_name Proxy user name for the workspace string "" no
sql_compute_dbu_price Contract price for DBSQL DBUs number 0.22 no
success_batch_size API Tunable - Indicates the size of the buffer on filling of which the result will be written to a temp location. This is used to tune performance in certain circumstance string "" no
thread_pool_size API Tunable - Max number of API calls Overwatch is allowed to make in parallel string "" no
use_existing_overwatch_ws Overwatch ETL storage prefix, which represents a mount point to the ETL storage account string false no

Outputs

No outputs.