-
Notifications
You must be signed in to change notification settings - Fork 0
z Need to be able to push data from ADF into a RDBMS in the workspace.
Azure Data Factory (ADF) provides a seamless way to push data into a relational database management system (RDBMS) within a workspace, enabling efficient data integration and transformation. By leveraging ADF's built-in connectors, data pipelines can be created to extract, transform, and load (ETL) data from various sources into an RDBMS. This can be done using activities like Copy Data, which allows for bulk data movement, or Data Flow, which supports complex transformations before inserting data into the target database.
To establish a connection between ADF and the RDBMS in the TRE workspace, a linked service, a private endpoint, and an approval must be configured, providing the necessary authentication credentials and connection details.
AzureSQL was used as the RDBMS for this configuration as the organization is a heavy user of the Microsoft Azure to leverage on direct Microsoft security and easy configuration.
Architectural Explanation of the setup.
In the AzureTRE/templates/workspace_services/azuresql/terraform/azuresql.tf
, the following resources was created:
- A data resource: This was used to retrieve the existing ADF named adf_core.
- An azurerm_data_factory_linked_service_azure_sql_database linked service resource: This is the service that links the data factory and the Azure SQL server together.
- An azurerm_data_factory_managed_private_endpoint private endpoint resource: This is the resource that enables the secured connection between the data source and the data destination.
- A null_resource approval resource: This is used to point to the approve_pe.sh script.
- An approve_pe.sh file: This is used to automatically approve the private endpoint on the sql server.