Parameterize Connections in Azure Data Factory

Azure Data Factory is a managed cloud service that is built for complex data orchestration processes and hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects. In this blog, we will demonstrate how to parameterize connection information in a Linked Service, which will enable the passing of connection information dynamically, and eliminate the need to create multiple Linked Services for accessing servers with many databases.

Linked Services

Linked Services can be thought of as connection strings, which define the connection information needed for Azure Data Factory to connect to external data sources.

For example, if you want to parameterize the Azure SQL database connection in Azure Data Factory you would start by creating a new Linked Service and then add the dynamic content in the Linked Service property that you want to parameterize like Server Name, Database Name or the User Credentials.

Create Parameters

Linked Service enables you to parameterize connection information so that values can be passed dynamically.

Add Dynamic Content

• Choose the ‘Enter manually’ option
• Click on ‘Add Dynamic Content’ under each input box for mapping the parameter

After creating the Linked Service, you will need a dataset to invoke the dynamic content in the Linked Service that accesses a table or query in the database. For example, if you want to move data from Azure SQL Database to Azure SQL Data Warehouse, then you would need to create two datasets one for the source Azure SQL Database and the other one for the sink Azure SQL Data Warehouse.

Create Datasets

• Invoke the Linked Service which we already created and map the Linked Service to the Dataset for accessing the data dynamically, without creating another Linked Service
• Image (1)represents the Dataset accessing the Azure SQL Database“sqldb” by passing values into the parameter fields
• Image (2)represents the Dataset accessing the Azure SQL Data Warehouse“sqldw” by passing values into the parameter fields
• All the values are parameterized, so we can dynamically pass values into Server Name, Database Name, Username and Password
• To illustrate our scenario, only database names are dynamically passed and the data is accessed from the database

Image (1) – Accessing ‘sqldb’ dynamically through parameters

Image (2) – Accessing ‘sqldw’ dynamically through parameters

Now let us see how to use the parametrized connection in a pipeline.

Create a Pipeline

Create a pipeline and invoke the dataset in the source or sink based on your needs. Once the dataset is selected, it will prompt for the parameters you created in the dataset.

• Pass the parameter value from the pipeline into the dataset
• The same dataset can be used in different pipelines to access different databases or servers
• In the pipeline below, the same Dataset and Linked Service is used, however, it is extracting data out of the SQL Data Warehouse, sqldw

We now have a pipeline that uses a single Linked Service for connecting to multiple databases within a single server. This way of implementation of Linked Services will reduce overhead and improve the manageability of your data factories.

Learn more about Visual BI’s Microsoft Azure offerings here.

The post Parameterize Connections in Azure Data Factory appeared first on Visual BI Solutions.