Executing SSIS Package using Azure Data Factory

Microsoft has the powerful feature of either lifting the existing SSIS package to Azure Data Factory (V2) or Integrating a new SSIS package with ADF(V2). This feature adds more strength to Azure like,

• Reduction in Infrastructure maintenance cost. It does not have SQL Server Licensing cost as ADF meets with ETL need without SQL Server License and with reduced cost.
• Allows for easy scaling up as your processing needs to grow.
• Less time to set up a new server for SSIS ETL in Azure Cloud approach.

Now let me explain how to integrate and execute an SSIS Package on Azure Data Factory

To proceed with the execution, the following pre-requisites should be available:

• SQL Server Management Studio Version 17.2 or later
• SSIS Database (Integration Catalog) in Azure
• Azure SSIS Integration Runtime
• Visual Studio 2015 or above for creating Packages

Step 1: Creating SSIS package

I have considered a sample scenario with ADLS as Source and Azure SQLDB as Destination. The Input file has Shipment Order Details with OrderDate and ShipDate. And the requirement is to find the Elapsed Days of Shipment for each Order excluding Weekends and Public Holidays for which I have taken the US Holidays List.

For this use case, I have created an SSIS package as below in Visual Studio using the Integration services project template.

Control Flow		Data Flow

Step 2: Deploy the SSIS Package in Azure SSIS Catalog

The deployment in Azure is like how we deploy the traditional SSIS Package and can be done through SSMS or Visual Studio.

• You can connect the Azure SSIS Catalog through SQL Server Management Studio. For connecting the SSIS catalog, enter the Server Type, Server Name, Authentication.
• Then select Options>> before connecting. In the Connection properties tab, select SSISDB in the Connect to Database dropbox.
• Then click on Connect. And expand the Integration Services Catalogs to view SSISDB

• In SSMS, right-click the project folder. Select Deploy Project option. Enter the Source details in the Deployment wizard.

• Choose project Deployment and browse for the ISPAC file.
• ISPAC file will usually reside in the path:  //SSISProject/bin/Development/SSISProject.ispac
• Click Next to proceed with destination details.
• Enter the Server Name in which Azure SSISDB resides, authentication details and the path in which the project is to be deployed.

• Click next to validate and then click on deploy.
• In case, you need to deploy in Visual Studio, right-click the project in Solution Explorer. The deployment wizard opens, follow the same steps as above.

Step 3: Create a pipeline in the Azure Data Factory V2

• Open the Azure portal, go to Azure data factory(V2).
• Select Create pipeline. Choose Execute SSIS Package activity. Configure the activity in the Settings.

**Make sure you have given the right package path from SSISDB

• Then select Trigger option in the pipeline for executing the package. The SSIS Package gets executed through the pipeline. We can also schedule the package using triggers to run at predefined intervals.

Step 4: Trigger/Schedule the Package on ADF(V2) (optional)

• Before setting the trigger for a pipeline, the trigger parameter must be set. This can be set through Trigger Tab in ADF V2 UI. Click on the Trigger Tab, Add New Trigger.
• In the New Trigger Window, set the properties to be scheduled.

• And Click Finish. Now go to the pipeline, select Trigger then choose New/Edit to choose the Trigger.

• Once New/Edit is selected, you will be prompted with the Add Trigger Window.
• Choose the Trigger from the Drop down, click Next to edit if any changes needed and then click finish to start the trigger.

So, with the above mentioned steps we can execute an SSIS package on Azure data factory(V2). The SSIS can be used in many cases to solve the complex transformation problems in Azure ETL.

Please reach out Visual BI for questions on setting up the SSIS Integration Runtime or clarifications regarding integrating SSIS package with Azure Data Factory(V2).

The post Executing SSIS Package using Azure Data Factory appeared first on Visual BI Solutions.