Cross Querying in Azure SQL Database

Azure supports cross querying in Azure SQL Database through elastic queries. Elastic queries allow us to run Transact-SQL that works with multiple Azure SQL Databases and can connect to Microsoft tools like Excel, PowerBI and other third-party tools like Tableau to query across data tiers with multiple databases. Through this feature, we can query out in- large data tiers and visualize the result in business intelligence (BI) Reporting tools.

Advantages of using Elastic Queries

• Elastic queries support read-only querying of remote databases, and SQL server users can migrate applications by linking servers between an Azure SQL environment and on-premises
• Elastic queries are available on both the standard tier and premium tier
• We can execute stored procedures or remote functions using sp_execute _remote and push SQL parameters for execution on a remote database
• Through elastic query, external tables can now refer to remote tables with a different table name or schema
• According to customer scenarios, elastic queries are categorized as the following partitioning,
  • Vertical partitioning – Cross-database queries: A vertical elastic query is to query among vertically partitioned databases i.e., multiple databases that contain tables of different schema on different data sets. For instance, all tables for HR are on one database while all Finance tables are on another database. This partitioning helps one to query across or to build reports on top of tables in multiple databases
  • Horizontal Partitioning – Sharding: The process sharding is to distribute a huge volume of data having identical schema among different databases. For instance, this means distributing a huge amount of transaction table data among multiple databases for improved performance. To achieve this, elastic database tools are used where an elastic query is required to query or compile reports across multi shards

Elastic Queries in Vertical Partitioning

Data located in one SQL Database can be made available to other remote SQL Databases through elastic queries. The schema and structure of these databases can vary. This process is also known as scaling up.

Steps for implementation

Let’s assume that there are four databases namely HR, Finance, Products, CRM and here we will perform cross querying in Azure SQL Database. To execute the below queries, the user must have to ALTER ANY EXTERNAL DATA SOURCE permission under ALTER DATABASE permission. These permissions are needed to refer to the underlying data source.

1. Create database Master Key i.e., a symmetric key which is used to protect private keys of certificates and asymmetric keys that are available within the HR database as shown below.

2. Create a database scoped credential which is not mapped to a server login or database user but used by the database to access the external location anytime to perform an operation that requires access.

3. Create other external data sources for remote databases like Finance, Products, CRM with type as RDMS within the HR database. Here in the below image, we have created a data source for Finance but one or many data sources can be created as per the number of databases.

4. Create an external Table for Elastic Database query. For an external table, only the metadata is stored in SQL along with basic statistics about the table referenced. No actual data is moved or stored in SQL Server. Here I have created an external table for Finance with the above-created data source.

Now we can access remote database finance from HR Database. Likewise, we can create data sources and external tables for other databases also.

Elastic Queries in Horizontal Partitioning

Database sharding is a technique to split large databases into smaller partition across identically structured databases. These individual units are called shards which reside on a separate database. This mechanism is also called scaling out. Through this process, the data maintenance became easier.

Steps to implementation

As a prerequisite, you need to create a shard map manager along with multi shards, followed by insertion of data into the shards. For more information on the development of shards, please refer here.

Let’s take CRM Database as an instance,

Once the shard map manager has been set,

1. Create database master key and database scoped credential as shown in vertical partitioning but here the database should have the same structure

2. Create an external data source in the CRM database and pass the name of the shard map created in shard map manager to SHARD_MAP_NAME

3. Create an external table in CRM database for the usage of an elastic database query. This table contains only metadata of the table that has been referenced.

4. Now we can connect the CRM database to any third-party tools like Excel and query out the data from the remote database, namely CRMdbsrc1.

Limitations of using Elastic Queries

• • While using elastic queries in the standard tier, the performance over large databases will be slow
  • Elastic queries are not preferred for ETL when there is a large amount of data movement to a remote database
  • Only the Read-Only operations over remote databases are supported.

Learn more about Visual BI’s Microsoft Azure offerings here.

The post Cross Querying in Azure SQL Database appeared first on Visual BI Solutions.