UncategorizedLeave a comment

ADF Data Flow Metadata Functions Explained

mssqldude

A key feature of Azure Data Factory’s data flows is the ability to inspect and manipulate metadata as well as processing and transforming the data cell values. ADF enables cloud-scale ETL data transformations with data flows, meaning that you can leverage these built-in metadata functions for data introspection for powerful solutions. In this post, I’ll talk about 3 such examples that will allow you to perform these complex data engineering tasks at scale with zero code: Transform data conditionally based on metadata traits Create data quality rules Manipulate column properties of source data Transform data conditionally based on metadata traits … Continue reading ADF Data Flow Metadata Functions Explained

UncategorizedLeave a comment

ADF Data Flows: Here’s a look at different Azure IR Configurations

mssqldude

To help you decide which Azure IR configuration is the best fit for your data flows, I picked 3 different configurations below and captured their timings to show you how these selections can impact the performance, cost, and scale of your data flows. The data I used for the tests is from a CSV file with 74 columns, 887k rows, and the transformations I used were New Branch, Aggregate, Derived Column, and Sink to a Blob Store folder. I left all file settings and partition settings default, no optimizations. The timings below were all executed from the active debug session … Continue reading ADF Data Flows: Here’s a look at different Azure IR Configurations

Big Data Analytics, business intelligence, cloud, Cloud BI, Data Warehouse, ETL, Monitoring2 Comments

Custom logging and auditing of ADF Data Flows

mssqldude

ADF Data Flows Custom Logging and Auditing Video ADF has a number of built-in capabilities for logging, monitoring, alerting, and auditing your pipelines. There are UI monitoring tools, telemetry logs, and integration with Azure Monitor to provide a rich set … Continue reading Custom logging and auditing of ADF Data Flows

Azure, BI, Big Data, Big Data Analytics, business intelligence, Cloud BI, ETLLeave a comment

ADF Mapping Data Flows Parameters

mssqldude

Using Azure Data Factory Mapping Data Flows, you can make your data transformations flexible and general-purpose by using parameters. Use Data Flow parameters to create dynamic transformation expressions and dynamic contents inside of transformation settings. The online documentation for Data … Continue reading ADF Mapping Data Flows Parameters

Big Data, Big Data Analytics, Data Management, Data Warehouse, ETL, sql server2 Comments

Dynamic SQL Table Names with Azure Data Factory Data Flows

mssqldude

You can leverage ADF’s parameters feature with Mapping Data Flows to create pipelines that dynamically create new target tables. You can set those table names through Lookups or other activities. I’ve written a very simply post below on the tools … Continue reading Dynamic SQL Table Names with Azure Data Factory Data Flows