In ADF Mapping Data Flows, there are 2 working modes: Debug mode and Pipeline mode. Debug mode is active when you turn on the Data Flow debug switch and the light is green, showing debug as active. You will also … Continue reading Reduce Execution Time for Data Flow Activities in ADF Pipelines
Here’s a brief posting on taking an ETL script written in Pig. I took an ETL example using Pig from the Hortonworks tutorials site and migrating it to ADF using Mapping Data Flows. It took me approximately 10 minutes to … Continue reading ETL with ADF: Convert Pig to Data Flows
Using Azure Data Factory Mapping Data Flows, you can make your data transformations flexible and general-purpose by using parameters. Use Data Flow parameters to create dynamic transformation expressions and dynamic contents inside of transformation settings. The online documentation for Data … Continue reading ADF Mapping Data Flows Parameters
You can leverage ADF’s parameters feature with Mapping Data Flows to create pipelines that dynamically create new target tables. You can set those table names through Lookups or other activities. I’ve written a very simply post below on the tools … Continue reading Dynamic SQL Table Names with Azure Data Factory Data Flows
I’m going to use this blog post as a dynamic list of performance optimizations to consider when using Azure Data Factory’s Mapping Data Flow. I am going to focus this only to files. I will post subsequent articles that list ways to optimize other source, sinks, and data transformation types. As I receive more good practices, feedback, and other performance tunings, I will update this article accordingly. Here is Azure SQL DB Optimizations for ADF Data Flows Here is Azure SQL DW Optimizations for ADF Data Flows Optimizations to consider when using ADF Mapping Data Flows with files NOTE: When … Continue reading ADF Mapping Data Flows: Optimize for File Source and Sink
I’m going to use this blog post as a dynamic list of performance optimizations to consider when using Azure Data Factory’s Mapping Data Flow. I am going to focus this only to Azure SQL DW. I will post subsequent articles … Continue reading ADF Mapping Data Flows: Optimize for Azure SQL Data Warehouse
I’m going to use this blog post as a dynamic list of performance optimizations to consider when using Azure Data Factory’s Mapping Data Flow. I am going to focus this only to Azure SQL DB. I will post subsequent articles … Continue reading ADF Mapping Data Flows: Optimize for Azure SQL Database
I have been putting together a series of posts and videos around building SCD Type 1 and Type 2 using Mapping Data Flows with Azure Data Factory. In this latest post, I’m going to walk through a complete end-to-end Type … Continue reading ADF Slowly Changing Dimension Type 2 with Mapping Data Flows (complete)
A very common practice when designing Big Data ETL and Analytics solutions in the Cloud is to find creative ways to work with very large data files. Of course, Data Engineers who are working primarily on-prem also face challenges processing … Continue reading Partition Large Files with ADF using Mapping Data Flows
One of the most commonly used execution environments for Big Data transformations in ADF is Azure Data Lake Analytics (ADLA) using U-SQL scripts to transform data at scale. The new ADF Data Flow feature allows you to build data transformations … Continue reading Azure Data Factory: Build U-SQL Tweet Analysis with ADF Data Flows
MSSQLDUDE Blog 