Description
TL;DR. This course will introduce Azure Data Factory and how it can help in the batch processing of data. Students will learn with hands-on activities, quizzes, and a project, how Data Factory can be used to integrate many other technologies together to build a complete ETL solution, including a CI/CD pipeline in Azure DevOps. Some topics related to Data Factory required for the exam DP-203: Data Engineering on Microsoft Azure, are covered in this course. Learn by DoingTogether, you and I are going to learn everything you need to know about using Microsoft Azure Data Factory. This course will prepare you with hands-on learning activities, videos, and quizzes to help you gain knowledge and practical experience as we go along. At the end of this course, students will have the opportunity to submit a project that will help them to understand how ADF works, what are the components, and how to integrate ADF and Databricks. Student key takeaways: The student should understand how ADF orchestrates the features of other technologies to transform or analyze data. The student should be able to explain and use the components that make up ADF. The student should be able to integrate two or more technologies using ADF. The student should be able to confidently create medium complex data-driven pipelinesThe student should be able to develop a CI/CD pipeline in Azure DevOps to deploy Data Factory pipelinesWhat You'll Learn: Introduction to Azure Data Factory. You will understand how it can be used to integrate many other technologies with an ever-growing list of connectors. How to set up a Data Factory from scratch using the Azure Portal and PowerShell. Activities and Components that makeup Data Factory. It will include Pipelines, Datasets, Triggers, Linked Services, and more. How to transform, ingest, and integrate data code-free using Mapping Data Flows. How to integrate Azure Data Factory and Databricks. We'll cover how to authenticate and run a few notebooks from within ADF. Azure Data Factory Deployment using Azure DevOps for continuous integration and continuous deployment (CI/CD)Data Factory Essentials Training - OutlineIntroductionModules introductionGetting StartedUnderstand Azure Data Factory ComponentsIngesting and Transforming Data with Azure Data FactoryIntegrate Azure Data Factory with DatabricksContinuous Integration and Continuous Delivery (CI/CD) for Azure Data FactoryGetting startedSign up for your Azure free accountSetting up a BudgetHow to set up Azure Data FactoryAzure PortalPowerShellAzure Data Factory ComponentsLinked ServicesPipelinesDatasetsData Factory ActivitiesParametersPipeline ParametersActivity ParametersGlobal ParametersTriggersIntegration Runtimes (IR)Azure IRSelf-hosted IRLinked Self-Hosted IRAzure-SSIS IRQuizIngesting and Transforming DataIngesting Data using Copy Activity into Data Lake Store Gen2How to Copy Parquet Files from AWS S3 to Azure SQL DatabaseCreating ADF Linked Service for Azure SQL DatabaseHow to Grant Permissions on Azure SQL DB to Data Factory Managed IdentityIngesting Parquet File from S3 into Azure SQL DatabaseCopy Parquet Files from AWS S3 into Data Lake and Azure SQL Database (intro)Copy Parquet Files from AWS S3 into Data Lake and Azure SQL DatabaseMonitoring ADF Pipeline ExecutionTransforming data with Mapping Data FlowMapping Data Flow Walk-throughIdentify transformations in Mapping Data FlowMultiple Inputs/OutputsSchema ModifierFormattersRow ModifierDestinationAdding source to a Mapping Data FlowDefining Source Type; Dataset vs InlineDefining Source OptionsSpinning Up Data Flow Spark ClusterDefining Data Source Input TypeDefining Data SchemaOptimizing Loads with PartitionsData Preview from Source TransformationHow to add a Sink to a Mapping Data FlowHow to Execute a Mapping Data FlowQuizIntegrate Azure Data Factory with DatabricksProject Walk-throughHow to Create Azure Databricks and Import NotebooksHow to Transfer Data Using Databricks and Data FactoryValidating Data Transfer in Databricks and Data FactoryHow to Use ADF to Orchestrate Data Transformation Using a Databricks NotebookQuizContinuous Integration and Continuous Delivery (CI/CD) for Azure Data FactoryHow to Create an Azure DevOps Organization and ProjectHow to Create a Git Repository in Azure DevOpsHow to Link Data Factory to Azure DevOps RepositoryHow to version Azure Data Factory with BranchesData Factory Release WorkflowMerging Data Factory Code to Collaboration BranchHow to Create a CI/CD pipeline for Data Factory in Azure DevOpsHow to Create a CICD pipeline for Data Factory in Azure DevOpsHow to Execute a Release Pipeline in Azure DevOps for ADFQuiz
