Keerthana Blogs

Azure Pipeline Creation and Configuration Steps To log in to the Azure portal, a Microsoft account is required. After creating the account, sign in to the Azure portal and proceed with the following steps to build the data pipeline. 1. Create Resource Group and Storage Account Create a Resource Group in Azure. Under the resource group, create the required resources for the pipeline. Azure Data Lake Storage (ADLS Gen2) Create a Storage Account . Enable Hierarchical Namespace to convert it into Data Lake Storage Gen2 . Inside the storage account: Create a container (e.g., blog-container ). Organize data using folders/subfolders (can be created dynamically or manually using directory structure). Storage Structure Example Create a storage account named sdmm : sdmm gold processing sales silver sd mm bronze sd mm 2. Azure Data Factory (ADF) Setup Create an Azure Data Factory instance. Go to Managed Identities and enabl...

  Data Flow Name : df_transform_hospital_admissions Pipeline Steps : Source (HospitalAdmissionSource) : Pulls data from ds_raw_hospital_admission . SelectReqdFields : Renames or selects specific fields: country , indicator , etc. LookupCountry : Performs a lookup using CountrySource (likely from ds_country_lookup ) to enrich the data. SelectReqdFields2 : Refines the result further with a new set of selected or renamed fields. Split into Weekly and Daily : A Conditional Split divides the data into two branches: Weekly (9 columns total) Daily (filtering on indicator column, likely conditional logic) Right Panel : Shows general properties. Name: df_transform_hospital_admissions . Description: Empty. Bottom Panel (Data preview) : Currently loading: “Fetching data…”. Status: Data flow debug is enabled (green). Operation counts like INSERT , UPDATE , DELETE , etc., are N/A , meaning this is likely a preview r...