RQ08079 - Software Developer - ETL - Senior

Scope :

TheOffice of the Public Guardian and Trustee (OPGT) requires a SeniorSoftware Developer ETL to perform data migration activities fromthe existing OPGT legacy applications to the new Dynamics 365 OPGTsolution and datawarehouse / lakehouse.

AssignmentDeliverables :

As a member ofthe data migration team you will be responsible to migrate the datafrom the existing OPGT legacy applications to the new Dynamics 365OPGT solution and the new data warehouse / lakehouse.

A highlevellist of deliverables for the data migration teamfollows :

• DataAnalysis : analyze the existing data in the legacy applicationsunderstand its structure quality and relationships and help indesigning an appropriate migrationstrategy;
• DataMapping and Transformation : map the data elements from the legacyapplication to the corresponding entities and fields in Dynamics365 CE F&O and Data Lakehouse.
• Handle necessary datatransformations ensuring compatibility and consistency between thelegacy data and the targetsystem;
• DataExtraction : help extract the required data from the legacyapplication develop and implement extraction processes to retrievedata from various sources such as databases files APIs or otherrelevant legacysystems;
• DataCleansing and Validation : cleanse and validate the extracted datato ensure its accuracy completeness and consistency.
• Help withidentifying and resolving data quality issues performingdeduplication and applying business rules to ensure the integrityof the migrateddata;
• DataMigration Strategy and Execution : review the present migrationstrategy that outlines the overall approach sequence and timelinefor migrating the data from the legacy application to Dynamics 365using a deltaload approach;
• execute the migration plan efficientlymanaging data transfers and ensuring minimal disruption to ongoingoperations;
• DataTesting and Quality Assurance : conduct thorough testing to verifythe accuracy and integrity of the migrated data;
• define test casesperform data reconciliation and address any issues or discrepanciesthat arise during the testing phase;
• develop KPIs to report on theprogress completeness and quality of the data migrationeffort;
• Documentation : document the entire data migration process including data mappingrules transformation logic migration scripts and any specificconfigurations;
• OngoingSupport : provide postmigration support analyze and addressdatarelated issues or questions; help optimize data managementprocesses in the newenvironment;
• DataLakehouse : help expand the current data lakehouse implementationand help build a gold layer for reporting; conduct data analysisand profiling;
• develop data transformation and cleansing pipelines;implement data integration andharmonization;
• Otherduties asassigned;

Requirements

Experienceand Skill SetRequirements :

MustHaves :

5years of proven working experience in an ETLrole ; strong understanding of ETL principlesincluding data extraction transformation and loading processes;

knowledge of common ETL design patterns. Understanding of datapipeline architectures Azure workflow orchestration tools andconcepts related to data ingestion transformation andmovement.

• Proficiencyin Azure Data Factory Azure Synapse workspaces PolyBase includingknowledge of pipeline creation data flows integration runtimestriggers and monitoring.
• Strong SQLskills and experience working with Azure SQL Databases andDataverse; good understanding of Azure storage concepts andtechnologies.
• Proficiencyin scripting languages like Python experience with Azurespecificscripting using PowerShell or AzureCLI.
• Expertproficiency with data manipulation languages (TSQL PL / SQL) datadefinition languages physical database design data modeling queryperformance analysis &tuning
• Knowledgeof integration technologies commonly used with Dynamics such asDataVerse / Common Data Service (CDS) Data Entities andAPIs.
• Experiencewith continuous integration / continuous deployment (CI / CD) processesaround DevOps data workflows Synapseworkspaces.
• Familiaritywith data warehousing concepts and experience working with AzureSynapse Analytics or similar platforms for building and managinglargescale data warehousing / lakehousesolutions.
• Experiencewith SSIS SSRSPowerBI

Nice tohave :

• Azurecloud certifications (e.g. Azure fundamentals Azure Data Engineerassociate Azure Database Administratorassociate)
• Experiencewith PowerApps platform Power Automate Dynamics CE andF&O

Skill SetRequirements :

DesiredSkills andExperience :

5years of proven working experience in an ETL role; strongunderstanding of ETL principles including data extractiontransformation and loading processes;

knowledge of common ETLdesign patterns. Understanding of data pipeline architectures Azureworkflow orchestration tools and concepts related to data ingestiontransformation andmovement.

• Experiencein integrating various data sources and systems both onpremises andin the cloud using Azure ETL services or other ETLtools
• Knowledgeof integration technologies commonly used with Dynamics such asDataVerse / Common Data Service (CDS) Data Entities andAPIs.
• Expertisein data transformation techniques such as data cleansingaggregation enrichment and normalization using Azure cloudtechnologies
• Understandingof data quality management practices including data profiling datavalidation and error handling within ETL processes.
• Understandingof data governance principles data privacy regulations andexperience working with highsensitivity data and knowledge of bestpractices for data security and compliance in Azure.
• Ability tomonitor and troubleshoot ETL processes optimize query performanceand implement efficient data processing techniques inAzure.
• Proficiencyin Azure Data Factory Azure Synapse workspaces PolyBase includingknowledge of pipeline creation data flows integration runtimestriggers and monitoring.
• Strong SQLskills and experience working with Azure SQL Databases andDataverse; good understanding of Azure storage concepts andtechnologies.
• Proficiencyin scripting languages like Python and experience withAzurespecific scripting using PowerShell or Azure CLI.
• Expert indata manipulation languages (TSQL PL / SQL) data definition languagesphysical database design data modelling query performance analysis&tuning
• Familiaritywith version control systems (e.g. Azure Repos) and collaborationtools (e.g. Azure DevOps) for managing code tracking changes andcollaborating with teammembers.
• Experiencewith continuous integration / continuous deployment (CI / CD) processesaround DevOps data workflows Synapseworkspaces.
• Experiencewith SQL Server Management Studio Azure data management tools XRMtoolbox data modeling tools (preferably ERWIN).
• Familiaritywith data warehousing concepts and experience working with AzureSynapse Analytics or similar platforms for building and managinglargescale data warehousing / lakehousesolutions.
• Experiencewith SSIS SSRSPowerBI

Data MigrationETL :

• Demonstratedexperience with ETL development data pipelines workfloworchestration and data ingestion transformation andmovement
• Demonstratedexperience in integrating various data sources and systems bothonpremises and in the cloud using Azure ETL services or other ETLtools
• Demonstratedexperience working with Azure Data Factory Azure Synapse workspacesPolyBase including knowledge of pipeline creation data flowsintegration runtimes triggers andmonitoring.
• Demonstratedexperience with data manipulation languages (TSQL PL / SQL) datadefinition languages query performance analysis &tuning
• Demonstratedexperience with SQL Server Oracle Azure SQLDatabases
• Demonstratedexperience with data modeling tools (preferablyERWIN)
• Demonstratedexperience in scripting languages like Python and withAzurespecific scripting using PowerShell or AzureCLI.
• Experiencewith software developmentlifecycle
• Experiencewith data modeling physical database design data flowdiagrams

Data Warehouse andReporting :

• Demonstratedexperience working with Azure Synapse Analytics or similarplatforms for building and managing largescale datawarehousing / lakehousesolutions
• Experiencewith data warehousing modelling concepts such as star and snowflakeschemas
• Experiencewith SSIS SSRSPowerBI
• Experiencewith supporting a data warehouse in a productionenvironment

AzurePlatform :

• Experiencewith Azure Data Factory (ADF) and SynapseWorkspaces
• Demonstratedexperience Azure data management tools DevOps SynapseStudio
• Experiencein Azure resource configuration and administration such as AzureData Lake Blob Storage Key Vault Application Insight resourcesresource groups andsubscriptions.
• Familiarwith Azure cloudplatform
• Azurecloudcertifications

Dynamics365 :

• Demonstratedexperience working with integration technologies commonly used withDynamics such as DataVerse / Common Data Service (CDS) DataEntities andAPIs.
• Demonstratedexperience with PowerApps platform Power Automate Dynamics CE&F&O

DevOps andCI / CD :

• Demonstratedexperience with continuous integration / continuous deployment(CI / CD) tools and processes around DevOps data workflows Synapseworkspaces.
• Experience and Skill Set Requirements : Must Haves : 5+ years ofproven working experience in an ETL role; strong understanding ofETL principles, including data extraction, transformation, andloading processes;

knowledge of common ETL design patterns.Understanding of data pipeline architectures, Azure workfloworchestration tools, and concepts related to data ingestion,transformation, and movement.

Proficiency in Azure Data Factory,Azure Synapse workspaces, PolyBase, including knowledge of pipelinecreation, data flows, integration runtimes, triggers, andmonitoring.

Strong SQL skills and experience working with Azure SQLDatabases and Dataverse; good understanding of Azure storageconcepts and technologies.

Proficiency in scripting languages likePython, experience with Azure-specific scripting using PowerShellor Azure CLI. Expert proficiency with data manipulation languages(T-SQL, PL / SQL), data definition languages, physical databasedesign, data modeling, query performance analysis & tuningKnowledge of integration technologies commonly used with Dynamics,such as DataVerse / Common Data Service (CDS), Data Entities, andAPIs.

Experience with continuous integration / continuous deployment(CI / CD) processes around DevOps, data workflows, Synapseworkspaces.

Familiarity with data warehousing concepts andexperience working with Azure Synapse Analytics or similarplatforms for building and managing large-scale datawarehousing / lakehouse solutions.

Experience with SSIS, SSRS,PowerBI Nice to have : Azure cloud certifications (e.g. Azurefundamentals, Azure Data Engineer associate, Azure DatabaseAdministrator associate) Experience with PowerApps platform, PowerAutomate, Dynamics CE and F&O Skill Set Requirements : DesiredSkills and Experience : 5+ years of proven working experience in anETL role;

strong understanding of ETL principles, including dataextraction, transformation, and loading processes; knowledge ofcommon ETL design patterns.

Understanding of data pipelinearchitectures, Azure workflow orchestration tools, and conceptsrelated to data ingestion, transformation, and movement.

Experiencein integrating various data sources and systems, both on-premisesand in the cloud, using Azure ETL services or other ETL toolsKnowledge of integration technologies commonly used with Dynamics,such as DataVerse / Common Data Service (CDS), Data Entities, andAPIs.

Expertise in data transformation techniques, such as datacleansing, aggregation, enrichment, and normalization using Azurecloud technologies Understanding of data quality managementpractices, including data profiling, data validation, and errorhandling within ETL processes.

Understanding of data governanceprinciples, data privacy regulations and experience working withhigh-sensitivity data, and knowledge of best practices for datasecurity and compliance in Azure.

Ability to monitor andtroubleshoot ETL processes, optimize query performance, andimplement efficient data processing techniques in Azure.

Proficiency in Azure Data Factory, Azure Synapse workspaces,PolyBase, including knowledge of pipeline creation, data flows,integration runtimes, triggers, and monitoring.

Strong SQL skillsand experience working with Azure SQL Databases and Dataverse; goodunderstanding of Azure storage concepts and technologies.

Proficiency in scripting languages like Python, and experience withAzure-specific scripting using PowerShell or Azure CLI.

Expert indata manipulation languages (T-SQL, PL / SQL), data definitionlanguages, physical database design, data modelling, queryperformance analysis & tuning Familiarity with version controlsystems (e.

g., Azure Repos) and collaboration tools (e.g., AzureDevOps) for managing code, tracking changes, and collaborating withteam members.

Experience with continuous integration / continuousdeployment (CI / CD) processes around DevOps, data workflows, Synapseworkspaces.

Experience with SQL Server Management Studio, Azuredata management tools, XRM toolbox, data modeling tools(preferably, ERWIN).

Familiarity with data warehousing concepts andexperience working with Azure Synapse Analytics or similarplatforms for building and managing large-scale datawarehousing / lakehouse solutions.

Experience with SSIS, SSRS,PowerBI Data Migration, ETL : Demonstrated experience with ETLdevelopment, data pipelines, workflow orchestration and dataingestion, transformation, and movement Demonstrated experience inintegrating various data sources and systems, both on-premises andin the cloud, using Azure ETL services or other ETL toolsDemonstrated experience working with Azure Data Factory, AzureSynapse workspaces, PolyBase, including knowledge of pipelinecreation, data flows, integration runtimes, triggers, andmonitoring.

Demonstrated experience with data manipulationlanguages (T-SQL, PL / SQL), data definition languages, queryperformance analysis & tuning Demonstrated experience with SQLServer, Oracle, Azure SQL Databases Demonstrated experience withdata modeling tools (preferably, ERWIN) Demonstrated experience inscripting languages like Python and with Azure-specific scriptingusing PowerShell or Azure CLI.

Experience with software developmentlifecycle Experience with data modeling, physical database design,data flow diagrams Data Warehouse and Reporting : Demonstratedexperience working with Azure Synapse Analytics or similarplatforms for building and managing large-scale datawarehousing / lakehouse solutions Experience with data warehousingmodelling concepts such as star and snowflake schemas Experiencewith SSIS, SSRS, PowerBI Experience with supporting a datawarehouse in a production environment Azure Platform : Experiencewith Azure Data Factory (ADF) and Synapse Workspaces Demonstratedexperience Azure data management tools, DevOps, Synapse StudioExperience in Azure resource configuration and administration suchas Azure Data Lake, Blob Storage, Key Vault, Application Insightresources, resource groups and subscriptions.

Familiar with Azurecloud platform Azure cloud certifications Dynamics 365 : Demonstrated experience working with integration technologiescommonly used with Dynamics, such as DataVerse / Common DataService (CDS), Data Entities, and APIs.

Demonstrated experiencewith PowerApps platform, Power Automate, Dynamics CE &F&ODevOps and CI / CD : Demonstrated experience with continuousintegration / continuous deployment (CI / CD) tools and processesaround DevOps, data workflows, Synapse workspaces.