Software Developer ETL

Client : Government

Type : Contract

Role : Software Developer ETL

Term : 4 Months + 1 year extension

Pay Rate : $90.00 - $100

Start date : 2-3 weeks

Location : Toronto / Hybrid

Req ID : RQ08100

Requirements : What you'll need

Skills, Knowledge, Experience, and Qualifications :

Must Have Skills :

• 7+ years using ETL tools such as Microsoft SSIS, stored procedures, T-SQL
• 2+ years using Delta Lake, Databricks and Azure Databricks pipelines
• Strong knowledge of Delta Lake for data management and optimization.
• Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
• 2+ years Python and PySpark
• Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments.
• Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data.
• SQL Server, Oracle

Development, Database and ETL experience :

• Experience in developing and managing ETL pipelines, jobs, and workflows in Databricks.
• Deep understanding of Delta Lake for building data lakes and managing ACID transactions, schema evolution, and data versioning.
• Experience automating ETL pipelines using Delta Live Tables, including handling Change Data Capture (CDC) for incremental data loads.
• Proficient in structuring data pipelines with the Medallion Architecture to scale data pipelines and ensure data quality.
• Hands-on experience developing streaming tables in Databricks using Structured Streaming and readStream to handle real-time data.
• Expertise in integrating CDC tools like GoldenGate or Debezium for processing incremental updates and managing real-time data ingestion.
• Experience using Unity Catalog to manage data governance, access control, and ensure compliance.
• Skilled in managing clusters, jobs, autoscaling, monitoring, and performance optimization in Databricks environments.
• Knowledge of using Databricks Autoloader for efficient batch and real-time data ingestion.
• Experience with data governance best practices, including implementing security policies, access control, and auditing with Unity Catalog.
• Proficient in creating and managing Databricks Workflows to orchestrate job dependencies and schedule tasks.
• Strong knowledge of Python, PySpark, and SQL for data manipulation and transformation.
• Experience integrating Databricks with cloud storage solutions such as Azure Blob Storage, AWS S3, or Google Cloud Storage.
• Familiarity with external orchestration tools like Azure Data Factory
• Implementing logical and physical data models
• Knowledge of FHIR is an asset

Certifications :

• Certified in one or more of the following certifications :
• Databricks Certified Data Engineer Associate
• Databricks Certified Professional Data Engineer
• Microsoft Certified : Azure Data Engineer Associate
• AWS Certified Data Analytics - Specialty
• Google Cloud Professional Data Engineer

PREFERRED SKILLS

ETL + Azure + Databricks + Delta Lake + Python

HOW TO APPLY

Submit your resume in confidence through .

We thank all candidates for their interest but advise that only candidates selected for interviews will be contacted directly.

Rest assured that your resume will be kept on our database and will be considered for all future opportunities.

Applicants must be authorized to work in Canada.

Wirehead Talent Inc. is proud to be an Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, sexual orientation, gender identity or expression, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, status as a crime victim, disability, protected veteran status, or any other characteristic protected by law.

Wirehead is committed to working with and providing reasonable accommodation to individuals with physical and mental disabilities.

If you need special assistance or an accommodation while seeking employment, please contact a member of our Human Resources team to make arrangements.

Email : [email protected]

Your Right to Work All persons hired will be required to verify identity and eligibility to work in the Canada and to complete the required employment eligibility verification document form upon hire.

Copyright 2007 - 2024 Wirehead Talent All rights reserved